Stephen Nuske, a senior project scientist at Carnegie Mellon’s Robotics Institute is working on a robotic system that could revolutionize the routine at vineyards. His team’s system could improve both the quality and quantity of ripening grapes, which should be delicious news to everyone from wine enthusiasts to jelly lovers.
The sun is setting over California’s vineyards. Tourist groups, feeling no pain from a fruitful day of wine tasting, have returned to their hotels. The day’s warm temperatures have predictably cooled off, which is good news for row after row of fruit-laden vines. By anyone’s account, it’s a peaceful scene. But only for a moment.
For thousands of vineyard laborers, the workday is about to begin. The USDA estimates that there are nearly 800,000 fruit-bearing grape acres in California, an area equivalent to some 605,000 football fields. This vast spread of land, encompassing table grapes, wine grapes, and raisin grapes, takes an equally vast upkeep effort. Grapes must be given the proper amount of water and fertilizer. They must be sprayed with pesticides to deter insects and fungicide to prevent mildew. They must be harvested at the proper point in their growing cycle and transported at a temperature that disallows fermentation or freezing. A considerable amount of the work that goes into maintaining these conditions happens at night, when temperatures cool.
For the past two summers, some of the nighttime field laborers have been joined by a team of Carnegie Mellon scientists, data analysts, and engineers. The makeup of the team occasionally changes, as members rotate in and out of roles. But Stephen Nuske is always at the head of the group, leading a project that seeks to increase vineyard efficiency and quality through something called High-Resolution Spatiotemporal Crop Load Measurement and Management.
In essence, Nuske, a senior project scientist at CMU’s Robotics Institute, is conducting a trial of an automated grape-counting system that will allow vineyard managers to track variations in their fields and more accurately estimate their harvest sizes. Ideally, the system will allow managers to make changes to their growing practices during the season to improve their overall yields as well as the quality of the fruit.
The general field of agricultural robotics clicked with Nuske early in his CMU career, which began in 2008. He had always been interested in creating relevant, need-based technology. Meanwhile, agriculture, which is so dependent on manual labor, is a field in search of robotic applications that could improve efficiency, control costs, and provide better yields.
In 2012, he collaborated with Research Professor Sanjiv Singh on a project designed to test “Automated Crop Yield Estimation for Apple Orchards.” The program caught the attention of the National Grape and Wine Initiative, a coalition that coordinates grape research and growing practices across the United States. The NGWI reached out to Nuske to see whether the technology he was working on could be adapted to the grape industry, and Nuske jumped on board, partnering with Cornell researchers as well as his CMU colleagues to pioneer the new grape-counting technology.
Of course, “grape counting” is a simplification of what Nuske’s system actually does. The rig involves a camera (to take photos of the vines, from which a computer system will later detect and count the fruit), a laser scanner (which takes a three-dimensional measurements of vine foliage), a computer system (which tracks the data as they come in), and the vehicle itself (which allows the team to quickly cover large areas of the vineyard). The system is typically manned by three people: one to drive, one to watch the data, and one to process the numbers. The result is an odd-looking contraption, futuristic in function but humble in design. If you were to stumble onto one of the trial vineyards at 1am, you’d find a startling combination of robot, strobe light, and tractor. The rumbling vehicle rolls through the dark vineyard like a low-flying UFO, sending out bright camera flashes at a rate of five per second.
Nuske works the nocturnal hours in order to minimize light variation in the photographs, ensuring more accurate estimations. But keeping that schedule for a month can be brutal. He usually tries to finish up by 2am, in order to keep “some sort of sanity.” But if there are issues at any point in the week—a mechanical failure, a software glitch—it may well mean several nights of working till dawn. Luke Yoder, a graduate student at CMU’s Robotics Institute and the mechatronic engineer on the project, spent two weeks on the California trials. He remembers the team would often arrive back at their hotel just in time for the morning breakfast buffet. The 15-hour days (or, rather, nights) took “lots of caffeine,” Yoder laughs. It’s a schedule that may sound familiar to any CMU student who’s slogged through a difficult finals week.
But the long hours have paid off. The data Nuske and his team collected over their past two trials are striking. When the predicted harvest yield (gathered by the grape-counting system) is compared to the actual harvest yield (determined months later), the overall error is less than 5%; in comparison, current industry standard predictions may be in error as high as 20-30%, says Nuske.
It’s an impressive improvement, and it’s not the only use of Nuske’s system. The most important aspect of the technology is its ability to show variation within the vineyards. He explains that viticulture (the study of grapes) is all about vine balance: the amount of leaf area as compared to the amount of fruit. The idea is summed up in a concept called “crop load,” which refers to both the amount of fruit and the health of the vine. Terry Bates, the director of Cornell University’s Lake Erie Research and Extension Lab and a collaborator on Nuske’s project, compares crop load to the process of losing weight. A large number of leaves mean that a large amount of photosynthates are in the vine, available to the fruit. These photosynthates, Bates explains, are the “calories in.” And the grapes drawing the photosynthates from the vine are the “calories out.” In a healthy vine, the calories in and calories out are balanced. Too little or too much leaf area and the grapes suffer, leading to consequences that can go far beyond a bad glass of vino. According to the NGWI, grapes are the sixth-largest crop in the United States, supporting an industry that is valued at $4.9 billion and contributes $162 billion to the U.S. economy (and $33 billion in wages) each year. These numbers illustrate why the NGWI is concerned about maximizing crop yield.
Once all the data are collected, Nuske creates a spatial map of the parts of the vineyard through which his equipment has traveled. The spatial maps are constructed with a combination of each area’s crop load number, determined by the mathematical relationship between the amount of fruit and the amount of leaf area. The maps show which parts of the vineyards are doing well, and which aren’t. Growers can then adjust their methods to improve the balance between each area’s foliage and fruit. “This is kind of the cutting edge in viticulture,” Nuske explains. “A hundred years ago, if you wanted to grow a really good bottle of wine ... you’d want to actually reduce the amount of fruit.” So farmers would remove grapes from the vine. But, in fact, Nuske explains, the past 20 years have shown that “you can produce a really good-quality crop with a lot of fruit with really big plants. ... It’s not the old case of less gives you better quality.” As long as leaf and fruit weight are balanced, a large crop can be a quality crop.
This is good news for growers, Bates explains, because competition from domestic and foreign markets means that the price paid for grapes has not increased very much during the past 40 years. The result, he says, is that growers have been forced to become more efficient to offset rising expenses such as fuel and labor over the years, so they can still make a profit. And Nuske is putting the tools for greater efficiency in the palms of their hands, literally. His spatial maps can be downloaded to smartphones, so that growers can walk through their fields along the course of the maps, look at the high- and low-yield areas in person, and determine how to improve the crop load: water here, prune here, fertilize here. In an industry that is endlessly variable, the information that Nuske’s system offers is a kind of superpower, enabling growers to adapt with the season, rather than rely on the conditions of previous years.
Before growers can take full ownership of the technology, Nuske’s team must consider the logistics of mass production and commercial adoption. For instance, although growers can use the spatial maps, Nuske still has to make them—his team’s expertise translates the data into an accessible form. Once the system is scaled up, vineyard managers will need to interpret their data without a team of CMU experts on hand. So Nuske is focusing on making the entire system automated. To do so, he must verify that the numbers his team has collected over the past two years (and plans to collect again this summer) are consistent enough to be plugged into algorithms across all types of grape varieties and terrains.
His results, so far, are encouraging. And once the system is automated, he anticipates that growers will be eager to incorporate the technology into their growing process. “It’s really like the last 20 years of viticulture meeting the last 10 years of progression in robotics,” he explains. For commercial growers, especially, who need large quantities of healthy grapes to fill the fruit bowls, wine glasses, and juice cups of the world, the technology could soon be invaluable.
“Steve has found this really nice medium between theoretical and applied [research],” Yoder says. “He can go out in the field and get his hands dirty ... and then he can also write an amazing scientific [paper] on the theory behind why what we’re doing works. I haven’t met a lot of people who have been able to gracefully move back and forth between theoretical and practical. Steve has mastered it.” And he has mastered it at an astoundingly quick pace. When Yoder first met Nuske last year, the two sat down to discuss the project, kicking around various ideas for the setup of the cameras. “And then you jump forward maybe six months or so,” Yoder says, “[and we’re] in a hotel room with machine parts all over the floor. I’m soldering something inside one of the machines, and the other guys are programming.”
Multiple fields intersect to make Nuske’s work possible: biology, computer programming, engineering, robotics, agriculture. The system is complex, and Nuske acknowledges that the process of research and development can be tiring, even tedious. But in the pre-dawn hours, watching field laborers in other parts of the vineyard, he recognizes the privilege of his work. “I think what we’re doing is probably not anywhere near as tough as what they’re doing,” he says. Yet, despite the difficulty of farm labor, some critics question the future of agricultural technology, worrying that the human component of farming will be phased out, that jobs will suffer. But Nuske points out that his system isn’t taking over anyone’s livelihood. There aren’t any workers counting grapes full-time. But he’s still conscious of the ethical implications of precision agriculture. In general, he believes it will do far more good than harm.
“My cousins, back in Australia, harvest thousands of acres of crops, just the two of them. And they can do that because they’re driving these big combine harvesters—which have been around since the 1930s. So technology has always had this effect on transitioning jobs from low skilled to high skilled. ... It’s been happening for thousands of years. And if some of the technology brings about a benefit to society and improves the economy, then, logically, the jobs will be improved.”
If all goes well during the upcoming summer trials, Nuske believes that within two years his vineyard robotic system will be on the market.
