Agricultural equipment is involved in a significant number of accidents each year, often resulting in serious injuries or death. Most of these accidents are due to operator error, and could be prevented if the operator could be warned about hazards in the vehicle’s path or operating environment.

NREC developed a perception system that provides that early warning.

The system uses multiple sensing modalities (color, infrared and range data) that can adapt easily to the different environments and operating conditions to which agricultural equipment is exposed. The system detects obstacles and hazards based on color and infrared imagery, together with range data from laser range finders. These sensing modalities are complementary and have different failure modes. By fusing the information produced by all the sensors, it improves the robustness of the overall system beyond the capabilities of individual perception sensors.

Perception Software
NREC implemented feature extractors that analyze the images in real time and extract color, texture and infrared information that is combined with the range estimates from the laser in order to build accurate maps of the operating environment of the system.

Machine Learning
NREC developed machine learning for classifying the area around the vehicle in several different classes of interest such as obstacle vs. non-obstacle or solid vs. compressible. Novel algorithms were developed for incorporating smoothness constraints in the process of estimating the height of the weight supporting surface in the presence of vegetation, and for efficiently training our learning algorithms from very large data sets.