Carnegie Mellon University

Enhanced Teleoperation

NREC developed Enhanced Teleoperation: a real-time 3D video system to improve situation awareness in teleoperation and indirect driving of unmanned ground vehicles in complex environment.

Teleoperating a vehicle without line-of-sight is a very difficult and tedious task. Many factors contribute to the degradation of performance: narrow field of view of cameras, lack of 3D perception, jitter in images due to vibration, etc. Furthermore, many operators suffer from various levels of motion sickness when operating from another vehicle.

To improve teleoperation, our team developed the Soldier Awareness through Colorized Ranging (SACR) to provide a 3D environment with photo-realistic computer graphic models of the surroundings, all in real-time. These improvements give drivers a better view of a vehicle’s surroundings, improving their awareness of its environment and making remote and indirect driving safer, easier, and faster.  Other potential SACR applications include mapping, mission visualization, mission rehearsal, and localization of personnel and vehicles. In field trials, operators performed 20% to 50% better on a range of driving tasks with SACR than they did with existing 2D video systems.

SACR uses 3D video to improve a driver’s awareness of the environment. The SACR sensor pod includes a high-definition video camera and laser range finder. One or more sensor pods can be mounted on a vehicle. SACR fuses video and range input from the sensor pods in real time to build a 3D computer graphics model of the vehicle’s surroundings.

Features: 
  • A geometrically correct “virtual driver’s windshield” can be shifted to different points around the vehicle. This makes it easier to view the outside the vehicle and compensates for a driver’s offset from the sensor’s physical location.
  • Widening the field of view allows the driver to see more of the vehicle’s surroundings at a glance.
  • Drivers can zoom into and out of images to better see points of interest and even “fly around” the outside of the vehicle. An overhead view is especially useful for teleoperating UGVs (similar to driving a radio-controlled toy car).
  • Video memory allows the camera to “see through” the vehicle, showing areas that would ordinarily be blocked by parts of the vehicle. This is especially useful for seeing the ground immediately in front of, underneath, or in back of the vehicle – none of which are easily observable from fixed-mount cameras.
  • One 3D video feed can produce multiple views for multiple operators.
  • Maps, autonomy plots, and other mission-critical information can be superimposed on the 3D video image.

Photos

NREC’s miniaturized Situational Awareness Through Colorized Ranging (SACR) system fuses video and range data from a small panoramic camera ring and scanning LADAR sensor
NREC’s miniaturized Situational Awareness Through Colorized Ranging (SACR) system fuses video and range data from a small panoramic camera ring and scanning LADAR sensor
NREC’s miniaturized Situational Awareness Through Colorized Ranging (SACR) system fuses video and range data from a small panoramic camera ring and scanning LADAR sensor