First Large-Scale Robotic Object Discovery (LROD) Method Builds Robot "Memories" to Enable Personalized Assistance and Additional Services in the Home-Quality of Life Technology Center - Carnegie Mellon University

First Large-Scale Robotic Object Discovery (LROD) Method Builds Robot "Memories" to Enable Personalized Assistance and Additional Services in the Home

Outcome / Accomplishment:

For personal service robots like the Home Exploring Robot Butler (HERB) to successfully assist humans, they will need to operate autonomously in cluttered and variable human environments. For this reason, such robots also need to automatically discover and codify novel objects so that specific personal belongings and other household items can be recognized again in the future for task-based use.

By incorporating constraints and non-visual information - or "robotic metadata" - into the automatic processing of a robot's full day of raw video streams, QoLT PhD student  Alvaro Collet Romea (CMU-RI '12) has proven that robust object discovery is possible in a reasonable amount of time. His HerbDisc (for HERB Discovery), is a first example of an optimized implementation of automated large-scale robotic object discovery (LROD).

Impact / Benefit:

Collet's approach is a viable solution to the two main challenges in comptuer vision for robotics: robust performance in complex scenes and low latency for real-time operation. Collet's large-scale robotic object discovery method has proven capable of processing a six-hour 20 minute video stream of challenging human environments in under 19 minutes with a total of 283 never-before-seen objects discovered. Using LROD, the HERB robot discovered 2.7x more correct objects and processed the data 190x times faster than it had using visual information alone.

The HerbDisc example also provides a first step towards a lifelong robotic object perception framework that will allow a robot to recognize known objects and to discover unknown objects in any environment for as long as the robot operates. With HerbDisc, a personal robot can recount what it has seen over the course of the day and then continually grow or revise its catalog of objects in a house and how it can manipulate them. Such a catalog increases a personal robot's autonomous and independent operation, thereby allowing it to perform new and extended services for its human user. LROD and HerbDisc will allow personal service robots to interact with people, their belongings and their variable environments on a much more flexible and individualized basis.

Explanation / Background:

Large-Scale Robotic Object Discovery (LROD) addresses the challenges of scalability and robustness in the context of a robot's long-term (or life-long) operation.

3D data acquired by robots has been proven to peform well for recognition tasks. When used with natural and arbitrary constraints - such as 3D geometry, support surfaces, spatial and temporal traits - as well as other non-visual sensory information, a body of robotic metadata can be encoded as logic operators for service robots. Such robotic metadata remains accessible to a robot throughout its standard operation and can be used to reduce the computationally intensive tasks of new object discovery (such as indexing or matching partially visible objects.) As an emerging methodology, LROD relies on robotic metadata and incremental algorithms for object recognition to increase the speed and performance of a robot's ability to learn and remember new objects quickly during the course of its real-world applications.

LROD leverages machine perception, scene understanding, and cognitive processing and modeling to achieve its gains. The robot locates interesting structures in its daily video stream logs and then parses them against environmental information and other contextual cues. Parsing involves using constrained similarity graphs to group interesting structures together within scenes as similar sets of computationally efficient regions, filtering out irrelevant regions, and then selecting the remaining clusters that logically correspond to identifiable objects in image, 3D sensor or other big data and product-oriented databases. The resulting "discovered objects" can then be modeled for future recognition and manipulation tasks performed by the robot.

Because the discovery process is automated with LROD, known objects need not be continually updated off-line by the developer or user; instead,the robot simply discovers and learns "on the job." As such, the LROD approach removes the need for supervision by a trained, technical expert while dramatically reducing the need to deal with unique personal effects individually and exhaustively. LROD significantly increases the flexibility of robots in the home and other variable human environments.

Current and ongoing work will extend HerbDisc produced object model representations into the existing object recognition system, MOPED, as well as other object representations used for grasping and manipulation.

NSF Achievement Category:

Research and Technology Advancements

By: Kristen Sabol, ksabol @