Perception & Awareness-Quality of Life Technology Center - Carnegie Mellon University

Perception and Awareness Thrust (PAT)

Lead:  Martial Hebert, Carnegie Mellon

PAT is organized around three guiding principles for perception technology:

  • User Centric: Perception technology must concentrate on modeling the user, her intent, and interaction with the environment. This leads to new ways of designing sensory systems by emphasizing acquiring data from the user's perspective, for which we have coined the term "inside-out" perception.

  • Data-Driven Learning: Models used for extracting intent prediction from perceptual data must be learned from data acquired from the user and from external data. This leads to new research in designing learning algorithms that can support and utilize large data sets.

  • Multiple Data Sources: A QoLT perception system must be able to combine data from different sensors, including cameras, motion sensors, microphones, other wearable sensors, tactile sensors and direct interfaces to the brain and other parts of the nervous system. In addition, it must be able to combine user-specific data acquired on-line from the user's perspective with prior data acquired off-line from a large number of users, such as data acquired in instrumented environments. Specifically, it must be able to combine information that is specific to the user's cognitive behavior (e.g., as indicated by the typical order of actions he takes to execute a task) with prior knowledge (e.g., motion models from observations of a large population of people performing the same task).

Research in the PAT is organized around three main themes: 

  1. Sensing technology that is specifically designed to acquire the perceptual data that are most directly relevant to the ultimate goal of understanding user intent

  2. Data-driven perception technology for identifying objects, places and events in perceptual data

  3. Learning techniques for extracting models from very large data sets that combine data acquired from the user's perspective combined with prior data from external sources, and for generating predictions from observations. These three themes are mapped onto projects that correspond to a natural progression from data, to intermediate interpretation from sensor data (objects, places, and events), to high-level interpretation (activity association and intent prediction)