Sensing-Quality of Life Technology Center - Carnegie Mellon University


The goal of the project is to develop new approaches to sensing, such as “inside-out” vision in which the system experiences the environment and the user’s behavior from the user’s point of view, thus facilitating the analysis of her behavior and intentions.

We’ve identified two key differentiators between the QoLT scenarios and other applications of computer perception (such as robotics or surveillance) are that:

  1. The perception system should capture as much as possible the experience of the user from her perspective.  This is termed inside-out” or “first-person” perception.  This is appropriate to the type of scenarios envisioned in QoLT because it provides a user-centric view of interpretation.
  2. For many scenarios, the perception system should be designed so as to minimize the need for special instrumentation of the user’s environment. In this project, we address this requirement explicitly by developing sensors that will be used to observe continuously the user’s activities and environment.
A key obstacle to the effective use of machine perception, especially vision, is that the sensor package needs to be in a form factor that can be accepted by the users while, at the same time, including the computation and storage capacity needed for the perception techniques. Recent sensor and computer technologies provide novel solutions. We teamed with an industry partner to best address this issue by developing prototype wearable sensors suitable for the QoLT scenarios. After completion of these initial prototypes, our plan is to work with design researchers and with users and practitioners (through the HSIT and PST thrusts) to converge to several designs with which we can conduct user experimentation.

A key hypothesis is that:

  1. In contrast to other applications such as surveillance, monitoring, and robotics, sensor data from the first person perspective (termed “inside-out” perception) is most appropriate to the type of scenarios envisioned in QoLT, in which a more user-centric view of perception is necessary.  The perception system should capture as much as possible the experience of the user from her perspective.
  2. Corresponding sensing devices can be built within the span of the project with the desired form factor for acceptance and with the level of computation power and storage needed for our algorithms. Two actions were taken this year to validate this hypothesis:
    • we teamed with an industry partner, Seagate Research, to guarantee access to the best possible technology and designs, in particular in the area of storage; and
    • we biased the development of the algorithms in the perception thrust toward using inside-out perception data in order to validate their feasibility and performance.

    We started by building simple, early lab prototypes, for the express purpose to collect data to be processed by our research algorithms. With this approach, we are able to assess the idea of inside-out sensing and to validate that the algorithms include the innovations necessary to deal with this type of data, without waiting for the actual, final devices to be ready. At the same time, with Seagate Research, we have designed a sensor package which includes high resolution, miniature cameras, and motion sensors (accelerometers and gyroscopes). This design will be the basis for our first prototypes. This initial design is based on a head-mounted concept. However, we plan to work with design experts and with the PST thrust to develop other concepts based on user feedback. Also, while we have emphasized the use of vision and its integration with motion sensors in a single package, our design supports specifically interfacing with other devices, most notably those, such as the “E-watch”, that are used in the HIST and Virtual Coach areas of QoLT.

    Project Team

    • Martial Hebert, Lead
    • Mark Baskinger
    • Takeo Kanade
    • Hong-Wen Kang
    • Ki Ho Kwak
    • Asim Smailagic

    Inside-out sensing device

    Inside-out sensing device 

     Inside-out sensing device