Computer Access Coach-Quality of Life Technology Center - Carnegie Mellon University

Computer Access Coach

The goal of this project is to demonstrate a computer dynamically adjusting to changes in human ability. One of the main reasons computers are inaccessible is that they treat all users the same way, and usually do not know much about each user.  This is particularly unfortunate given the amount of accessibility software that comes pre-installed on most computers or is available online including pointer configuration utilities, screen magnifiers, and onscreen keyboards.  Not only do most users not know what technology they need to make a computer accessible, insurance limitations and high cost can prevent them from having an assistive technology clinician assess their ability.  As a result, it is not uncommon for people to end up using either no accessibility tools, or tools that are not suited for their pointing abilities.

This project is a virtual coach, which is one of the Center’s families of engineered systems. This project will enable increased access to online resources such as job opportunities and information that can increase quality of life and independence.

This project pushes machine learning in a direction that is difficult and unsolved—namely, how to help both the user and the underlying algorithms deal with the inevitable messiness introduced by including a human in the loop. What happens when the user changes? What about when those changes are due to the system? How should they be interpreted? How does the user recover from errors and help the system understand when errors occur? How does the system differentiate between errors that are caused by an incorrect diagnoses or adaptation choice rather than a change in the user’s abilities? We believe that it is only through human computer interaction methods, namely studying how the user reacts to different possible solutions, that the best answers to these question scan be found.

Any foray into this area requires first a working dynamic system that can react to the user and diagnose interaction difficulties. Thus, most of the problems described above are second order problems. The issues of dynamism and diagnosis are difficult issues in their own right, and they are the focus of the initial work on this project. In particular, we are exploring the relative benefits of machine learning and heuristic algorithms for supporting diagnosis of user ability.



Project Team

  • Jennifer Mankoff, Lead
  • Scott Hudson
  • Rich Simpson
  • Amy Hurst

Magnetic dust marks previous target locations and enhance snapping to the appropriate application key

Computer Access Coach graphic

Magnifying glass effect in vicinity of cursor to assist the visually impaired 

Computer Access Coach graphic