The goal of this project is to build computer software that is able to automatically detect when a user is having difficulty using a computer and automatically deploy software to assist them.
Computer technology has become an integral component in people’s lives for employment, recreation and socializing. Unfortunately, computers are not universally accessible and there is a growing population of people who are motivated to use computers, but find it difficult to do so. Many of these individuals lack access to experts who can advise them how to make their computers more accessible, and even if they have access, their needs may change from day to day or over time. Our solution is to build software that is able to automatically detect what sort of difficulties an individual is having using a computer, and automatically deploy software to assist them. Our approach involves gathering longitudinal-real world data about desktop computer use (specifically, pointing and targeting) and develop models that can predict what adaptations are needed.
A fundamental difficulty in addressing accessibility problems is creating systems that can function in real-world environments and on real-world data. This is confounded by the fact that many computer accessibility researchers find that they have limited access to subjects and that gathering significant amounts of data is difficult due to issues such as fatigue and computer access. An important component of our work is addressing this problem by developing techniques for gathering extensive real-world data that can be analyzed at the same level as controlled laboratory data. Our approach is to gather far more data than we need by creating software that can run in the background during typical computer use. We then identify the subset of that data is sufficiently well defined to be analyzed and compared using statistical techniques such as machine learning (we call these chunks of data “informative moments”). An example problem we are able to tackle because of our focus on real-world data is understanding how pointing performance differs in the real world and the laboratory.
The ultimate goal of our work is to increase personal agency by automating the deployment of adaptations that increase access to online resources such as job opportunities and information that can increase quality of life and independence. The problem of automating assessment is a general one applicable to other QoLT virtual coaches, which must deal with changing user needs. Our data set can also enable other researchers within QoLT and more broadly to explore the potential of possible adaptations in the context of a rich, real-world, longitudinal set of data. The primary thrust of this project is Human-System Interaction, but our modeling of human action is relevant to Perception and Awareness. Also our human-centered methods make this project relevant to Person and Society.