CMU Researchers Make Transformational AI Seem "Unremarkable"
AI must be unobtrusive to be accepted as part of clinical decision making
Physicians making life and death decisions about organ transplants, cancer treatments or heart surgeries typically don't give much thought to how artificial intelligence might help them. And that's how researchers at Carnegie Mellon University say clinical AI tools should be designed — so doctors don't need to think about them.
A surgeon might never feel the need to ask an AI for advice, much less allow an AI make a clinical decision for them, said John Zimmerman, professor in CMU's Human-Computer Interaction Institute (HCII). But an AI might guide decisions if it was embedded in the decision-making routines already used by the clinical team, providing AI-generated predictions and evaluations as part of the overall mix of information.
Zimmerman and his colleagues call this approach "Unremarkable AI."
"The idea is that AI should be unremarkable in the sense that you don't have to think about it and it doesn't get in the way," Zimmerman added. "Electricity is completely unremarkable until you don't have it."
Qian Yang, a Ph.D. student in HCII, will address how the Unremarkable AI approach guided the design of a clinical decision support tool (DST) at CHI 2019, the Association for Computing Machinery's Conference on Human Factors in Computing Systems, May 4-9 in Glasgow, UK.
Yang, along with Zimmerman and Aaron Steinfeld, associate research professor in HCII and the Robotics Institute, are working with biomedical researchers at Cornell University and CMU's Language Technologies Institute on a DST to help physicians evaluate heart patients for treatment with a ventricular assist device (VAD). This implantable pump aids diseased hearts in patients who are unable to receive heart transplants. Many recipients die shortly after the implant. The DST under development uses machine learning methods to analyze thousands of cases and calculate a probability of whether an individual might benefit.
DSTs have been developed to help diagnose or plan treatment for a number of medical conditions and surgical procedures, but most fail to make the transition from lab to clinical practice and fall into disuse.
"They all assume you know you need help," Zimmerman said. They often face resistance from physicians, many of whom don't think they need help, or see the DST as technology designed to replace them.
Yang used the Unremarkable AI principles to design how the clinical team would interact with the DST for VADs. These teams include mid-level clinicians, such as nurse practitioners, social workers and VAD coordinators, who routinely use computers, and surgeons and cardiologists, who value their colleagues' advice over computational support.
The natural time to incorporate the DST's prognostications is during multidisciplinary patient evaluation meetings, Yang said. Though physicians make the ultimate decision about when or if to implant a VAD, the entire team is often present at these meetings and computers are being used.
Her design automatically incorporates the DST prognostications into the slides prepared for each patient. In most cases, the DST information won't be significant, Steinfeld suggested, but for certain patients, or at certain critical points for each patient, the DST might provide information that demands attention.
Though the DST itself is still under development, the researchers tested this interaction design at three hospitals that perform VAD surgery, with DST-enhanced slides presented for simulated patients.
"The mid-levels — the support staff — loved this," Yang said, because it enhanced their input and helped them be more active in the discussion. Physician reaction was less enthusiastic, reflecting skepticism about DSTs and the conviction that it was impossible to fully evaluate the interaction without a fully functioning system and real patients.
But Yang said physicians didn't display the same defensiveness and feelings about being replaced by technology typically associated with DSTs. They also acknowledged that the DST might inform their decisions.
"Prior systems were all about telling you what to do," Zimmerman said. "We're not replacing human judgment; we're trying to give humans inhuman abilities."
"And to do that we need to maintain the human decision-making process," Steinfeld added.
The National Heart, Lung and Blood Institute and the CMU Center for Machine Learning and Health supported this research.