Explainable AI: What Happens Inside the Black Box
By Scottie Barsotti
Carnegie Mellon University Professor Leman Akoglu specializes in anomaly detection models. The Heinz College of Information Systems and Public Policy faculty member is an expert on machine learning and data mining and uses that knowledge to explain how algorithms make decisions.
A successful anomaly detection model will comb through transactions on those reports and flag items that seem out of place so they can be investigated. Anomaly detection can apply to many domains and in some cases directly impact people's lives, such as alerting a social worker when a report of child abuse significantly stands out from the others, or when data from emergency rooms and social media might indicate a potential disease outbreak or societal unrest.
Akoglu's work in this area spans many topics, from identifying emerging news that could constitute risk for corporate partners, to identifying fraudulent users and fabricated reviews on sites like Yelp and TripAdvisor.
But it is not always enough to know that an anomaly exists, Akoglu said. Humans using the outcomes of detection models must understand what the anomaly means.
"If an auditor is looking at an insurance claim or expense report detected by an algorithm as anomalous, they cannot simply say 'this is anomalous, so we are not paying it.' They have to investigate, verify and specifically pinpoint the misinformation in the report if any. Similarly, if a user account is flagged as anomalous by a detector, the system administrator would not simply want to shut down the account, but rather investigate and validate the suspicious behavior if any. In such cases it is beneficial to be able to tell why something is anomalous," she said. "If the algorithm can tell not only that there is a potential error or suspicious activity, but explain why it thinks that, and how the anomalies stand out, the human analyst can make sense of the situation and look into it."
Explanations are essential particularly for these types of anomaly detection scenarios, which cannot be fully automated and require a human in-the-loop (like an auditor) for verification. However, explanations can help beyond sense-making. In some cases, they may bring up issues with the detection algorithm itself, by exposing reliance on unexpected or undesirable clues for detection purposes - for example, claiming terrorist activity based on someone's nationality or biometric data.
The problem? Many algorithms exist in what's called "black boxes," meaning the human beings who use the algorithm may know what its outcome was, but not necessarily how it reached that determination.
Akoglu said explanations are a step toward algorithmic transparency, a societal issue that touches many disciplines and is of great concern to many at Heinz College and the new Block Center for Technology and Society.
Akoglu said a better explanation relies in part on the creation of "interpretable rules."
"We can determine what are the distinctive characteristics of an anomaly, and determine instances that exist that are similar to an anomalous instance and yet are not anomalous," Akoglu said. "While the former shows how an anomaly stands out from the rest, the latter outlines how far it is from being considered a non-anomaly in an interpretable way."
This could be useful for reducing false positives (when a decision model predicts an outcome that is unlikely to occur) and false negatives (when an outcome occurs after being classified by a model as unlikely). It could also give us insight into why an algorithm took a specific action - or didn't take a specific action - in contrast to what a human being might have taken in the same situation with the same input.
In some contexts, this can directly benefit consumers.
"Say a mortgage application is marked for rejection by an algorithm, the consumer can be given useful information about why they were tagged that way and what they could seek to change to get a better outcome," Akoglu said.
The inner-workings of machine learning algorithms can be opaque, which leads to many ethical concerns. However, Akoglu said she believes that "explainable AI" is both possible and desirable. Devising clearer explanation frameworks for algorithms will provide those using these algorithms and those affected by the outcomes of the algorithms with better information, and could improve trust in these technologies over time.
"We have to trust that our systems are truly doing things that are logical in their context, and that can be understood by a human at the end of the process," Akoglu said.