SEI Tool Helps Federal Agencies Detect AI Bias and Build Trust
Media Inquiries
As artificial intelligence becomes more central to national security — powering logistics, mission planning, intelligence and cybersecurity — ensuring these systems are trustworthy is critical. When AI systems fail, the consequences can affect security, resources and even lives.
To meet this challenge, Carnegie Mellon University’s Software Engineering Institute(opens in new window) has developed the AI robustness (AIR)(opens in new window) tool, a free, open-source platform that helps agencies uncover why an AI system may produce biased or unreliable results. Unlike conventional methods which spot surface-level patterns in data, AIR shows whether those patterns are actually cause and effect.
“Powerful AI and machine learning (ML) tools are revolutionizing fields of prediction, automation, cybersecurity, intelligence gathering, training and simulation, object detection and more. Yet we know there are weaknesses associated with these tools that we must consider,” said Anita Carleton(opens in new window), director of the SEI Software Solutions Division(opens in new window). “The AIR tool offers insight into not only where AI might go astray, but also why it happens.”
Confidence through cause, not correlation
AI and ML classifiers are powerful tools for prediction and automation, but rely heavily on correlations in data. When conditions shift — from new threats, evolving data or an unexpected scenario that the AI wasn’t trained on — those correlations can break down. They undermine AI systems by adding bias, reducing robustness and eroding trust.
The AIR tool provides a new method to test and evaluate classifiers. By applying methods from causal discovery and causal inference — techniques for finding cause and effect in data — AIR allows users to assess various AI and ML classifications and predictions with more nuance. As a result, users can have greater confidence that models are less biased, more dependable in guiding performance and more transparent in explaining outliers and causes.
“The AIR tool fills a critical void,” said Linda Parker Gates(opens in new window), AIR project principal investigator. “It uses causal learning to improve the correctness of AI classifications and predictions that are largely dependent on data correlation. Its effect is to increase confidence in the use of AI in development, testing and operations decision-making,”
Supporting federal partners
The work aligns with recent guidance(opens in new window) from the U.S. Office of Management and Budget, which calls on federal agencies to conduct ongoing testing and human review of AI systems. The AIR tool helps agencies meet this critical need by determining the causes of adverse impacts of AI classifiers and offering a transparent approach to continuous evaluation.
The SEI is currently seeking Department of Defense partners to use and provide feedback on this technology. Collaborators will partner with SEI researchers to test their systems, gaining actionable insights into classifier health while advancing the state of trustworthy AI for national security.
