Human-AI Complementarity for Decision Making
Academic Workshop - September 25-26, 2025
The NSF AI Institute for Societal Decision Making (NSF AI-SDM) sponsors the participation of selected speakers and students in an annual workshop of Human-AI Complementarity for Decision Making. Human-AI Complementarity, defined as the condition in which Humans + AI working together results in better decisions than humans or AI working alone, is a broad goal pursued in several projects of the NSF AI-SDM.
In 2025, we will focus on how to create flexible Human-AI teams to achieve complementarity. This theme refers to the interdisciplinary study of how to design and deploy AI systems in ways that are dynamically aligned with human values, robust to unexpected behavior, and safe even under failure modes. It encompasses short-term concerns about deployed systems (e.g., fairness, robustness, interpretability, and misuse) and long-term concerns about advanced general AI (AGI) that could have large-scale societal impacts if not aligned with human interests.
The goals of the workshop are:
- To deliver state of the art instruction on desirable ideas to achieve Human-AI complementarity for decision making.
- To generate common knowledge about pressing research challenges
- To generate new shared ideas to address these challenges in future research
We aim for concrete outcomes spawned by the collective AIdeas of the participants in the workshop including: specific challenges that need to be addressed to achieve complementarity in flexible Human-AI Teams and concrete proposals to address those challenges.
Venue
The workshop will take place September 25-26, 2025 at the Cohon University Center, located on Carnegie Mellon University's campus. Workshop activities will be held on the second floor of the University Center, 5032 Forbes Ave, Pittsburgh, PA 15213.
Click here for a complete list of workshop participants
| Thursday, September 25th | |
|---|---|
| Breakfast 7:45-8:15 |
Cohon University Center - 2nd Floor |
| Introduction 8:15-8:30 |
Introductions and Goals Rangos 1 & 2 Coty Gonzalez & Aarti Singh |
| Tutorials 8:30-11:30 |
Session 1: Parallel Tutorials Tutorial Track A Tutorial on LLM & Agent Alignment: Vulnerabilities, Detection, and Mitigation McKenna, Peter, Wright Rooms Ahmad Beirami, Google Deepmind Hamad Hassani, University of Pennsylvania Tutorial Track B Modeling and Measuring Human Decisions: From Cognitive Theories to Data Collection Practices Rangos 1 & 2 Ngoc Nguyen, University of Dayton Coty Gonzalez, Carnegie Mellon University Stephanie Eckman, University of Maryland Frauke Kreuter, University of Maryland |
|
Coffee Break |
Coffee Rangos 3 |
|
Poster Spotlights |
Session 2: Spotlight Lightning Talks (Group 1) Rangos 1 & 2 9x 2 minute presentations |
| Lunch and Poster Session 12:15-1:15 |
Lunch and Poster Session Rangos 2 |
|
Plenary I |
Session 3: AI Challenges Rangos 1 & 2 4x 15 minute presentations + discussion Jessica Hullman, Northwestern University Justin Weisz, IBM Research Jon Bogard, Washington University in St. Louis Sunnie S.Y. Kim, Apple |
| Coffee Break 2:30-2:45 |
Coffee Rangos 3 |
| Plenary II 2:45-4:00 |
Session 4: AI Alignment Rangos 1 & 2 4x 15 minute presentations +discussion Benjamin Laufer, Cornell University Guojun Xiong, Harvard University Woody Zhu, Carnegie Mellon University Wilka Carvalho, Harvard University |
| Coffee Break 4:00-4:15 |
Coffee Rangos 3 |
| Plenary III 4:15-5:30 |
Session 5: Human-AI Teaming |
| Networking & Dinner 5:30-8:00 |
Dinner Rangos 3 |
| Friday, September 26th | |
|---|---|
| Breakfast 8:30-9:00 |
Cohon University Center - 2nd Floor |
| Introduction 9:00-9:15 |
Overview and Goal Setting Rangos 1 & 2 Coty Gonzalez & Aarti Singh |
| AIdea Exchange Rotation 9:15-11:00 |
Session 6: AIdea Exchange Rotation Group 1: Human-AI collaboration in high-stakes, uncertain environments Led by Terri Adams, Howard University Bryan Wilder, Carnegie Mellon University Rangos 1 & 2 Group 2: Measuring and improving human-AI complementarity Led by Paul Lehner, MITRE Corporation Ken Holstein, Carnegie Mellon University Rangos 3 Group 3: Trust, calibration, and appropriate reliance Led by Michael Lee, University of California, Irvine Valerie Chen, Carnegie Mellon University McKenna Room Group 4: Societal and ethical alignment of AI in decision making Led by Atoosa Kasirzadeh, Carnegie Mellon University Vincent Conitzer, Carnegie Mellon University Wright Room Group 5: Metrics and benchmarks beyond accuracy Led by Sherry Tongshuang Wu, Carnegie Mellon University Frauke Kreuter, University of Maryland Pake Room Group 6: Multi-party and group decision making with AI Led by Anita Woolley, Carnegie Mellon University Aarti Singh, Carnegie Mellon University Class of 1987 Room |
| Coffee Break 11:00-11:30 |
Coffee |
|
Poster Spotlights |
Session 7: Spotlight Lightning Talks (Group 2) Rangos 1 & 2 9x 2 minute presentations |
| Lunch and Poster Session 12:00-1:00 |
Lunch and Poster Session Rangos 2 Industry highlights by Unanimous AI and Bosch Research |
| AIdea Writeup 1:00-2:30 |
Session 8: Parallel Writing Groups Group 1: Human-AI collaboration in high-stakes, uncertain environments Led by Terri Adams, Howard University Bryan Wilder, Carnegie Mellon University McKenna Room Group 2: Measuring and improving human-AI complementarity Led by Paul Lehner, MITRE Corporation Ken Holstein, Carnegie Mellon University Peter Room Group 3: Trust, calibration, and appropriate reliance Led by Michael Lee, University of California, Irvine Valerie Chen, Carnegie Mellon University Wright Room Group 4: Societal and ethical alignment of AI in decision making Led by Atoosa Kasirzadeh, Carnegie Mellon University Vincent Conitzer, Carnegie Mellon University Dowd Room Group 5: Metrics and benchmarks beyond accuracy Led by Sherry Tongshuang Wu, Carnegie Mellon University Frauke Kreuter, University of Maryland Pake Room Group 6: Multi-party and group decision making with AI Led by Anita Woolley, Carnegie Mellon University Aarti Singh, Carnegie Mellon University Class of 1987 Room |
| Coffee Break 2:30-2:45 |
Coffee |
| AIdea Pitch 2:45-4:15 |
Session 9: Report Out |
| Wrap-Up 4:15-5:00 |
Deliver Written Summaries to Coty/Aarti |
| Departure |
|
Travel Information
How to travel to and around Pittsburgh:
- By plane via Pittsburgh International Airport. - Pittsburgh Regional Transit offers public transit service to and from Pittsburgh International Airport via the 28X Airport Flyer. The 28X route serves Pittsburgh International Airport, Downtown Pittsburgh, and Oakland seven days a week. Riders using cash to pay their transit fares must have exact change; credit cards are not accepted on vehicles. Credit cards are accepted at the ticket vending machine in Baggage Claim inside Door #2. The ticket vending machine allows riders to buy daily, 7-day or 30-day tickets, or add stored value onto a ConnectCard or ConnecTix.
- By train via Amtrak. - Amtrak's Union Station is located at 1100 Liberty Avenue, Pittsburgh, PA 15222. Numerous Pittsburgh Regional Transit buses run from Downtown to CMU's campus in Oakland.
- By long distance bus via Greyhound. - The Greyhound Bus Terminal is located five miles from campus in downtown Pittsburgh. The station terminal is located at the intersection of 11th Street and Liberty Avenue
- Local bus via Pittsburgh Regional Transport. - The PRT network offers bus, light rail, and incline services in Allegheny County. Please see their website for schedules and rider information.
- Visitor parking is available at the East Campus Garage, 5040 Forbes Ave, Pittsburgh, PA 15213. For more information about visitor parking, including rates, please visit CMU's visitor information page.
The following hotels are located approximately 1 mile from CMU's campus.
- Wyndham Pittsburgh University Center
100 Lytton Ave, Pittsburgh, PA 15213
+1 412-682-6200
www.wyndhamhotels.com - The Oaklander Hotel, Autograph Collection
5130 Bigelow Blvd, Pittsburgh, PA 15213
+1 412-578-8500
www.marriott.com - Hilton Garden Inn Pittsburgh University Place
3454 Forbes Ave, Pittsburgh, PA 15213
+1 412-683-2040
www.hilton.com - Residence Inn by Marriott Pittsburgh Oakland/University Place
3341 Forbes Ave, Pittsburgh, PA 15213
+1 412-621-5600
www.marriott.com - Hampton Inn Pittsburgh University/Medical Center
3315 Hamlet St, Pittsburgh, PA 15213
+1 412-681-1000
www.hilton.com
Numerous cultural sites, entertainment options, and historical landmarks are located near Carnegie Mellon University. We have marked several examples of each, as well as university locations and travel hubs, on the map below.
This project is supported by the U.S. National Science Foundation under award number 2229881. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. National Science Foundation. More information on the National AI Research Institutes can be found at aiinstitutes.org.