Carnegie Mellon University

AI

December 04, 2018

Faculty Spotlight: Can People Teach a Self-Driving Car to be Safe?

In the classic film “2001: A Space Odyssey,” HAL 9000 was the artificial intelligence that controlled a spacecraft with deadly consequences. Fifty years later, the introduction of AI to vehicles such as self-driving cars raises a new ethical question: can a modern day HAL 9000 make better decisions than the programmers who programmed it?

A recent article published in Nature suggests that individual preferences can be used to determine the moral values used to program self-driving cars, raising ethical dilemmas. Should a car that must hit a person choose a senior citizen over an infant? Or should it choose to spare five passengers inside over one pedestrian on the street?

But new research coauthored by Tae Wan Kim, Associate Professor of Business Ethics at the Tepper School of Business, suggests that the methodology often used to evaluate AI ethics — including that in Nature — is flawed.

Kim’s research, which is coauthored with John Hooker, T. Jerome Holleran Professor of Business Ethics and Social Responsibility and Professor of Operations Research at the Tepper School, as well as Thomas Donaldson of the University of Pennsylvania, instead proposes a more objective method for assessing ethics, one that quantifies the concept’s application to an eminently quantifiable field. 

The task, according to Kim, is not to program the car to value some lives over others; rather, the task is to program the car to not hit any humans.

The evaluation issue in contest is called “value alignment” — a principal concept in machine learning. Value alignment refers to the need for machines to adapt to human standards. By placing too much emphasis on how well a machine can mimic human preferences, some research neglects whether those preferences are actually ethical, Kim notes. “Much of the existing research focuses on the ‘alignment’ part and not the value,” he explains. 

Value alignment typically falls into two categories: mimetic and anchored. A machine using mimetic value alignment mimics human behavior through things such as surveys or interaction, which can lead to unintended problems. For example, back in 2016 Microsoft created an infamous Twitter bot called Tay that was taught to engage with people through tweets. When people sent Tay racist and misogynistic tweets, the bot responded in kind, because it was mimicking the behavior it was given.

Likewise, in October Amazon had to halt an AI project that sought to review job applicants’ resumes after realizing that because it was gathering most of its data from males, it was creating a bias that worked against female applicants.

The problem with mimetic value alignment is that it confuses opinions or preferences with ethics, says Kim. Using the mimetic model, if a large number of people said murder was acceptable, the machine would align. But that still doesn’t mean it is behaving ethically.

The second category of value alignment, which Kim and his coauthors prefer, is anchored. That means the machine’s behavior is tied directly to specific values, such as honesty, fairness, health, happiness, or environmental integrity, which can be expressed mathematically. If that framework includes a mimetic component, it’s considered a hybrid.  

Kim notes that an anchored or hybrid form of value alignment can help machines avoid some of the problems inherent in the mimetic model, bringing them more closely in line with the best version of the human beings they are built to imitate.

“When I started my Ph.D. in the early 2000s, business ethics meant finance ethics. Most talented MBAs got a job on Wall Street. Now, business ethics means tech, and most talented MBAs get jobs in Silicon Valley — so business ethics means tech ethics,” said Kim.

As the world evolves to include more and more robots as members of its workforce, it is critical to instill ethical decision-making skills into our collective toolbox. AI is defining the new frontier in ethics research and more work needs to be done to ensure we avoid mimicking the HAL 9000 doomsday scenario.