World Economic Forum Young Scientist

Morency's Research Makes MultiSense for Doctors

By Susie Cribbs / cribbs@cs.cmu.edu

A patient walks into a doctor’s office, complaining of a sore throat. After an initial exam, the doctor orders a culture and a few blood tests that she uses — along with the exam results and professional judgment — to make a fairly objective diagnosis.

But what about patients struggling with mental illnesses? There’s no blood test for depression or anxiety, and diagnosis is often based on more subjective criteria.

Language Technologies Institute Assistant Professor Louis-Philippe Morency wants to change that. His primary method? Building algorithms and programs that not only understand how people communicate, but that also allow computers to understand subtle human behaviors during social interactions.

“I want to understand how people express themselves through language and gestures, and I want to understand them together,” he said. “There’s a lot of great work in analyzing speech and gesture, but there’s been little work that puts them together.”

Morency’s research into these multimodal algorithms has taken the form of MultiSense, the sensing and analysis technologies designed to help clinicians assess psychological distress. Outfitted with a simple webcam, the technology can automatically measure patient’s facial expressions, gaze directions and voice quality. This data is then provided to doctors, who can use it as a more objective way to identify depression and anxiety.

The research behind MultiSense covers a wide range of topics, including multimodal interaction, social psychology, computer vision, machine learning and artificial intelligence. This natural inclination toward multidisciplinarity is what brought him to Carnegie Mellon.

“The Language Technologies Institute (LTI) is the perfect place for me to forward my goal for building multimodal algorithms that bring together speech and gestures — like facial expressions, posture and reciprocity — because of its established expertise in speech and language,” he said.

Morency’s work in speech and gesture analysis has also put him on the international stage. Last month, the World Economic Forum recognized him as one of this year’s class of Young Scientists — scientists under the age of 40 who are advancing the frontiers of science, engineering or technology in areas of high societal impact.

“As part of this award, I had the chance to attend the summer meeting of the World Economic Forum,” Morency said. “It was a great opportunity to better understand the global view of scientific research and how to interact with policymakers.”

And having that understanding may be incredibly important, since his work could change the face of mental health care forever.

Pictured: Louis-Philippe Morency presents at the World Economic Forum.