A Pioneer of Computing for Imagination and Social Impact

D. Fox Harrell (DC 1998) combines his passion for the arts and computation to take AI to the next level and affect social change

As a kid, D. Fox Harrell spent his summers computer programming, but he also enjoyed writing poetry, making artwork, reading fiction and playing in a punk rock band with his friends. Even from an early age, Harrell knew that each of his eclectic interests was worth pursuing, so he began envisioning how to combine them all through computing and the arts.

Today, Harrell continues to combine seemingly disparate fields —computer science, philosophy, psychology, cultural studies, media arts and more — to explore the relationship between imagination and computation. In his research, theoretical writing and computational practice, Harrell is at the forefront of a new way of thinking about computation. His work centers around the ethical development and application of computing and artificial intelligence research to create systems that are engaging, dynamic and foster positive impact on critical social issues. 

Harrell is a professor of Digital Media and Artificial Intelligence in the Comparative Media Studies Program and Computer Science and Artificial Intelligence Laboratory (CSAIL) at MIT. He is also the founding director, studio head and principal investigator of MIT’s Center for Advanced Virtuality, a studio, laboratory and research center that pioneers innovative experiences that use technologies, such as virtual reality, augmented reality and synthetic media.

Harrell’s undergraduate experience at CMU was early proof that crossing disciplinary fields could yield new ways of thinking about and creating computational experiences. After seeing a brochure for CMU’s Pre-College Program at a friend’s house, he applied and was admitted to the music program on the basis of a demo tape. During his summer at CMU, he collected brochures for as many university programs as he could find, and he met with Wilfred Sieg, then the department head for Logic and Computation. He decided to come to CMU as an undergraduate, earning a Bachelor of Fine Arts degree with a focus on electronic and time-based media, a Bachelor of Science in Logic and Computation and a minor in computer science.

From the first-year rigors of a drawing class taught by Herb Olds and concept studio class with Andrew Johnson to a senior year animation course with James Duesing, his coursework had a transformative impact on his future career. In particular, an art theory course taught by Steve Kurtz moved Harrel. “The way the course integrated the philosophy of art, cultural studies, critical theory and more brought together my interest in work for a societal impact with my interest in art,” he said.

At CMU, Harrell also honed his interest in AI and found that bringing together the arts and AI opened up new possibilities for storytelling. For instance, he found Manuela Veloso’s AI course and Tom Mitchell’s Machine Learning courses illuminating. He also found Jeremy Avigad’s course on logic and computability offered a brilliant approach to mathematical formalization. As opposed to a static medium, AI enables creators to reconfigure stories in terms of social perspective, cultural metaphor, values and styles, allowing for the possibility of creating flexible experiences that address a wide variety of viewpoints. 

According to Harrell, many current forms of AI, largely developed for commercial use, do not integrate flexibility and responsiveness to diverse needs yet. For example, a recommender system is designed only with a utilitarian aim in mind: to give people exactly what they want. While these systems originate from the idea of ‘personalization,’ they ultimately build consensus through categorizing people and creating self-reinforcing loops. People who fall outside of the group consensus (i.e., those with more idiosyncratic tastes) are left out. Many of the same biases that exist in the physical world are too often replicated, or even amplified, by this type of system.

Harrell designs systems that are more akin to how an improvisational performer might interact with an audience. Rather than deliver exactly want a person might want, the system can also deliver what a person might need. This is where art plays a crucial role in his work: art has a special ability to “provoke, challenge, offer new ways to think, spark imagination and more,” he said. To accomplish this technically, Harrell invented his own platform, called Chimera that he and his team have been refining ever since.

 “The [R]Evolution of Hip Hop Breakbeat Narratives,” a collaboration between the MIT Center for Advanced Virtuality, Microsoft and the Universal Hip Hop Museum, provides another example of a system behaving in an improvisational way. The project is a kiosk-based system that uses fictional characters that he created in collaboration with graphic novelists John Jennings and Stacey Robinson to go beyond the basics in order to tell the complex histories of Hip Hop and its socio-cultural impact. Visitors enter into a conversation with these fictional characters to learn about their musical tastes and interests through a conversation in order to tailor a specific narrative about Hip Hop history for them. 

“A lot of time, people feel that AI systems have to operate 100 percent autonomously,” Harrell said, “but I’m interested in ways that AI systems can operate with people and in the way that systems might do the unexpected, but in a manner constrained by careful ways of formalizing semantics.” 

His approach does not produce random ambiguity, he explained, instead producing “improvisationally-generated content with coherent meaning to provoke thought and ideally positive social change.”

Harrel elaborated on the expressive potential of computation in his book 2013 book “Phantasmal Media: An Approach to Imagination, Computation and Expression.” This unique medium can construct and reveal phantasms, “a particular pervasive kind of imagination, one that encompasses cognitive phenomena including sense of self, metaphor, social categorization, narrative and poetic thinking.”

Phantasms can be both oppressive, creating and reinforcing biases, or empowering. Therefore, it is up to creators to be conscious of this unique power and work to create empowering phantasms.

“The Enemy,” a project directed by the war photographer Karim Ben Khelifa that Harrell collaborated on as the Human-Computer Interaction Producer, challenges visitors to experience and look at war in a new way. In an era where people are increasingly desensitized to images of violence, “The Enemy” is a virtual reality experience that puts visitors in conversation with combatants on both sides of conflicts. The system reads the user’s body language and other cues to create unique experiences, both in the conversation with combatants and in the physical surrounding. If the user is inattentive, the skylights will darken as if on a cloudy day. In the end, the experience seeks to “reveal humanity and show how dehumanization is a precondition for war,” he said.

Harrell’s work also goes beyond computation. While he has been leading his center’s research in exploring computationally-supported role-playing to develop a perspective transformation, he has written for role-playing in popular culture gaming to do the same.

Last year, he wrote an adventure called “The Nightsea’s Succor,” which was published in the book “Journey’s Through the Radiant Citadel,” the first Dungeons and Dragons book entirely based in non-Western settings. Harrell’s story is based on cultures from Africa and the African diaspora. The book was recently nominated for a 2023 Nebula Award, given by the Science Fiction and Fantasy Writers Association.

No matter what medium he is working in, Harrell seeks to create works that cut across academic settings, fine art venues, popular culture and more.

“There is a research side and there is an impact side,” he says, “and it’s really important to me the works coming out of [the Center for Advanced Virtuality] have a real impact.”