Video Game Helps Neuroscientists Understand Second Language Learners

Research in the Holt Lab continues as the CMU community works remotely

An alien ship appears on the left side of the screen. The player hears it coming and has a laser ready. With a quick tap of the space bar, the alien is toast. The player is safe for just a moment before hearing another alien approaching, this time from the right.

The sounds the alien ships make as they enter the screen are not arbitrary — each corner of the screen represents a different Mandarin tone group. For example, when a ship comes from the left side of the screen, the player may hear voice speak a syllable with a neutral, level tone. When a ship enters from the bottom, the same syllable is heard, but with a rising tone.

The player starts to learn where ships are coming from based on what they hear, earning a higher score and, incidentally, a better way to perceive tonal differences.

The space invaders in this online game are helping the player with a lesson in Mandarin Chinese. Part of a research study supported by the National Institutes of Health and led by auditory neuroscientist Lori Holt and psycholinguist Seth Wiener, the game could help scientists understand how second language learners learn a new language, and could even help them learn it faster.

"Incidental learning happens when the learner doesn't pay attention to the thing that they're trying to learn," explained Tim Murphy, a graduate student in Holt's Department of Psychology lab and a Neuroscience Institute Presidential Fellow. "When people are consciously trying to learn these sounds and think about them, for whatever reason, it's very difficult. The game helps them learn these foreign language sounds in a way that allows them to generalize to understand sounds that they haven't heard before."

Murphy said that results indicate that students who play the game were better able to categorize Mandarin tones relative to students who only learned in a traditional classroom. What's more, they are starting to see data that suggests that game players not only can differentiate between these sounds better, but can produce the sounds better when they practice speaking Mandarin.

Holt, Weiner and Murphy have been working on the study for two years with Carnegie Mellon students in Mandarin classes. In March, the university community learned they would finish the semester remotely as a result of the COVID-19 crisis.

Murphy did not want to lose valuable research time, so he devised a way to continue the research online.

"A crucial part of the project happens at the end of the semester," Murphy said. "Students who have been playing the game all semester come into our lab on three consecutive days, and we are able to see what they learned."

Without that time in the lab, a semester's worth of research would be lost. Murphy was able to move the in-person experiments online, and was with the students via Zoom as they answered questions about their progress and performed tasks in their browser.

"There was a lot that could have gone wrong," Murphy said. "The tough part was asking participants to make audio recordings using their laptops. But we invested so much into these students and want to see the study through."

Technology came through. With a lot of hard work in the weeks leading up to the end of the semester Murphy was able to transfer his research tasks online and successfully collect data from his classroom learners. Fittingly, he presented the results of this research remotely, via Zoom, in a Department of Psychology graduate student research symposium in May.