December 11, 2019
Using Typing to Decode Language Production
New study uses brain waves triggered while typing to explore the similarities between typing and speech production.
By Stacy W. Kish
Touch typing is a skill where a person can type quickly without looking at their hands. Researchers at Carnegie Mellon University have found that people can type words quite accurately without immediately seeing what they type, although they slow down and become more cautious. Importantly, the brain can detect errors, sometimes unconsciously, even when they do not see what they have typed. These results are opening new doors to understand how vision, language, and motor systems in the brain interact.
“Our results, along with those in other labs, indicate that a bigger and more complex system of language production are at work behind your fingertips when you type,” said Bonnie Nozari, associate professor in the department of Psychology at CMU and senior author on the study. “We can use brain waves to show similarities between typing and both language and motor production.”
The results are published online by the Journal of Cognitive Neuroscience.
The researchers examined typing accuracy for 17 right-handed participants (10 women, mean = 25 years old) who performed a typing-to-dictation task. Under one condition, the participants could see what they were typing on the screen in real time. In another condition, they only saw what they typed after they had finished typing.
While the participants could detect their errors, when the visual reference has been removed, they had a much harder time correcting their errors.
Nozari and her postdoctoral fellow, Svetlana Pinet, examined the electroencephalogram (EEG) record, captured as the participants performed the typing task. By time-locking the EEG signal to each word and each key-stroke, they were able to uncover differences in the mechanisms of typing when people did or did not see what they typed. They found that with visual feedback removed, participants relied more on the internal monitoring processes.
Interestingly, they found that typing errors were not just a result of clumsy fingers slipping on the keys. No. According to Nozari, the errors were similar to errors in speech production. Mistyping was akin to misspeaking. Instead of saying ‘red bag,’ you say “bed rag,” switching the first sounds of the words. Many typing errors are very similar, switching the ‘r’ and the ‘b,’ which are not next to each other on the keyboard.
But how realistic are EEG studies in context of interconnected speech? Good question.
“We are tying the time log of the brain wave record to each individual keystroke,” Nozari cautions. “This is challenging, because there are lingering processes from one keystroke to another, mushing the EEG signal together. The same is true for words in connected speech. In collaboration Stephanie Riès at San Diego State University, we are trying to see how far we can get with collecting clean signal in connected speech.”
“I believe this [work gives us] an important and broad picture of brain functioning,” said Pinet at the Basque Center on Cognition, Brain and Language in Spain and first author on the study. “The language system is not isolated from other functions, but rather that they all work together and rely on similar mechanisms.”
According to Pinet, this work could be beneficial in future studies to help patients suffering brain damage to help restore their lives
Nozari and Pinet published their findings in the study, titled “Electrophysiological correlates of monitoring in typing with and without visual feedback.” The data for the study was collected at the Department of Neurology at Johns Hopkins, where Nozari was a faculty member before moving to Carnegie Mellon earlier this year. The study was supported by the Therapeutic Cognitive Neurology Fund, the Therapeutic Cognitive Neuroscience Professorship, the Benjamin and Adith Miller Family Endowment for Aging, Alzheimer’s, and Autism, The Binder Family Charitable Fund, the Murren Family Foundation, the Wockenfuss Endowment and Fund and anonymous donors.