Dietrich Graduate Students Advance Neuro Studies with AI

Three graduate students from Carnegie Mellon University’s Dietrich College of Humanities and Social Sciences — Andrew Luo, Joel Ye and Gabriel Sarch — are using artificial intelligence to advance neuroscience research.

Their work, co-authored with faculty and/or post-doctoral researchers from the Department of Psychology, Neuroscience Institute and Department of Machine Learning, presents new or improved methods for understanding human vision and cognition.

Luo, a Ph.D. student in the Neuroscience Institute and Machine Learning Department, has developed Brain Diffusion for Visual Exploration (BrainDiVE), a data-driven approach to synthesize images predicted to activate different regions of the brain. This technique bypasses the current approach that requires category-specific stimuli in order to characterize differences between regions of the brain. With this new technique, researchers can improve their understanding of human visual cortex using diffusion models. Luo worked in collaboration with Margaret M. Henderson, post-doctoral researcher in the Neuroscience Institute, and co-corresponding authors Leila Wehbe, an assistant professor in the Neuroscience Institute and Machine Learning Department, and Michael Tarr, the Kavčić-Moura University Professor of Cognitive and Brain Science, on the article currently available on arXiv.

Ye, a Ph.D. student in the Neuroscience Institute, has developed the program Neural Data Transformer 2 (NDT2), a spatiotemporal transformer for neural spiking activity. The model reads the pattern of spikes that occur when neurons in the brain fire in a coordinated fashion. This approach improves upon current models that are limited to individual experiments, robbing deep network models of a vast amount of potential data. NDT2 aims to leverage the various data sources that span multiple sessions, participants and experimental tasks. The ability to aggregate data opens new opportunities to decode behavior from brain activity efficiently. Ye worked in collaboration with Wehbe, Jennifer Collinger and Robert Gaunt, both assistant professors in the Neuroscience Institute, on the article currently available on bioRxiv.

Sarch, a Ph.D. student in the Neuroscience Institute and Machine Learning Department, has developed a technique to train neural networks to predict how the brain will respond to images from large-scale datasets of natural scenes. Sarch has enhanced this technique by using a process called “brain dissection” to identify how different brain regions selectivity encode spatial features, such as depth, surface normals, curvature and object relations. Understanding how different areas of the brain selectively encode these spatial characteristics enhances our knowledge of how a coherent 3D percept of the world is constructed. This work will improve how researchers approach functional characteristics of vision, specifically the human visual cortex, when viewing natural scenes. Sarch worked in collaboration with Tarr, Wehbe and Katerina Fragkiadaki, the JPMorgan Chase Associate Professor of Computer Science in the Machine Learning Department, on the article currently available on bioRxiv.

All three graduate students are presenting their work at the 2023 NeurIPS conference, Dec. 10–16 in New Orleans. Their papers will be published in Advances in Neural Information Processing Systems.

“Carnegie Mellon’s graduate students shine with their cutting-edge blend of disciplines in probing the mysteries of the mind and brain,” said Tarr, who also is head of the Department of Psychology at Dietrich College. “At NeurIPS, their presentations showcase an impressive command of both computational neuroscience and machine learning, reflecting deep, foundational expertise."

In addition, the team that created BrainDiVE has been selected for an oral presentation.

“Acceptance at NeurIPS is very competitive — around 20% — and being selected for an oral at NeurIPS is a great accomplishment. Fewer than 80 orals are chosen from more than 12,000 submitted papers,” Wehbe said.

The 2023 NeurIPS conference, scheduled for Dec.10–16 in New Orleans, will feature presentations from all three graduate students. This top machine learning conference, known for its rigorous selection process, accepted only about 26% of submitted papers this year after peer review. In addition, the BrainDIVE paper has been chosen for an oral presentation, a distinction granted to fewer than 80 papers out of over 12,000 submissions. All three papers will be published in Advances in Neural Information Processing Systems.