Three graduate students from Carnegie Mellon University’s Dietrich College of Humanities and Social Sciences — Andrew Luo(opens in new window), Joel Ye(opens in new window) and Gabriel Sarch(opens in new window) — are using artificial intelligence to advance neuroscience research.

Their work, co-authored with faculty and/or post-doctoral researchers from the Department of Psychology(opens in new window), Neuroscience Institute(opens in new window) and Department of Machine Learning(opens in new window), presents new or improved methods for understanding human vision and cognition.

Luo, a Ph.D. student in the Neuroscience Institute and Machine Learning Department, has developed Brain Diffusion for Visual Exploration (BrainDiVE), a data-driven approach to synthesize images predicted to activate different regions of the brain. This technique bypasses the current approach that requires category-specific stimuli in order to characterize differences between regions of the brain. With this new technique, researchers can improve their understanding of human visual cortex using diffusion models. Luo worked in collaboration with Margaret M. Henderson, post-doctoral research in the Neuroscience Institute, Leila Wehbe, an assistant professor in the Neuroscience Institute and Machine Learning Department, and Mike Tarr, the Kavčić-Moura Professor of Cognitive and Brain Science, on the article that is available on Arxiv(opens in new window), a preprint server for articles published prior to peer review.

Ye, a Ph.D. student in the Neuroscience Institute, has developed the program Neural Data Transformer 2 (NDT2), a spatiotemporal transformer for neural spiking activity. The model reads the pattern of spikes that occur when neurons in the brain fire in a coordinated fashion. This approach improves upon current models that are limited to individual experiments, robbing deep network models of a vast amount of potential data. NDT2 aims to leverage the various data sources that span multiple sessions, participants and experimental tasks. The ability to aggregate data opens new opportunities to decode behavior from brain activity efficiently. Ye worked in collaboration with Wehbe, Jennifer Collinger and Robert Gaunt, both assistant professors in the neuroscience institute, on the article published on BioRxiv(opens in new window), a preprint server for articles published prior to peer review.

Sarch, a Ph.D. student in the Neuroscience Institute and Machine Learning Department, has developed a technique to train neural networks to predict how the brain will respond to images from large-scale datasets of natural scenes. Sarch has enhanced this technique by using a process called “brain dissection” to identify how different brain regions selectivity encode spatial features, such as depth, surface normals, curvature and object relations. Understanding how different areas of the brain selectively encode these spatial characteristics enhances our knowledge of how a coherent 3D percept of the world is constructed. This work will improve how researchers approach functional characteristics of vision, specifically the human visual cortex, when viewing natural scenes. Sarch worked in collaboration with Tarr, Wehbe, and Katerina Fragkiadaki, the JPMorgan Chase Associate Professor of Computer Science in the Machine Learning Department, on an article to be published in the Neural Information Processing Systems proceedings and available publicly on BioRxiv(opens in new window).

All three graduate students are presenting their work at the 2023 NeurIPS conference(opens in new window), Dec. 10–16 in New Orleans.

“Carnegie Mellon’s graduate students shine with their cutting-edge blend of disciplines in probing the mysteries of the mind and brain,” said Tarr, who also is head of the Department of Psychology. “At NeurIPS, their presentations showcase an impressive command of both Computational Neuroscience and Machine Learning, reflecting deep, foundational expertise."