Washington University in St. Louis brain scholars will join teams from four other universities in a five-year, $7.5 million research project that aims to build and test the most comprehensive model yet of how people understand and remember events.
“The ultimate goal is a system that can watch the same movies and read the same stories that we show to our human subjects and make detailed, moment-by-moment predictions about what is going on in their minds and brains,” said Jeff Zacks, lead investigator for the Washington University team and professor of psychological and brain sciences in Arts & Sciences.
“If we are successful, this research will enable computer systems that are better collaborators and better teachers, and it will reveal fundamental mechanisms of how people understand their complex everyday worlds.”
Funded by the U.S. Department of Defense, the project will involve a combination of brain imaging (functional MRI and intracranial electrode recordings), behavioral experiments and computational models. The effort has potential to spur big leaps in the development of artificial intelligence and may open new avenues for understanding Alzheimer’s disease, dementia and other memory disorders.
Led by researchers at the Center for Neuroscience at the University of California, Davis, the project also involves brain scholars from Princeton University, Harvard University and New York University. Each investigator brings a different kind of expertise needed to map and create a computer model of the brain’s memory networks.
Zacks’ team at Washington University will receive $1.5 million for their part of the project, which will involve a range of behavioral and brain-imaging experiments designed to identify the cognitive and neural causes and consequences of event segmentation.
His collaborators on campus are Todd Braver, professor of psychological and brain sciences, and Aaron Bobick, the James M. McKelvey Professor and dean of the School of Engineering & Applied Science.
Zacks, author of “Flicker: Your Brain on Movies,” is director of the psychology department’s Dynamic Cognition Laboratory, which studies how the representations in the brain and the world work together in cognition. He studies perception and cognition using behavioral experiments, brain imaging, computational modeling and testing of neurological patients. One line of research examines how people parse the continuous stream of behavior into meaningful events — and how this affects memory and cognition.
Braver studies the cognitive and neural mechanisms underlying memory, attention and controlled processing. One area of ongoing research is the testing of models that predict how the prefrontal cortex represents and maintains information in working memory.
Bobick’s research primarily focuses on action recognition by computer vision, an area in which he was a pioneer. He later extended his research to robot perception for human-robot collaboration.
“At Washington University, we will first be developing a large stimulus set of movies of everyday activities,” Zacks said. “The stimuli will be coded so that we can feed the same inputs into human observers and into computational models. We will then conduct experiments measuring behavioral judgments during comprehension and memory, and measuring brain activity in parallel with fMRI brain imaging. The overall goal is to have a computationally explicit description of how people represent everyday actions as they watch them, in terms of features like actors, actions, objects, goals, spatial layouts and temporal sequences.”