Nav: Home

Beyond bananas: 'Mind reading' technology decodes complex thoughts

June 26, 2017

Carnegie Mellon University scientists can now use brain activation patterns to identify complex thoughts, such as, "The witness shouted during the trial."

This latest research led by CMU's Marcel Just builds on the pioneering use of machine learning algorithms with brain imaging technology to "mind read." The findings indicate that the mind's building blocks for constructing complex thoughts are formed by the brain's various sub-systems and are not word-based. Published in Human Brain Mapping and funded by the Intelligence Advanced Research Projects Activity (IARPA), the study offers new evidence that the neural dimensions of concept representation are universal across people and languages.

"One of the big advances of the human brain was the ability to combine individual concepts into complex thoughts, to think not just of 'bananas,' but 'I like to eat bananas in evening with my friends,'" said Just, the D.O. Hebb University Professor of Psychology in the Dietrich College of Humanities and Social Sciences. "We have finally developed a way to see thoughts of that complexity in the fMRI signal. The discovery of this correspondence between thoughts and brain activation patterns tells us what the thoughts are built of."

Previous work by Just and his team showed that thoughts of familiar objects, like bananas or hammers, evoke activation patterns that involve the neural systems that we use to deal with those objects. For example, how you interact with a banana involves how you hold it, how you bite it and what it looks like.

The new study demonstrates that the brain's coding of 240 complex events, sentences like the shouting during the trial scenario uses an alphabet of 42 meaning components, or neurally plausible semantic features, consisting of features, like person, setting, size, social interaction and physical action. Each type of information is processed in a different brain system--which is how the brain also processes the information for objects. By measuring the activation in each brain system, the program can tell what types of thoughts are being contemplated.

For seven adult participants, the researchers used a computational model to assess how the brain activation patterns for 239 sentences corresponded to the neurally plausible semantic features that characterized each sentence. Then the program was able to decode the features of the 240th left-out sentence. They went through leaving out each of the 240 sentences in turn, in what is called cross-validation.

The model was able to predict the features of the left-out sentence, with 87 percent accuracy, despite never being exposed to its activation before. It was also able to work in the other direction, to predict the activation pattern of a previously unseen sentence, knowing only its semantic features.

"Our method overcomes the unfortunate property of fMRI to smear together the signals emanating from brain events that occur close together in time, like the reading of two successive words in a sentence," Just said. "This advance makes it possible for the first time to decode thoughts containing several concepts. That's what most human thoughts are composed of."

He added, "A next step might be to decode the general type of topic a person is thinking about, such as geology or skateboarding. We are on the way to making a map of all the types of knowledge in the brain."

CMU's Jing Wang and Vladimir L. Cherkassky also participated in the study.

Discovering how the brain decodes complex thoughts is one of the many brain research breakthroughs to happen at Carnegie Mellon. CMU has created some of the first cognitive tutors, helped to develop the Jeopardy-winning Watson, founded a groundbreaking doctoral program in neural computation, and is the birthplace of artificial intelligence and cognitive psychology. Building on its strengths in biology, computer science, psychology, statistics and engineering, CMU launched BrainHub, an initiative that focuses on how the structure and activity of the brain give rise to complex behaviors.
-end-


Carnegie Mellon University

Related Brain Articles:

Scientists predict the areas of the brain to stimulate transitions between different brain states
Using a computer model of the brain, Gustavo Deco, director of the Center for Brain and Cognition, and Josephine Cruzat, a member of his team, together with a group of international collaborators, have developed an innovative method published in Proceedings of the National Academy of Sciences on Sept.
BRAIN Initiative tool may transform how scientists study brain structure and function
Researchers have developed a high-tech support system that can keep a large mammalian brain from rapidly decomposing in the hours after death, enabling study of certain molecular and cellular functions.
Wiring diagram of the brain provides a clearer picture of brain scan data
In a study published today in the journal BRAIN, neuroscientists led by Michael D.
Blue Brain Project releases first-ever digital 3D brain cell atlas
The Blue Brain Cell Atlas is like ''going from hand-drawn maps to Google Earth'' -- providing previously unavailable information on major cell types, numbers and positions in all 737 brain regions.
Landmark study reveals no benefit to costly and risky brain cooling after brain injury
A landmark study, led by Monash University researchers, has definitively found that the practice of cooling the body and brain in patients who have recently received a severe traumatic brain injury, has no impact on the patient's long-term outcome.
Brain cells called astrocytes have unexpected role in brain 'plasticity'
Researchers from the Salk Institute have shown that astrocytes -- long-overlooked supportive cells in the brain -- help to enable the brain's plasticity, a new role for astrocytes that was not previously known.
Largest brain study of 62,454 scans identifies drivers of brain aging
In the largest known brain imaging study, scientists from Amen Clinics (Costa Mesa, CA), Google, John's Hopkins University, University of California, Los Angeles and the University of California, San Francisco evaluated 62,454 brain SPECT (single photon emission computed tomography) scans of more than 30,000 individuals from 9 months old to 105 years of age to investigate factors that accelerate brain aging.
Is whole-brain radiation still best for brain metastases from small-cell lung cancer?
University of Colorado Cancer Center study compares outcomes of 5,752 small-cell lung cancer patients who received whole-brain radiation therapy (WBRT) with those of 200 patients who received stereotactic radiosurgery (SRS), finding that the median overall survival was actually longer with SRS (10.8 months with SRS versus 7.1 months with WBRT).
Atlas of brain blood vessels provides fresh clues to brain diseases
Even though diseases of the brain vasculature are some of the most common causes of death in the West, knowledge of these blood vessels is limited.
Brain sciences researcher pinpoints brain circuit that triggers fear relapse
Steve Maren, the Claude H. Everett Jr. '47 Chair of Liberal Arts professor in the Department of Psychological and Brain Sciences at Texas A&M University, and his Emotion and Memory Systems Laboratory (EMSL) have made a breakthrough discovery in the process of fear relapse.
More Brain News and Brain Current Events

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Rethinking Anger
Anger is universal and complex: it can be quiet, festering, justified, vengeful, and destructive. This hour, TED speakers explore the many sides of anger, why we need it, and who's allowed to feel it. Guests include psychologists Ryan Martin and Russell Kolts, writer Soraya Chemaly, former talk radio host Lisa Fritsch, and business professor Dan Moshavi.
Now Playing: Science for the People

#538 Nobels and Astrophysics
This week we start with this year's physics Nobel Prize awarded to Jim Peebles, Michel Mayor, and Didier Queloz and finish with a discussion of the Nobel Prizes as a way to award and highlight important science. Are they still relevant? When science breakthroughs are built on the backs of hundreds -- and sometimes thousands -- of people's hard work, how do you pick just three to highlight? Join host Rachelle Saunders and astrophysicist, author, and science communicator Ethan Siegel for their chat about astrophysics and Nobel Prizes.