Nav: Home

From expressions to mind wandering: Using computers to illuminate human emotions

May 05, 2020

May 5, 2020 - A common view of human emotions is that they are too idiosyncratic and subjective to be studied scientifically. But as being presented at the Cognitive Neuroscience Society (CNS) virtual meeting today, cognitive neuroscientists are using contemporary, data-driven computational methods to overturn old ideas about the structure of emotions across humanity.

Researchers are applying computing power to understanding everything from how we generate spontaneous emotions during mind wandering to how we decode facial expressions across cultures. Their findings are important in characterizing how emotions contribute to well-being, the neurobiology of psychiatric disorders, and even how to make more effective, social robots.

"Artificial intelligence (AI) enables scientists to study emotions in ways that were previously thought to be impossible, which is leading to discoveries that transform how we think emotions are generated from biological signals," says Kevin LaBar of Duke University who is chairing the symposium on this topic at the CNS virtual meeting.

Decoding facial expressions across cultures

Six core human emotions -- fear, anger, disgust, sadness, happiness and surprise -- have been considered as universal in human psychology for decades and popularized in such places as the Pixar film "Inside Out." Yet despite the societal prevalence of this idea dating back to the work of Paul Ekman, the scientific consensus actually shows that these emotions are far from universal, with a significant gap in facial recognition of these emotions across cultures in particular for people from East Asia, says Rachael Jack, a researcher at the University of Glasgow.

Jack has been working to understand what she calls the "language of the face" -- how individual face movements combine in different ways to create meaningful facial expressions (like how letters combine to create words). "I think of this a bit like trying to crack hieroglyphics or an unknown ancient language," Jack says. "We know so much about spoken and written language, even hundreds of ancient languages, but we have comparatively little formal knowledge of the non-verbal communication systems we use every day and which are so critical to all human societies."

In new work she is presenting at the CNS annual meeting, Jack will showcase the novel data-driven methods her team has used to develop dynamic models of these face movements, like a recipe book of facial expressions of emotions. Her team is now transferring these models to digital agents, such as social robots and virtual humans, so that they can generate facial expressions that are socially nuanced and culturally sensitive.

In their work, a novel face movement generator, created at the Institute of Neuroscience and Psychology at the University of Glasgow, randomly selects a subset of individual face movements, such as eyebrow raiser, nose wrinkler, or lip stretcher, and randomly activates the intensity and timing of each. These randomly activated face movements then combine to create a facial animation. Study participants from different cultures then categorize the facial animation according to the six classic emotions, or they can select "other" if they do not perceive any of these emotions. After many such trials, the researchers build a statistical relationship between the face movements presented on each trial and the participants' responses, which produces a mathematical model.

"In contrast to traditional theory-driven approaches where experimenters took a hypothesized set of facial expressions and showed them to participants across the world, we have added a psychophysical approach," she explains. "It is more data-driven and more agnostic in sampling and testing facial expressions and, critically, uses the subjective perceptions of cultural participants to understand what face movements drive their perception of a given emotion, for example, 'he is happy.'"

These studies have whittled the six commonly thought of universal facial expressions of emotions to only four cross-cultural expressions. "There are substantial cultural differences in facial expressions that can hinder cross-cultural communication," Jack says. "We often, but not always, find that East Asian facial expressions have more expressive eyes than Western facial expressions, which tend to have more expressive mouths, just like Eastern v. Western emoticons!"

She adds that there are also cultural commonalities that can be used to support accurate cross-cultural communication of specific messages - for example, facial expressions of happy, interested, and bored are similar across Eastern and Western cultures and can be recognized across cultures easily.

Jack and her team are now using their models to enhance the social signalling capabilities of robots and other digital agents that can be used globally. "We're very excited to transfer our facial expression models to a range of digital agents and to see the dramatic improvement in performance," she says.

Understanding spontaneous emotions at rest

Understanding how the subjective experience of emotion is mediated in the brain is the holy grail of affective neuroscience, says LaBar of Duke. "It is a hard problem, and there has been little progress to date."

In his lab, LaBar and colleagues are working to understand the emotions that emerge while the brain is mind-wandering at rest. "Whether triggered by internal thoughts or memories, these 'stream-of-consciousness' emotions are the targets of rumination and worry that can lead to prolonged mood states, and can bias memory and decision-making," he explains.

Until recently, researchers have been unable to decode these emotions from resting-state signals of brain function. As he is presenting in Boston today, LaBar's team has been able to apply machine learning tools to derive neuroimaging markers of a small set of emotions like fear, anger, and surprise. And the researchers have modeled how these emotions spontaneously emerge in the brain while subjects are resting in an MRI scanner.

The core of the work has been training a machine learning algorithm to differentiate patterns of brain activity that separate emotions from one another. The researchers present a pattern classifier algorithm with a training data set from a group of participants who were presented with music and movie clips that induced specific emotions. Using feedback, the algorithm learns to weigh the inputs coming from different regions of the brain to optimize the signaling of each emotion. The researchers then test how well the classifier can predict the elicited emotions in a new sample of participants using the set of brain weights it generated from the test sample.

"Once the emotion-specific brain patterns are validated across subjects in this way, we look for evidence that these patterns emerge spontaneously in participants who are merely lying at rest in the scanner," he says. "We can then determine whether the pattern classifier accurately predicts the emotions that people spontaneously report in the scanner, and identify individual differences."

With this method, they have shown, for example, how individual differences in anxiety and depression bias the expression of spontaneous emotions, how emotions change across the lifespan as people age, and how the temporal dynamics of emotions differentiate healthy subjects from individuals with psychopathology.

"Our hope is that this endeavor will lead to more reliable, objective indices of specific emotional experiences that do not rely on self-report, as these verbal reports are not reliable in certain populations, like children or even some adults who have little self-insight into their emotions," he says. "If so, then we might gain traction into understanding the emotional life of these individuals better."
-- The symposium "Contemporary Approaches to Emotion Representations" is taking place at CNS 2020 Virtual, from May 2-5 (

CNS is committed to the development of mind and brain research aimed at investigating the psychological, computational, and neuroscientific bases of cognition. Since its founding in 1994, the Society has been dedicated to bringing its 2,000 members worldwide the latest research to facilitate public, professional, and scientific discourse.

Cognitive Neuroscience Society

Related Neuroscience Articles:

Researchers rebuild the bridge between neuroscience and artificial intelligence
In an article in the journal Scientific Reports, researchers reveal that they have successfully rebuilt the bridge between experimental neuroscience and advanced artificial intelligence learning algorithms.
The evolution of neuroscience as a research
When the first issue of the JDR was published, the field of neuroscience did not exist but over subsequent decades neuroscience has emerged as a scientific field that has particular relevance to dentistry.
Diabetes-Alzheimer's link explored at Neuroscience 2019
Surprising links exist between diabetes and Alzheimer's disease, and researchers are beginning to unpack the pathology that connects the two.
Organoid research revealed at Neuroscience 2019
Mini-brains, also called organoids, may offer breakthroughs in clinical research by allowing scientists to study human brain cells without a human subject.
The neuroscience of autism: New clues for how condition begins
UNC School of Medicine scientists found that a gene mutation linked to autism normally works to organize the scaffolding of brain cells called radial progenitors necessary for the orderly formation of the brain.
Harnessing reliability for neuroscience research
Neuroscientists are amassing the large-scale datasets needed to study individual differences and identify biomarkers.
Blue Brain solves a century-old neuroscience problem
In a front-cover paper published in Cerebral Cortex, EPFL's Blue Brain Project, a Swiss Brain Research Initiative, explains how the shapes of neurons can be classified using mathematical methods from the field of algebraic topology.
Characterizing pig hippocampus could improve translational neuroscience
Researchers have taken further steps toward developing a superior animal model of neurological conditions such as traumatic brain injury and epilepsy, according to a study of miniature pigs published in eNeuro.
The neuroscience of human vocal pitch
Among primates, humans are uniquely able to consciously control the pitch of their voices, making it possible to hit high notes in singing or stress a word in a sentence to convey meaning.
Study tackles neuroscience claims to have disproved 'free will'
For several decades, some researchers have argued that neuroscience studies prove human actions are driven by external stimuli -- that the brain is reactive and free will is an illusion.
More Neuroscience News and Neuroscience Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Listen Again: The Power Of Spaces
How do spaces shape the human experience? In what ways do our rooms, homes, and buildings give us meaning and purpose? This hour, TED speakers explore the power of the spaces we make and inhabit. Guests include architect Michael Murphy, musician David Byrne, artist Es Devlin, and architect Siamak Hariri.
Now Playing: Science for the People

#576 Science Communication in Creative Places
When you think of science communication, you might think of TED talks or museum talks or video talks, or... people giving lectures. It's a lot of people talking. But there's more to sci comm than that. This week host Bethany Brookshire talks to three people who have looked at science communication in places you might not expect it. We'll speak with Mauna Dasari, a graduate student at Notre Dame, about making mammals into a March Madness match. We'll talk with Sarah Garner, director of the Pathologists Assistant Program at Tulane University School of Medicine, who takes pathology instruction out of...
Now Playing: Radiolab

What If?
There's plenty of speculation about what Donald Trump might do in the wake of the election. Would he dispute the results if he loses? Would he simply refuse to leave office, or even try to use the military to maintain control? Last summer, Rosa Brooks got together a team of experts and political operatives from both sides of the aisle to ask a slightly different question. Rather than arguing about whether he'd do those things, they dug into what exactly would happen if he did. Part war game part choose your own adventure, Rosa's Transition Integrity Project doesn't give us any predictions, and it isn't a referendum on Trump. Instead, it's a deeply illuminating stress test on our laws, our institutions, and on the commitment to democracy written into the constitution. This episode was reported by Bethel Habte, with help from Tracie Hunte, and produced by Bethel Habte. Jeremy Bloom provided original music. Support Radiolab by becoming a member today at     You can read The Transition Integrity Project's report here.