Nav: Home

Hearing with your eyes -- a Western style of speech perception

November 14, 2016

Which parts of a person's face do you look at when you listen them speak? Lip movements affect the perception of voice information from the ears when listening to someone speak, but native Japanese speakers are mostly unaffected by that part of the face. Recent research from Japan has revealed a clear difference in the brain network activation between two groups of people, native English speakers and native Japanese speakers, during face-to-face vocal communication.

It is known that visual speech information, such as lip movement, affects the perception of voice information from the ears when speaking to someone face-to-face. For example, lip movement can help a person to hear better under noisy conditions. On the contrary, dubbed movie content, where the lip movement conflicts with a speaker's voice, gives a listener the illusion of hearing another sound. This illusion is called the "McGurk effect."

According to an analysis of previous behavioral studies, native Japanese speakers are not influenced by visual lip movements as much as native English speakers. To examine this phenomenon further, researchers from Kumamoto University measured and analyzed gaze patterns, brain waves, and reaction times for speech identification between two groups of 20 native Japanese speakers and 20 native English speakers.

The difference was clear. When natural speech is paired with lip movement, native English speakers focus their gaze on a speaker's lips before the emergence of any sound. The gaze of native Japanese speakers, however, is not as fixed. Furthermore, native English speakers were able to understand speech faster by combining the audio and visual cues, whereas native Japanese speakers showed delayed speech understanding when lip motion was in view.

"Native English speakers attempt to narrow down candidates for incoming sounds by using information from the lips which start moving a few hundreds of milliseconds before vocalizations begin. Native Japanese speakers, on the other hand, place their emphasis only on hearing, and visual information seems to require extra processing," explained Kumamoto University's Professor Kaoru Sekiyama, who lead the research.

Kumamoto University researchers then teamed up with researchers from Sapporo Medical University and Japan's Advanced Telecommunications Research Institute International (ATR) to measure and analyze brain activation patterns using functional magnetic resonance imaging (fMRI). Their goal was to elucidate differences in brain activity between the two languages.

The functional connectivity in the brain between the area that deals with hearing and the area that deals with visual motion information, the primary auditory and middle temporal areas respectively, was stronger in native English speakers than in native Japanese speakers. This result strongly suggests that auditory and visual information are associated with each other at an early stage of information processing in an English speaker's brain, whereas the association is made at a later stage in a Japanese speaker's brain. The functional connectivity between auditory and visual information, and the manner in which the two types of information are processed together was shown to be clearly different between the two different language speakers.

"It has been said that video materials produce better results when studying a foreign language. However, it has also been reported that video materials do not have a very positive effect for native Japanese speakers," said Professor Sekiyama. "It may be that there are unique ways in which Japanese people process audio information, which are related to what we have shown in our recent research, that are behind this phenomenon."

These findings were published in the nature.com journal Scientific Reports on August 11th and October 13th, 2016.
-end-
[References]

J. Shinozaki, N. Hiroe, M. Sato, T. Nagamine, K. Sekiyama et al, "Impact of language on functional connectivity for audiovisual speech integration," Sci. Rep., vol. 6, no. August, p. 31388, Aug. 2016. DOI: 10.1038/srep31388.

S. Hisanaga, K. Sekiyama, T. Igasaki, and N. Murayama, "Language/Culture Modulates Brain and Gaze Processes in Audiovisual Speech Perception," Sci. Rep., vol. 6, p. 35265, Oct. 2016. DOI: 10.1038/srep35265.

Kumamoto University

Related Brain Articles:

Study describes changes to structural brain networks after radiotherapy for brain tumors
Researchers compared the thickness of brain cortex in patients with brain tumors before and after radiation therapy was applied and found significant dose-dependent changes in the structural properties of cortical neural networks, at both the local and global level.
Blue Brain team discovers a multi-dimensional universe in brain networks
Using a sophisticated type of mathematics in a way that it has never been used before in neuroscience, a team from the Blue Brain Project has uncovered a universe of multi-dimensional geometrical structures and spaces within the networks of the brain.
New brain mapping tool produces higher resolution data during brain surgery
Researchers have developed a new device to map the brain during surgery and distinguish between healthy and diseased tissues.
Newborn baby brain scans will help scientists track brain development
Scientists have today published ground-breaking scans of newborn babies' brains which researchers from all over the world can download and use to study how the human brain develops.
New test may quickly identify mild traumatic brain injury with underlying brain damage
A new test using peripheral vision reaction time could lead to earlier diagnosis and more effective treatment of mild traumatic brain injury, often referred to as a concussion.
This is your brain on God: Spiritual experiences activate brain reward circuits
Religious and spiritual experiences activate the brain reward circuits in much the same way as love, sex, gambling, drugs and music, report researchers at the University of Utah School of Medicine.
Brain scientists at TU Dresden examine brain networks during short-term task learning
'Practice makes perfect' is a common saying. We all have experienced that the initially effortful implementation of novel tasks is becoming rapidly easier and more fluent after only a few repetitions.
Balancing time & space in the brain: New model holds promise for predicting brain dynamics
A team of scientists has extended the balanced network model to provide deep and testable predictions linking brain circuits to brain activity.
New view of brain development: Striking differences between adult and newborn mouse brain
Spikes in neuronal activity in young mice do not spur corresponding boosts in blood flow -- a discovery that stands in stark contrast to the adult mouse brain.
Map of teenage brain provides evidence of link between antisocial behavior and brain development
The brains of teenagers with serious antisocial behavior problems differ significantly in structure to those of their peers, providing the clearest evidence to date that their behavior stems from changes in brain development in early life, according to new research led by the University of Cambridge and the University of Southampton, in collaboration with the University of Rome Tor Vergata in Italy.

Related Brain Reading:

Best Science Podcasts 2019

We have hand picked the best science podcasts for 2019. Sit back and enjoy new science podcasts updated daily from your favorite science news services and scientists.
Now Playing: TED Radio Hour

Digital Manipulation
Technology has reshaped our lives in amazing ways. But at what cost? This hour, TED speakers reveal how what we see, read, believe — even how we vote — can be manipulated by the technology we use. Guests include journalist Carole Cadwalladr, consumer advocate Finn Myrstad, writer and marketing professor Scott Galloway, behavioral designer Nir Eyal, and computer graphics researcher Doug Roble.
Now Playing: Science for the People

#530 Why Aren't We Dead Yet?
We only notice our immune systems when they aren't working properly, or when they're under attack. How does our immune system understand what bits of us are us, and what bits are invading germs and viruses? How different are human immune systems from the immune systems of other creatures? And is the immune system so often the target of sketchy medical advice? Those questions and more, this week in our conversation with author Idan Ben-Barak about his book "Why Aren't We Dead Yet?: The Survivor’s Guide to the Immune System".