Sound and vision work hand in hand, UCLA psychologists report

December 08, 2011

Our senses of sight and hearing work closely together, perhaps more than people realize, a new UCLA psychology study shows.

"If we think of the perceptual system as a democracy where each sense is like a person casting a vote and all votes are counted to reach a decision -- although not all votes are counted equally -- what our study shows is that the voters talk to one another and influence one another even before each casts a vote," said Ladan Shams, a UCLA associate professor of psychology and the senior author of the new study.

In the study of how one sense can affect another, Shams, an expert on perception and cognitive neuroscience, and her colleagues showed 63 participants a large number of dots on a screen in two separate phases, with a break between the phases. In one phase, the dots moved around randomly; in the other, some of the dots moved together from right to left. In both phases, the dots were accompanied by sound.

Over a series of experiments, the researchers asked the participants to correctly identify the phase in which the dots moved together horizontally.

Participants were divided into three groups. One group heard sound moving from right to left as the dots moved from right to left, and they heard sound that remained stationary during the random phase. A second group heard the right-to-left sound during both phases. And a third group heard sound moving in the opposite direction -- from left to right -- during both phases. Then, each participant experienced trials in all three conditions.

As Shams expected, the participants were best able to identify the phase in which the dots moved horizontally when the sound moved in the same direction as the dots but remained stationary during the random phase. The researchers found that the sound that moved in the opposite direction neither enhanced nor worsened the participants' visual perception.

Surprisingly, the sound that traveled leftward both when the dots moved leftward and when the dots moved randomly -- that is, sound that provided no useful information for choosing between the two phases -- also helped people correctly choose the phase with the horizontal motion. Because the sound was identical in both phases, if the participants closed their eyes they, would have a 50-50 chance of successfully performing the task based on sound alone; with their eyes open, however, the interaction between sound and vision led to a significant improvement in detection of visual motion. Hearing enhanced seeing.

"Imagine you are playing ping-pong with a friend who serves the ball," Shams said. "You receive information about where and when the ball hit the table by both vision and hearing. Scientists have believed that each of the senses produces an estimate relevant for the task, and then these votes get combined subconsciously according to rules that take into account which sense is more reliable. This is how the senses interact in how we perceive the world. However, our findings show that the senses of hearing and vision can also interact at a more basic level, before they each even produce an estimate."

The study appears in the December issue of Psychological Science, a journal published by the Association for Psychological Science.

The senses affect one another other in many ways, said Robyn Kim, a former UCLA psychology postdoctoral scholar in Shams' laboratory and lead author of the study. There are connections between the auditory and visual portions of the brain at the cognitive level. When the information from one sense is ambiguous, another sense can step in and clarify or ratify the perception, Shams and Kim said.

"Most of us understand that smell affects taste," said Kim, who added that sound similarly affects vision.

"This study shows that at least in regards to perception of moving objects, hearing and sight are deeply intertwined, to the degree that even when sound is completely irrelevant to the task, it still influences the way we see the world," Shams said.

UCLA psychology graduate student Megan Peters, who conducts research in Shams' laboratory, was a co-author on the study.
-end-
UCLA is California's largest university, with an enrollment of nearly 38,000 undergraduate and graduate students. The UCLA College of Letters and Science and the university's 11 professional schools feature renowned faculty and offer 337 degree programs and majors. UCLA is a national and international leader in the breadth and quality of its academic, research, health care, cultural, continuing education and athletic programs. Six alumni and five faculty have been awarded the Nobel Prize.

For more news, visit the UCLA Newsroom and follow us on Twitter.

University of California - Los Angeles

Related Hearing Articles from Brightsurf:

Two molecular handshakes for hearing
Scientists have mapped and simulated those filaments at the atomic level, a discovery that shed lights on how the inner ear works and that could help researchers learn more about how and why people lose the ability to hear.

Proof-of-concept for a new ultra-low-cost hearing aid for age-related hearing loss
A new ultra-affordable and accessible hearing aid made from open-source electronics could soon be available worldwide, according to a study published September 23, 2020 in the open-access journal PLOS ONE by Soham Sinha from the Georgia Institute of Technology, Georgia, US, and colleagues.

Ultra-low-cost hearing aid could address age-related hearing loss worldwide
Using a device that could be built with a dollar's worth of open-source parts and a 3D-printed case, researchers want to help the hundreds of millions of older people worldwide who can't afford existing hearing aids to address their age-related hearing loss.

A promise to restore hearing
For the first time, researchers have used base editing to restore partial hearing to mice with a recessive mutation in the gene TMC1 that causes complete deafness, the first successful example of genome editing to fix a recessive disease-causing mutation.

Surprising hearing talents in cormorants
The great cormorant has more sensitive hearing under water than in air.

Veterinarians: Dogs, too, can experience hearing loss
Just like humans, dogs are sometimes born with impaired hearing or experience hearing loss as a result of disease, inflammation, aging or exposure to noise.

Older people who use hearing aids still report hearing challenges
A high proportion of older people with hearing aids, especially those with lower incomes, report having trouble hearing and difficulty accessing hearing care services, according to a study from researchers at Johns Hopkins Bloomberg School of Public Health.

Hearing class
New study finds that the class of neurons responsible for transmitting information from the inner ear to the brain is composed of three molecularly distinct subtypes.

Hearing tests on wild whales
Scientists published the first hearing tests on a wild population of healthy marine mammals.

Genes critical for hearing identified
Fifty-two previously unidentified genes that are critical for hearing have been found by testing over 3,000 mouse genes.

Read More: Hearing News and Hearing Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.