Nav: Home

Scientists translate brain signals into speech sounds

April 24, 2019

Scientists used brain signals recorded from epilepsy patients to program a computer to mimic natural speech--an advancement that could one day have a profound effect on the ability of certain patients to communicate. The study was supported by the National Institutes of Health's Brain Research through Advancing Innovative Technologies (BRAIN) Initiative.

"Speech is an amazing form of communication that has evolved over thousands of years to be very efficient," said Edward F. Chang, M.D., professor of neurological surgery at the University of California, San Francisco (UCSF) and senior author of this study published in Nature. "Many of us take for granted how easy it is to speak, which is why losing that ability can be so devastating. It is our hope that this approach will be helpful to people whose muscles enabling audible speech are paralyzed."

In this study, speech scientists and neurologists from UCSF recreated many vocal sounds with varying accuracy using brain signals recorded from epilepsy patients with normal speaking abilities. The patients were asked to speak full sentences, and the data obtained from brain scans was then used to drive computer-generated speech. Furthermore, simply miming the act of speaking provided sufficient information to the computer for it to recreate several of the same sounds.

The loss of the ability to speak can have devastating effects on patients whose facial, tongue, and larynx muscles have been paralyzed due to stroke or other neurological conditions. Technology has helped these patients to communicate through devices that translate head or eye movements into speech. Because these systems involve the selection of individual letters or whole words to build sentences, the speed at which they can operate is very limited. Instead of recreating sounds based on individual letters or words, the goal of this project was to synthesize the specific sounds used in natural speech.

"Current technology limits users to, at best, 10 words per minute, while natural human speech occurs at roughly 150 words/minute," said Gopala K. Anumanchipalli, Ph.D., speech scientist, UCSF and first author of the study. "This discrepancy is what motivated us to test whether we could record speech directly from the human brain."

The researchers took a two-step approach to solving this problem. First, by recording signals from patients' brains while they were asked to speak or mime sentences, they built maps of how the brain directs the vocal tract, including the lips, tongue, jaw, and vocal cords, to make different sounds. Second, the researchers applied those maps to a computer program that produces synthetic speech.

Volunteers were then asked to listen to the synthesized sentences and to transcribe what they heard. More than half the time, the listeners were able to correctly determine the sentences being spoken by the computer.

By breaking down the problem of speech synthesis into two parts, the researchers appear to have made it easier to apply their findings to multiple individuals. The second step specifically, which translates vocal tract maps into synthetic sounds, appears to be generalizable across patients.

"It is much more challenging to gather data from paralyzed patients, so being able to train part of our system using data from non-paralyzed individuals would be a significant advantage," said Dr. Chang.

The researchers plan to design a clinical trial involving paralyzed, speech-impaired patients to determine how to best gather brain signal data which can then be applied to the previously trained computer algorithm.

"This study combines state-of-the-art technologies and knowledge about how the brain produces speech to tackle an important challenge facing many patients," said Jim Gnadt, Ph.D., program director at the NIH's National Institute of Neurological Disorders and Stroke. "This is precisely the type of problem that the NIH BRAIN Initiative is set up to address: to use investigative human neuroscience to impact care and treatment in the clinic."
-end-
This research was funded by the NIH BRAIN Initiative (DP2 OD008627 and U01 NS098971-01), the New York Stem Cell Foundation, the Howard Hughes Medical Institute, the McKnight Foundation, the Shurl and Kay Curci Foundation, and the William K. Bowes Foundation.

For more information:

National Institute for Neurological Disorders and Stroke - http://www.ninds.nih.gov/

NIH Brain Research through Advancing Innovative Technologies (BRAIN) Initiative - https://www.braininitiative.nih.gov/

The NIH's Brain Research through Advancing Innovative Neurotechnologies® (BRAIN) Initiative is aimed at revolutionizing our understanding of the human brain. It is managed by 10 institutes whose missions and current research portfolios complement the goals of the BRAIN Initiative: NCCIH, NEI, NIA, NIAAA, NIBIB, NICHD, NIDA, NIDCD, NIMH, and NINDS.

NINDS is the nation's leading funder of research on the brain and nervous system. The mission of NINDS is to seek fundamental knowledge about the brain and nervous system and to use that knowledge to reduce the burden of neurological disease.

About the National Institutes of Health (NIH):

NIH, the nation's medical research agency, includes 27 Institutes and Centers and is a component of the U.S. Department of Health and Human Services. NIH is the primary federal agency conducting and supporting basic, clinical, and translational medical research, and is investigating the causes, treatments, and cures for both common and rare diseases. For more information about NIH and its programs, visit http://www.nih.gov.

NIH/National Institute of Neurological Disorders and Stroke

Related Brain Articles:

Transplanting human nerve cells into a mouse brain reveals how they wire into brain circuits
A team of researchers led by Pierre Vanderhaeghen and Vincent Bonin (VIB-KU Leuven, Université libre de Bruxelles and NERF) showed how human nerve cells can develop at their own pace, and form highly precise connections with the surrounding mouse brain cells.
Brain scans reveal how the human brain compensates when one hemisphere is removed
Researchers studying six adults who had one of their brain hemispheres removed during childhood to reduce epileptic seizures found that the remaining half of the brain formed unusually strong connections between different functional brain networks, which potentially help the body to function as if the brain were intact.
Alcohol byproduct contributes to brain chemistry changes in specific brain regions
Study of mouse models provides clear implications for new targets to treat alcohol use disorder and fetal alcohol syndrome.
Scientists predict the areas of the brain to stimulate transitions between different brain states
Using a computer model of the brain, Gustavo Deco, director of the Center for Brain and Cognition, and Josephine Cruzat, a member of his team, together with a group of international collaborators, have developed an innovative method published in Proceedings of the National Academy of Sciences on Sept.
BRAIN Initiative tool may transform how scientists study brain structure and function
Researchers have developed a high-tech support system that can keep a large mammalian brain from rapidly decomposing in the hours after death, enabling study of certain molecular and cellular functions.
Wiring diagram of the brain provides a clearer picture of brain scan data
In a study published today in the journal BRAIN, neuroscientists led by Michael D.
Blue Brain Project releases first-ever digital 3D brain cell atlas
The Blue Brain Cell Atlas is like ''going from hand-drawn maps to Google Earth'' -- providing previously unavailable information on major cell types, numbers and positions in all 737 brain regions.
Landmark study reveals no benefit to costly and risky brain cooling after brain injury
A landmark study, led by Monash University researchers, has definitively found that the practice of cooling the body and brain in patients who have recently received a severe traumatic brain injury, has no impact on the patient's long-term outcome.
Brain cells called astrocytes have unexpected role in brain 'plasticity'
Researchers from the Salk Institute have shown that astrocytes -- long-overlooked supportive cells in the brain -- help to enable the brain's plasticity, a new role for astrocytes that was not previously known.
Largest brain study of 62,454 scans identifies drivers of brain aging
In the largest known brain imaging study, scientists from Amen Clinics (Costa Mesa, CA), Google, John's Hopkins University, University of California, Los Angeles and the University of California, San Francisco evaluated 62,454 brain SPECT (single photon emission computed tomography) scans of more than 30,000 individuals from 9 months old to 105 years of age to investigate factors that accelerate brain aging.
More Brain News and Brain Current Events

Trending Science News

Current Coronavirus (COVID-19) News

Top Science Podcasts

We have hand picked the top science podcasts of 2020.
Now Playing: TED Radio Hour

Teaching For Better Humans 2.0
More than test scores or good grades–what do kids need for the future? This hour, TED speakers explore how to help children grow into better humans, both during and after this time of crisis. Guests include educators Richard Culatta and Liz Kleinrock, psychologist Thomas Curran, and writer Jacqueline Woodson.
Now Playing: Science for the People

#556 The Power of Friendship
It's 2020 and times are tough. Maybe some of us are learning about social distancing the hard way. Maybe we just are all a little anxious. No matter what, we could probably use a friend. But what is a friend, exactly? And why do we need them so much? This week host Bethany Brookshire speaks with Lydia Denworth, author of the new book "Friendship: The Evolution, Biology, and Extraordinary Power of Life's Fundamental Bond". This episode is hosted by Bethany Brookshire, science writer from Science News.
Now Playing: Radiolab

Space
One of the most consistent questions we get at the show is from parents who want to know which episodes are kid-friendly and which aren't. So today, we're releasing a separate feed, Radiolab for Kids. To kick it off, we're rerunning an all-time favorite episode: Space. In the 60's, space exploration was an American obsession. This hour, we chart the path from romance to increasing cynicism. We begin with Ann Druyan, widow of Carl Sagan, with a story about the Voyager expedition, true love, and a golden record that travels through space. And astrophysicist Neil de Grasse Tyson explains the Coepernican Principle, and just how insignificant we are. Support Radiolab today at Radiolab.org/donate.