Researchers reconstruct spoken words as processed in nonhuman primate brains

December 13, 2019

PROVIDENCE, R.I. [Brown University] -- A team of Brown University researchers has used a brain-computer interface to reconstruct English words from neural signals recorded in the brains of nonhuman primates. The research, published in the journal Nature Communications Biology, could be a step toward developing brain implants that may help people with hearing loss, the researchers say.

"What we've done is to record the complex patterns of neural excitation in the secondary auditory cortex associated with primates' hearing specific words," said Arto Nurmikko, a professor in Brown's School of Engineering, a research associate in Brown's Carney Institute for Brain Science and senior author of the study. "We then use that neural data to reconstruct the sound of those words with high fidelity.

"The overarching goal is to better understand how sound is processed in the primate brain," Nurmikko added, "which could ultimately lead to new types of neural prosthetics."

The brain systems involved in the initial processing of sound are similar in humans and non-human primates. The first level of processing, which happens in what's called the primary auditory cortex, sorts sounds according to attributes like pitch or tone. The signal then moves to the secondary auditory cortex, where it's processed further. When someone is listening to spoken words, for example, this is where the sounds are classified by phonemes -- the simplest features that enable us to distinguish one word from another. After that, the information is sent to other parts of the brain for the processing that enables human comprehension of speech.

But because that early-stage processing of sound is similar in humans and non-human primates, learning how primates process the words they hear is useful, even though they likely don't understand what those words mean.

For the study, two pea-sized implants with 96-channel microelectrode arrays recorded the activity of neurons while rhesus macaques listened to recordings of individual English words and macaque calls. In this case, the macaques heard fairly simple one- or two-syllable words -- "tree," "good," "north," "cricket" and "program."

The researchers processed the neural recordings using computer algorithms specifically developed to recognize neural patterns associated with particular words. From there, the neural data could be translated back into computer-generated speech. Finally, the team used several metrics to evaluate how closely the reconstructed speech matched the original spoken word that the macaque heard. The research showed the recorded neural data produced high-fidelity reconstructions that were clear to a human listener.

The use of multielectrode arrays to record such complex auditory information was a first, the researchers say.

"Previously, work had gathered data from the secondary auditory cortex with single electrodes, but as far as we know this is the first multielectrode recording from this part of the brain," Nurmikko said. "Essentially we have nearly 200 microscopic listening posts that can give us the richness and higher resolution of data which is required."

One of the goals of the study, for which doctoral student Jihun Lee led the experiments, was to test whether any particular decoding model algorithm performed better than others. The research, in collaboration with Wilson Truccolo, a computational neuroscience expert, showed that recurrent neural networks (RNNs) -- a type of machine learning algorithm often used in computerized language translation -- produced the highest-fidelity reconstructions. The RNNs substantially outperformed more traditional algorithms that have been shown to be effective in decoding neural data from other parts of the brain.

Christopher Heelan, a research associate at Brown and co-lead author of the study, thinks the success of the RNNs comes from their flexibility, which is important in decoding complex auditory information.

"More traditional algorithms used for neural decoding make strong assumptions about how the brain encodes information, and that limits the ability of those algorithms to model the neural data," said Heelan, who developed the computational toolkit for the study. "Neural networks make weaker assumptions and have more parameters allowing them to learn complicated relationships between the neural data and the experimental task."

Ultimately, the researchers hope, this kind of research could aid in developing neural implants the may aid in restoring peoples' hearing.

"The aspirational scenario is that we develop systems that bypass much of the auditory apparatus and go directly into the brain," Nurmikko said. "The same microelectrodes we used to record neural activity in this study may one day be used to deliver small amounts of electrical current in patterns that give people the perception of having heard specific sounds."
The research was supported by the U.S. Defense Advanced Research Projects Agency (N66001-17-C-4013) and a private gift to Brown. Other coauthors on the paper were Ronan O'Shea, Laurie Lynch and David Brandman.

Brown University

Related Brain Articles from Brightsurf:

Glioblastoma nanomedicine crosses into brain in mice, eradicates recurring brain cancer
A new synthetic protein nanoparticle capable of slipping past the nearly impermeable blood-brain barrier in mice could deliver cancer-killing drugs directly to malignant brain tumors, new research from the University of Michigan shows.

Children with asymptomatic brain bleeds as newborns show normal brain development at age 2
A study by UNC researchers finds that neurodevelopmental scores and gray matter volumes at age two years did not differ between children who had MRI-confirmed asymptomatic subdural hemorrhages when they were neonates, compared to children with no history of subdural hemorrhage.

New model of human brain 'conversations' could inform research on brain disease, cognition
A team of Indiana University neuroscientists has built a new model of human brain networks that sheds light on how the brain functions.

Human brain size gene triggers bigger brain in monkeys
Dresden and Japanese researchers show that a human-specific gene causes a larger neocortex in the common marmoset, a non-human primate.

Unique insight into development of the human brain: Model of the early embryonic brain
Stem cell researchers from the University of Copenhagen have designed a model of an early embryonic brain.

An optical brain-to-brain interface supports information exchange for locomotion control
Chinese researchers established an optical BtBI that supports rapid information transmission for precise locomotion control, thus providing a proof-of-principle demonstration of fast BtBI for real-time behavioral control.

Transplanting human nerve cells into a mouse brain reveals how they wire into brain circuits
A team of researchers led by Pierre Vanderhaeghen and Vincent Bonin (VIB-KU Leuven, Université libre de Bruxelles and NERF) showed how human nerve cells can develop at their own pace, and form highly precise connections with the surrounding mouse brain cells.

Brain scans reveal how the human brain compensates when one hemisphere is removed
Researchers studying six adults who had one of their brain hemispheres removed during childhood to reduce epileptic seizures found that the remaining half of the brain formed unusually strong connections between different functional brain networks, which potentially help the body to function as if the brain were intact.

Alcohol byproduct contributes to brain chemistry changes in specific brain regions
Study of mouse models provides clear implications for new targets to treat alcohol use disorder and fetal alcohol syndrome.

Scientists predict the areas of the brain to stimulate transitions between different brain states
Using a computer model of the brain, Gustavo Deco, director of the Center for Brain and Cognition, and Josephine Cruzat, a member of his team, together with a group of international collaborators, have developed an innovative method published in Proceedings of the National Academy of Sciences on Sept.

Read More: Brain News and Brain Current Events is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to