Read My Lips: Using Multiple Senses in Speech Perception

February 12, 2009

When someone speaks to you, do you see what they are saying? We tend to think of speech as being something we hear, but recent studies suggest that we use a variety of senses for speech perception - that the brain treats speech as something we hear, see and even feel. In a new report in Current Directions in Psychological Science, a journal of the Association for Psychological Science, psychologist Lawrence Rosenblum describes research examining how our different senses blend together to help us perceive speech.

We receive a lot of our speech information via visual cues, such as lip-reading, and this type of visual speech occurs throughout all cultures. And it is not just information from lips- when someone is speaking to us, we will also note movements of the teeth, tongue and other non-mouth facial features. It's likely that human speech perception has evolved to integrate many senses together. Put in another way, speech is not meant to be just heard, but also to be seen.

The McGurk Effect is a well-characterized example of the integration between what we see and what we hear when someone is speaking to us. This phenomenon occurs when a sound (such as a syllable or word) is dubbed with a video showing a face making a different sound. For example, the audio may be playing "ba," while the face looks as though it is saying "va." When confronted with this, we will usually hear "va" or a combination of the two sounds, such as "da." Interestingly, when study participants are aware of the dubbing or told to concentrate only on the audio, the McGurk Effect still occurs. Rosenblum suggests that this is evidence that once senses are integrated together, it is not possible to separate them.

Recent studies indicate that this integration occurs very early in the speech process, even before phonemes (the basic units of speech) are established. Rosenblum suggests that physical movement of speech (that is, our mouths and lips moving) create acoustic and visual signals which have a similar form. He argues that as far as the speech brain is concerned, the auditory and visual information are never really separate. This could explain why we integrate speech so readily and in such a way that the audio and visual speech signals become indistinguishable from one another.

Rosenblum concludes that visual-speech research has a number of clinical implications, especially in the areas of autism, brain injury and schizophrenia and that "rehabilitation programs in each of these domains have incorporated visual-speech stimuli."

Association for Psychological Science

---

---

	The Perception of Speech: from sound to meaning by Brian Moore (Author), Lorraine Tyler (Author), William Marslen-Wilson (Author) Humans are unique in being able to convey such complex information using speech, and in the range of ideas, thoughts, and emotions that can be expressed. This crossdisciplinary book explores how the motor gestures of a speaker are transformed to sounds and how those are mapped onto meaning in the comprehension of spoken language.
	The Handbook of Speech Perception (Blackwell Handbooks in Linguistics) by David Pisoni (Editor), Robert Remez (Editor) The Handbook of Speech Perception is a collection of forward-looking articles that offer a summary of the technical and theoretical accomplishments in this vital area of research on language. Now available in paperback, this uniquely comprehensive companion brings together in one volume the latest research conducted in speech perception Contains original contributions by leading researchers in the field Illustrates technical and theoretical accomplishments and challenges across the field of research and language Adds to a growing understanding of the far-reaching relevance of speech perception in the fields of phonetics, audiology and speech science, cognitive science, experimental psychology, behavioral neuroscience, computer science, and electrical engineering, among...
	Speech Science Primer: Physiology, Acoustics, and Perception of Speech by Lawrence J. Raphael (Author), Gloria J. Borden (Author), Katherine S. Harris (Author) This comprehensive textbook provides a clear, reader-friendly writing style, serves as an introduction to speech science, and covers basic information on acoustics, the acoustic analysis of speech, speech anatomy and physiology, and speech perception. The Fifth Edition also includes topics such as research methodology, speech motor control, and history/evolution of speech science.Features include: New material on speech respiration and expanded information on cross-language speech issues New organization and shortened chapters aid reading and comprehension Information on acoustics, speech physiology, and speech perception is integrated to create a unified treatment of speech science An explanation of the normal communication process serves as a basis for comparison with...
	Preclinical Speech Science: Anatomy, Physiology, Acoustics, Perception by Thomas J. Hixon (Author), Gary Weismer (Author), Jeannette D. Hoit (Author) Written in a user-friendly style by three distinguished scientists/editors/clinicians, who have taught the course to thousands of students at premier educational programs, the scope of the book is all-inclusive, comprehensively covering anatomy, physiology, acoustics, perception, and swallowing. The book represents a quantum leap forward toward preparing speech-language pathologists in training to understand importance of the science that underpins their work and provides framework for approaching the evaluation and management of each of their future clients. This text provides all that instructors need to ensure that their students are fully ready for the clinical practicum training. Describes scientific principles explicitly, and in translational terms that emphasize their relevance...
	Speech Science Primer: Physiology, Acoustics, and Perception of Speech by Lawrence J. Raphael (Author), Gloria J. Borden (Author), Katherine S. Harris (Author) Written in a clear, reader-friendly style, Speech Science Primer serves as an introduction to speech science and covers basic information on acoustics, the acoustic analysis of speech, speech anatomy and physiology, and speech perception. It also includes topics such as research methodology, speech motor control, and history/evolution of speech science. With its reader-friendly content and valuable online resources, Speech Science Primer: Physiology, Acoustics, and Perception of Speech, Sixth Edition is an ideal text for beginning speech pathology and audiology students and faculty.
	A Basic Introduction to Speech Perception (Cultural Heritage and Contemporary Change) by PhD Jack H Ryalls (Author) Introductory textbook for undergraduate speech-language pathology or audiology students on speech perception.
	Voice and Speech Quality Perception: Assessment and Evaluation (Signals and Communication Technology) by Ute Jekosch (Author) Foundations of Voice and Speech Quality Perception starts out with the fundamental question of: "How do listeners perceive voice and speech quality and how can these processes be modeled?" Any quantitative answers require measurements. This is natural for physical quantities but harder to imagine for perceptual measurands. This book approaches the problem by actually identifying major perceptual dimensions of voice and speech quality perception, defining units wherever possible and offering paradigms to position these dimensions into a structural skeleton of perceptual speech and voice quality. The emphasis is placed on voice and speech quality assessment of systems in artificial scenarios. Many scientific fields are involved. This book bridges the gap between two quite diverse fields,...
	Speech perception and the McGurk effect---a cross cultural study using event-related potentials. by Jia Wu (Author) Previous research has indicated the important role of visual information in the speech perception process. These studies have elucidated the areas of the brain involved in the processing of audiovisual stimuli. The McGurk effect, an audiovisual illusion, has been demonstrated to be a useful tool in the study of audiovisual integration. Brain imaging research suggests that the McGurk effect is modulated by brain structures in the Superior Temporal Gyrus, Supplemental Temporal Cortex and Broadman area 41. Electrophysiological studies suggest that the McGurk effect generates a different brainwave than the standard audiovisual congruent condition in frontal, central and parietal regions among N1, P2, N300 and later ERP components. The magnitude of the McGurk effect is influenced by lexical...
	Speech and Audio Signal Processing: Processing and Perception of Speech and Music by Ben Gold (Author), Nelson Morgan (Author), Dan Ellis (Author) When Speech and Audio Signal Processing published in 1999, it stood out from its competition in its breadth of coverage and its accessible, intutiont-based style.  This book was aimed at individual students and engineers excited about the broad span of audio processing and curious to understand the available techniques.  Since then, with the advent of the iPod in 2001, the field of digital audio and music has exploded, leading to a much greater interest in the technical aspects of audio processing. This Second Edition will update and revise the original book to augment it with new material describing both the enabling technologies of digital music distribution (most significantly the MP3) and a range of exciting new research areas in automatic music content processing (such as...
	Discovering Speech, Words, and Mind by Dani Byrd (Author), Toben H. Mintz (Author) Written in a lively style, Discovering Speech, Words, and Mind applies a scientific approach to the study of various aspects of speech, using everyday examples to introduce the beginning student to the world of language and cognition. An accessible introduction to the fundamentals of speech production, speech perception, word-formation, language acquisition and speech disordersConsiders how the informational content of the speech signal relates to phonological units – connecting the three areas of speech, words, and mindFocuses on speech production and recognition at the word-level and below, and includes sign languagesWritten in a highly accessible style for students with no background in linguistics or psychologyPacked with numerous student-friendly features, including engaging...