The McGurk Effect: How Seeing Lips Can Change What You Hear

The McGurk effect, named after Harry McGurk and John MacDonald, is a fascinating psychoacoustic phenomenon that demonstrates how visual information influences our perception of speech. Described in their 1976 paper "Hearing Lips and Seeing Voices," this effect reveals that what we hear is not solely based on auditory input, but also on the visual cues we receive from observing someone speaking.

In essence, the McGurk effect is a multisensory illusion where the auditory component of a sound is paired with the visual component of another sound, leading to the perception of a third, different sound. This highlights the brain's remarkable ability to integrate information from different senses to form a unified and often altered percept.

The McGurk effect shows the integration of auditory and visual information in speech perception and emphasizes the complex nature of how speech is perceived and interpreted.

McGurk Effect Demonstration

Understanding the McGurk Effect

The McGurk effect is a compelling demonstration of how we all use visual speech information. The effect shows that we can't help but integrate visual speech into what we 'hear'.

The key to this multisensory illusion is a mismatch between what you see and what you hear. The way your brain deals with the conflict of seeing and hearing is to override what you hear with what you see.

First, the McGurk effect should be defined as a categorical change in auditory perception induced by incongruent visual speech, resulting in a single percept of hearing something other than what the voice is saying. When integration takes place, it results in a unified percept, without access to the individual components that contributed to the percept.

Here is an example of how the effect works:

• If someone utters the sound “ba”, but the person shows “ga” with the mouth, it can be perceived as “da”.

The crazy thing is that the effect works even when you know what is happening - and it persists for as long as you experiment with it.

The McGurk Effect helps us to understand how our senses are integrated and what happens when they conflict. Depending on which sensory modality is providing more salient information, our brain will combine or override what the other modality is experiencing.

It is essential for the senses to work together in processing most information.

The different variants of the McGurk effect represent the outcome of audiovisual integration.

One challenge with this interpretation of the McGurk effect is that it is impossible to be certain that the responses the observer gives correspond to the actual percepts.

The real McGurk effect arises due to multisensory integration, resulting in an altered auditory percept.

However, if integration does not occur, the observer can perceive the components separately and may choose to respond either according to what he heard or according to what he saw.

The second main claim here is that the perception of the acoustic and visual stimulus components has to be taken into account when interpreting the McGurk effect. During experiments, when the task is to report what was heard, the observer reports the conscious auditory percept evoked by the audiovisual stimulus.

If there is no multisensory integration or interaction, the percept is identical for the audiovisual stimulus and the auditory component presented alone.

The fusion effect provides a prime example of this caveat. It has been interpreted to mean that acoustic and visual information is integrated to produce a novel, intermediate percept.

Depending on the relative weighting of audition and vision, the outcome for McGurk stimuli can range from hearing according to the acoustic component (when audition is more reliable than vision) to fusion and combination percepts (when both modalities are informative to some extent) to hearing according to the visual component (when vision is more reliable than audition).

Congruent audiovisual speech is treated no differently, showing visual influence when the auditory reliability decreases.

The McGurk effect is an excellent tool to investigate multisensory integration in speech perception.

Factors Influencing the McGurk Effect

This effect is influenced by several factors, including neurological conditions and individual differences.

People with brain damage, especially damage in the areas of the brain associated with speech perception and multi-sensory integration, are more likely to show an altered response to the McGurk effect. Damages in certain parts of the brain can disrupt the integration of auditory and visual information, resulting in a reduced or modified McGurk effect.

People with an autism spectrum disorder also show different reactions when it comes to the McGurk effect. Alzheimer’s disease primarily affects memory and cognitive brain function, but can also affect speech perception.

Schizophrenia is a mental disorder characterized by disturbances in perception, thought, and social functioning. People with schizophrenia often show impaired multi-sensory integration, including abnormalities in audio-visual integration during speech perception.

The effect works on perceivers with all language backgrounds.

The effect works on young infants.

The effect works when the visual and auditory components are from speakers of different genders.

The effect works with highly reduced face images.

The effect works when observers are unaware that they are looking at a face.

The effect works when observers touch-rather than look-at the face.

The effect works less well with vowels than consonants.

The effect works less well with nonspeech pluck & bow stimuli.

The effect works better with some consonant combinations than others.

Criticism and Controversies

While the scientific basis of the McGurk effect is solid, there are some critical points and controversies surrounding the phenomenon. They argue that the perception need not be a fusion of both modalities, but rather a visual distortion of the auditory signal. There are also some methodological considerations when investigating the McGurk effect. It is important to keep these factors in mind.

The omission of the unisensory visual condition in the original study is one factor that has contributed to the strong status of the fusion effect as the only real McGurk effect, reflecting true integration.

Here I shall make two main claims regarding the definition and interpretation of the McGurk effect since they bear relevance to its use as a measure of multisensory integration.

Practical Demonstration of the McGurk Effect

To produce a 'live' demonstration of the McGurk effect (you'll need two other people besides yourself):

1. Have an observer face you and keep looking at your face.
2. Have another person stand behind you so the observer can't see their face.
3. Starting synchronously, repeatedly mouth the word 'vase' (silently) while the person behind you repeats the word 'base' out loud - you can achieve synchronization by counting down '3, 2, 1. . vase, vase, vase, etc.
4. After about 8 repetitions, stop and ask the observer what they 'hear' - they should 'hear' vase.
5. Now do the same thing, and this time tell the observer to shut their eyes after a few repetitions.
6. They should hear 'base' with their eyes shut.
7. The observer can try opening and shutting their eyes, and what they 'hear' should change from 'vase' to 'base'.

Tips on Making Your Own McGurk Stimuli

• Audiovisual dubbing can be achieved by using two videotape players or digitizing stimuli onto a computer and using software to mix the audio and video components.
• The quality of the auditory channel should be good, but the quality of the visual channel can be fair without much loss in the effect.
• The auditory and visual components should be synchronized so that the sound of the syllable seems to be coming from the visible mouth. However, the components do not have to be perfectly synchronized for the effect to work.
• The syllable combinations used in the above demonstration are known to be especially strong.

Table of Factors Influencing the McGurk Effect

Factor	Description
Neurological Conditions	Brain damage, autism spectrum disorder, Alzheimer’s, schizophrenia
Language Background	Effect present across different language backgrounds
Age	Effect present in young infants
Speaker Gender	Effect works with different genders
Visual Clarity	Effect works with reduced face images
Observer Awareness	Effect works even when observers are unaware
Sensory Input	Touch can influence the effect
Speech Components	Works better with consonants than vowels
Stimulus Type	Works better with speech than non-speech stimuli

The effect and research on it also has applications in fields such as psychology, linguistics and communication research.

The McGurk effect shows that visual articulatory information is integrated into our perception of speech automatically and unconsciously. The syllable that we perceive depends on the strength of the auditory and visual information, and whether some compromise can be achieved. Regardless, integration of the discrepant audiovisual speech syllables is effortless and mandatory.

Our speech function makes use of all types of relevant information, regardless of the modality. In fact, there is some evidence that the brain treats visual speech information as if it is auditory speech.

The illusion has been termed the McGurk effect. It has been replicated many times, and it has sparked an abundance of research.

The reason for the great impact is that this is a striking demonstration of multisensory integration. It shows that auditory and visual information is merged into a unified, integrated percept.