The McGurk Effect: How Visual Cues Influence What You Hear
The McGurk effect is a psychoacoustic effect that occurs when the auditory component of speech conflicts with the visual component. This leads to a fused or altered perception of speech sounds.
The McGurk effect shows the integration of auditory and visual information in speech perception and emphasizes the complex nature of how speech is perceived and interpreted.
Origin and Development
The McGurk effect (named after Harry McGurk of McGurk & McDonald, 1976) is a compelling demonstration of how we all use visual speech information. Harry McGurk and John MacDonald developed the effect.
A Practical Example
For example, if someone utters the sound “ba”, but the person shows “ga” with the mouth, it can be perceived as “da”.
iLs co-founder Randall Redfield is repeatedly mouthing the syllable “va,” but the video is dubbed with a sound track of him saying the syllable “ba.” The way your brain deals with the conflict of seeing “va” and hearing “ba” is to override what you hear with what you see. As you noticed when you listened with your eyes closed, inhibiting the visual input allows your brain to accept that the auditory input is indeed “ba” the whole time. But open them again and you can only hear “va” (or “fa”).
Here's how to produce a 'live' demonstration of the McGurk effect:
- Have an observer face you and keep looking at your face.
- Have another person stand behind you so the observer can't see their face.
- Starting synchronously, repeatedly mouth the word 'vase' (silently) while the person behind you repeats the word 'base' out loud - you can achieve synchronization by counting down '3, 2, 1. . vase, vase, vase, etc.
- After about 8 repetitions, stop and ask the observer what they 'hear' - they should 'hear' vase.
- Now do the same thing, and this time tell the observer to shut their eyes after a few repetitions.
- They should hear 'base' with their eyes shut.
- The observer can try opening and shutting their eyes, and what they 'hear' should change from 'vase' to 'base'.
Factors Influencing the McGurk Effect
This effect is influenced by several factors, including neurological and psychological conditions.
People with brain damage, especially damage in the areas of the brain associated with speech perception and multi-sensory integration, are more likely to show an altered response to the McGurk effect. Damages in certain parts of the brain can disrupt the integration of auditory and visual information, resulting in a reduced or modified McGurk effect.
People with an autism spectrum disorder also show different reactions when it comes to the McGurk effect.
Alzheimer’s disease primarily affects memory and cognitive brain function, but can also affect speech perception.
Schizophrenia is a mental disorder characterized by disturbances in perception, thought, and social functioning. People with schizophrenia often show impaired multi-sensory integration, including abnormalities in audio-visual integration during speech perception.
Generality of the McGurk Effect
The effect works on perceivers with all language backgrounds (e.g., Massaro, Cohen, Gesi, Heredia, & Tsuzaki, 1993; Sekiyama. & Tokhura, 1993).
The effect works on young infants (Rosenblum, Schmuckler, & Johnson, 1997).
The effect works when the visual and auditory components are from speakers of different genders (Green, Kuhl, Meltzoff, & Stevens, 1991).
The effect works with highly reduced face images (Rosenblum & Saldaña, 1996).
The effect works when observers are unaware that they are looking at a face (Rosenblum & Saldaña, 1996).
The effect works when observers touch-rather than look-at the face (Fowler & Dekle, 1991).
The effect works less well with vowels than consonants (Summerfield & McGrath, 1984).
The effect works less well with non-speech pluck & bow stimuli (Saldaña & Rosenblum, 1994).
The effect works better with some consonant combinations than others (e.g, McGurk & MacDonald).
Criticism and Controversies
While the scientific basis of the McGurk effect is solid, there are some critical points and controversies surrounding the phenomenon. They argue that the perception need not be a fusion of both modalities, but rather a visual distortion of the auditory signal.
There are also some methodological considerations when investigating the McGurk effect. It is important to keep these factors in mind.
Applications
The effect and research on it also has applications in fields such as psychology, linguistics and communication research.