The McGurk Effect Explained: How Visual Cues Influence Speech Perception
The McGurk effect is a psychoacoustic effect that occurs when the auditory component of speech conflicts with the visual component. This leads to a fused or altered perception of speech sounds. This article explains the McGurk effect, developed by Harry McGurk and John MacDonald in a practical way. The article starts with the definition of the concept, followed by information about its origin and a practical example. You can also read about the different factors that influence this effect and what criticism there is of the phenomenon.
Perhaps the most famous is the McGurk effect, a phenomenon named for Harry McGurk, half of the research duo who first wrote about the auditory illusion back in the 1970s.
Diagram illustrating the McGurk effect.
Origin and Definition
McGurk and MacDonald (1976) reported a powerful multisensory illusion occurring with audiovisual speech. They recorded a voice articulating a consonant and dubbed it with a face articulating another consonant. Even though the acoustic speech signal was well recognized alone, it was heard as another consonant after dubbing with incongruent visual speech. The illusion has been termed the McGurk effect. It has been replicated many times, and it has sparked an abundance of research. The reason for the great impact is that this is a striking demonstration of multisensory integration. It shows that auditory and visual information is merged into a unified, integrated percept.
The McGurk effect shows the integration of auditory and visual information in speech perception and emphasizes the complex nature of how speech is perceived and interpreted. They conducted several experiments, including one where participants were shown a video of a person articulating 1 syllable.
For example, if someone utters the sound “ba”, but the person shows “ga” with the mouth, it can be perceived as “da”.
Dr. Kevin Franck, director of audiology at Massachusetts Eye and Ear, says it all goes back to the brain’s inherent desire to assign meaning to confusing stimuli, like an ambiguous sound. “Your brain’s just trying to make sense of what’s coming at it,” he says. “It hates it when things don’t make sense.”
In terms of the McGurk effect, the brain relies on the eyes to settle the confusion coming at the ears - particularly when they’re seeing something as clear as a pair of lips distinctly mouthing a certain sound or word. “I have one ambiguous signal, which is the actual thing you hear, but then I say, ‘Oh, I’ve got another input, which is very clear. I trust my eyes now, because my ears aren’t sure,'” Franck explains. “It’s just taking all of these different inputs and making sense out of the mess.”
Once you know what’s going on, Franck says, you’ll likely be able to tell that you’re hearing the same thing in both instances - but left to its own devices, the brain will try to eliminate confusion every time.
“The brain just likes things to be consistent. It wants things to be neat and tidy,” Franck says.
Variants of the McGurk Effect
Here I shall make two main claims regarding the definition and interpretation of the McGurk effect since they bear relevance to its use as a measure of multisensory integration. First, the McGurk effect should be defined as a categorical change in auditory perception induced by incongruent visual speech, resulting in a single percept of hearing something other than what the voice is saying.
There are many variants of the McGurk effect (McGurk and MacDonald, 1976; MacDonald and McGurk, 1978)1. The best-known case is when dubbing a voice saying [b] onto a face articulating [g] results in hearing [d]. This is called the fusion effect since the percept differs from the acoustic and visual components.
Many researchers have defined the McGurk effect exclusively as the fusion effect because here integration results in the perception of a third consonant, obviously merging information from audition and vision (van Wassenhove et al., 2007; Keil et al., 2012; Setti et al., 2013). This definition ignores the fact that other incongruent audiovisual stimuli produce different types of percepts. For example, a reverse combination of these consonants, A[g]V[b], is heard as [bg], i.e., the visual and auditory components one after the other. There are other pairings, which result in hearing according to the visual component, e.g., acoustic [b] presented with visual [d] is heard as [d].
Here my first claim is that the definition of the McGurk effect should be that an acoustic utterance is heard as another utterance when presented with discrepant visual articulation. This definition includes all variants of the illusion, and it has been used by MacDonald and McGurk (1978) themselves, as well as by several others (e.g., Rosenblum and Saldaña, 1996; Brancazio et al., 2003). The different variants of the McGurk effect represent the outcome of audiovisual integration. When integration takes place, it results in a unified percept, without access to the individual components that contributed to the percept.
One challenge with this interpretation of the McGurk effect is that it is impossible to be certain that the responses the observer gives correspond to the actual percepts. The real McGurk effect arises due to multisensory integration, resulting in an altered auditory percept. However, if integration does not occur, the observer can perceive the components separately and may choose to respond either according to what he heard or according to what he saw. This is one reason why the fusion effect is so attractive: If the observer reports a percept that differs from both stimulus components, he does not seem to rely on either modality alone, but instead really fuse the information from both.
The second main claim here is that the perception of the acoustic and visual stimulus components has to be taken into account when interpreting the McGurk effect. This issue has been elaborated previously in the extensive work by Massaro and colleagues (Massaro, 1998) and others (Sekiyama and Tohkura, 1991; Green and Norrix, 1997; Jiang and Bernstein, 2011). In general, the strength of the McGurk effect is taken to increase when the proportion of responses according to the acoustic component decreases and/or when the proportion of fusion responses increases. That is, the McGurk effect for stimulus A[b]V[g] is considered stronger when fewer B responses and/or more D responses are given. This is often an adequate way to measure the strength of the McGurk effect-if one keeps in mind that it implicitly assumes that perception of the acoustic and visual components is accurate (or at least constant across conditions that are compared).
The fusion effect provides a prime example of this caveat. It has been interpreted to mean that acoustic and visual information is integrated to produce a novel, intermediate percept. For example, when A[b]V[g] is heard as [d], the percept is thought to emerge due to fusion of the features (for the place of articulation) provided via audition (bilabial) and vision (velar), so that a different, intermediate consonant (alveolar) is perceived (van Wassenhove, 2013). However, already McGurk and MacDonald (1976) themselves wrote that “lip movements for [ga] are frequently misread as [da],” even though they did not measure speechreading performance, unfortunately. The omission of the unisensory visual condition in the original study is one factor that has contributed to the strong status of the fusion effect as the only real McGurk effect, reflecting true integration.
To demonstrate the contribution of the unisensory components more explicitly, I'll take two examples of my research, in which fusion-type stimuli produced different percepts depending on the clarity of the visual component. In one study, a McGurk stimulus A[epe]V[eke] was mainly heard as a fusion [ete] (Tiippana et al., 2004). This reflected the fact that in a visual-only identification task, the visual [eke] was confused with [ete] (42% K responses and 45% T responses to visual [eke]). In another study, a McGurk stimulus A[apa]V[aka] was mainly heard as [aka], and this could be traced back to the fact that in a visual-only identification task, the visual [aka] was clearly distinguishable from [ata], and thus recognized very accurately (100% correct in typical adults; Saalasti et al., 2012; but note the deviant behavior of individuals with Asperger syndrome). Thus, even though the McGurk stimuli were of a fusion type in both studies, their perception differed depending largely on the clarity of the visual components.
Exactly how to take the properties of the unisensory components into account in multisensory perception of speech is beyond this paper. Addressing this issue in detail requires carefully designed experimental studies (Bertelson et al., 2003; Alsius et al., 2005), computational modeling (Massaro, 1998; Schwartz, 2010), and investigation of the underlying brain mechanisms (Sams et al., 1991; Skipper et al., 2007).
An example of the McGurk effect in action.
Factors Influencing the McGurk Effect
This effect is influenced by several factors, including Alzheimer’s, schizophrenia, autism and brain damage.
People with brain damage, especially damage in the areas of the brain associated with speech perception and multi-sensory integration, are more likely to show an altered response to the McGurk effect. Damages in certain parts of the brain can disrupt the integration of auditory and visual information, resulting in a reduced or modified McGurk effect.
People with an autism spectrum disorder also show different reactions when it comes to the McGurk effect. Schizophrenia is a mental disorder characterized by disturbances in perception, thought, and social functioning. People with schizophrenia often show impaired multi-sensory integration, including abnormalities in audio-visual integration during speech perception.
Alzheimer’s disease primarily affects memory and cognitive brain function, but can also affect speech perception.
The effect and research on it also has applications in fields such as psychology, linguistics and communication research.
Criticism of the McGurk Effect
While the scientific basis of the McGurk effect is solid, there are some critical points and controversies surrounding the phenomenon. They argue that the perception need not be a fusion of both modalities, but rather a visual distortion of the auditory signal. There are also some methodological considerations when investigating the McGurk effect. It is important to keep these factors in mind.
During experiments, when the task is to report what was heard, the observer reports the conscious auditory percept evoked by the audiovisual stimulus. If there is no multisensory integration or interaction, the percept is identical for the audiovisual stimulus and the auditory component presented alone. If there is audiovisual integration, the conscious auditory percept changes. To which extent visual input influences the percept depends on how coherent and reliable information each modality provides. This perceptual process is the same for audiovisual speech-be it natural, congruent audiovisual speech or artificial, incongruent McGurk speech stimuli. The outcome is the conscious auditory percept. Depending on the relative weighting of audition and vision, the outcome for McGurk stimuli can range from hearing according to the acoustic component (when audition is more reliable than vision) to fusion and combination percepts (when both modalities are informative to some extent) to hearing according to the visual component (when vision is more reliable than audition). Congruent audiovisual speech is treated no differently, showing visual influence when the auditory reliability decreases.
The McGurk effect is an excellent tool to investigate multisensory integration in speech perception.