The Ventriloquism Effect Explained
Most of the time, the human brain works like a brilliant machine, seamlessly piecing together the disparate sensory impressions that bombard us from every angle during our waking hours. Our five main senses-sight, hearing, touch, smell and taste-take cues from our environment via specialized cells, sending a constant stream of signals to the brain. The brain usually does a good job of making sense of this cacophony, sometimes too good. When the brain comes across a “glitch” in the information it’s getting from its surroundings, it can work a bit too hard to make sense of it.
Ventriloquism, when we perceive sound coming from an object that is in fact silent, is one example. The classic example of this illusion is a performer on a stage with a puppet sitting on their knee. The performer talks while moving their lips as little as possible, while making much more visible movements with the puppet’s mouth.
The common misconception is that this trick involves the performer somehow “throwing” their voice through a clever trick of the voice box. In ordinary life, the assumption that sounds and movements go together is a very reasonable one. A loud bang and a flash of light that happen at the same time usually are the consequence of a single event-like a fizzle and spark from a plug or an ambulance’s siren and flashing lights.
“Imagine you hear a loud sound, and at exactly the same time, there is an abrupt appearance of something. Then, automatically-because of the coincidence in time-you would tend to associate these two events as originating from the same cause,” says Salvador Soto-Faraco, ICREA Research Professor at the Universitat Pompeu Fabra in Spain, who researches perception and ventriloquism. “The ventriloquist illusion can be made more or less powerful, depending on the timing between flashing light and sound,” says Soto-Faraco.
How the Brain Processes Sensory Information
But in the case of ventriloquism, the stumbling block that results in the illusion comes from the different ways that the senses transmit information. Our sense of hearing, for example, reports information about the location of a sound in a very different way than our sense of sight. For people with good vision, locating an object in space is extremely easy, but trying to pin down exactly where a sound originates from hearing alone-i.e., with eyes closed-is much, much harder.
“Here we have two different sensory modalities giving information about spatial location, but in very different ways,” says Soto-Faraco. “We could think of different currencies. They could not be directly used together. In this conversion calculation, the brain places much more weight on vision than hearing. This is because vision tends to be much more accurate. But this is where the ventriloquist illusion trips the brain up: The sensory information from vision is not reliable. The puppet’s mouth is moving, but there’s no sound coming out. The sense of hearing is overruled by vision.
There is in fact a remarkably simple law for how the brain weighs the information it receives from different sensory sources. It appears to work by a model called “optimal integration theory.” This theory can be described in a mathematical formula for deciding how much reliability to place on any particular bit of sensory information at a given time.
The quirks of the ventriloquism illusion don’t end there. Our brains find ventriloquism so convincing that the sensation of misdirected sound can persist up to half an hour after we stop seeing the trick. Consider the waterfall illusion. “This is explained by neural fatigue. After a while of the neurons for direction of motion being excited, they get tired,” says Soto-Faraco. “So they stop inhibiting the neurons that compete with them. A similar thing happens with the ventriloquism effect.
When someone is exposed to a sound and a flash that have different spatial origins for a while, the brain adapts to this difference. Picture this: First, you see a person at the center of your gaze moving their lips, but the sound is actually coming from your right. Your brain compensates for this, and it seems like the person moving their lips is actually the one speaking. So far, so simple. But then the person in front of you actually starts to speak. Your brain will continue to compensate for the spatial discrepancy.
This illusory aftereffect happens only very briefly, so you have to move fast to measure it in the lab, Soto-Faraco says. “It’s also not clear whether this happens because of neural fatigue, like in the waterfall illusion. Stranger still, scientists have now found that ventriloquism works even when there’s no puppet (or other similar object) involved, a recent study in the journal Psychological Science has found.
In the lab, researchers tend to use a slightly more banal set-up than a ventriloquist with a puppet, instead opting for the simpler cues of a tone and a flashing circle. In this latest study, participants were trained to associate a circle on a screen with the tone. Then researchers recreated the ventriloquism effect by shifting the sound to come from a source away from the circle.
“We were surprised to find that the effects on participants’ perception of acoustic space were almost as strong for imagined stimuli as they were for real visual stimuli,” says study author Christopher Berger of the Karolinska Institute in Sweden in a 2018 press release. As well as shedding light on how our brains process sensory information, researchers believe that studying ventriloquism could help in a number of practical fields.
Understanding this quirk of perception could help to train brain-computer interfaces, and to help stroke patients regain their neural function, the Swedish researchers hope. Similar illusions in different senses are of particular interest in developing virtual reality technology. More closely linking the sensory experiences that define a virtual world is key to advancing the sense of how real the virtual world feels.
“It’s beautiful but very complex,” says Soto-Faraco. Far from a seemingly trivial party trick, the ventriloquism illusion actually says a lot about the fundamental ways our brains make sense of the sensory information we receive from the world around us.
People can’t tell where sound originates if they can’t see movement.
Vision capture aids in the illusion that a dummy is talking in ventriloquism.
An example of visual capture experienced in daily life is the ventriloquism effect. Another popular example of visual capture happens while watching a movie in a theater, and the sound appears to be coming from the actors lips.
There is also a phenomenon known that while crossing a street, an individual can hear the sound of an oncoming car. However, when they look to the left the next car is a few blocks away so it is safe to cross. But when they look to the right, there is a car that is passing them that they did not even notice before. This occurs because the individual attributes the sound of oncoming traffic to the first car because they were unaware of the other, closer car.
In psychology, visual capture is the dominance of vision over other sense modalities in creating a percept. In this process, the visual senses influence the other parts of the somatosensory system, to result in a perceived environment that is not congruent with the actual stimuli. Through this phenomenon, the visual system is able to disregard what other information a different sensory system is conveying, and provide a logical explanation for whatever output the environment provides.
When two sensory stimuli are presented simultaneously, vision is capable of dominating and capturing the other. This occurs as visual cues can distract from other sensations, causing the origin of the stimulus to appear as if it is being produced by the visual cue. Therefore, when an individual is in an environment, and multiple stimuli reach the brain at once, there is a hierarchy that vision will guide the rest of the somatosensory cues to be perceived as though they align with the visual experience, despite where their original source may be.
Research has found that the visual and auditory reflexive spatial orienting are controlled through a common underlying neural substrate. Furthermore, studies have shown that vision has an effect in cognitive neuroscience, and provides for a significant effect when visually attended to.
The thalamus is a section of the brain responsible for relaying sensory and motor signals to the cerebral cortex. As stimuli pass through the thalamus, there are specific regions dedicated to each sense, and therefore is able to sort out the multiple parts of an environment an individual experiences in a given moment. The retina at the back of the eye is what perceives stimuli, allowing them to travel through the occipital tract to the lateral geniculate nucleus (LGN) within the thalamus.
The LGN is located near the medial geniculate nucleus (MGN) which is responsible for organizing auditory stimuli after one hears a specific sound. Because these two systems are closely located to each other, research has shown that this might be where vision is responsible for taking over the perception of an environment and resulting in visual capture.
This phenomenon was first demonstrated by Frenchman J. Tastevin in 1937, after studying the tactile Aristotle illusion in 1937. This illusion produces the sensation of touching two objects by crossing one's fingers and then holding a spherical object between them.
Attention was again tied to visual cues during an experiment conducted by Michael Posner in 1980. By indicating visually in which direction a stimulus will appear, response time will improve (decrease) if the correct direction is attended to. (Conversely, if the indicator is misleading, response time increases.) This ability to attend to a specific direction allows for a faster reaction time, despite the participant not physically shifting their visual focus during the pre-stimulus indicator. The evidence that vision has an impact on reaction time demonstrates that vision has a neurological effect on the attentional process.
A number of studies have demonstrated the visual capture effect. Another example of visual capture comes from Ehrsson, Spense, & Passingham (2004) who used a rubber hand to prove that vision is capable of determining how other senses react. As participants watched a rubber hand be stroked, their hand was also stroked in a similar fashion, allowing the individual to attribute their own sensation to what they were watching rather than what was happening to their own body. Therefore, when the rubber hand was then manipulated, for example hitting it with a hammer, the participant feels an immediate shock and pain as they fear that it is their own hand that is in danger.
A study by Remington, Johnston, & Yantis (1992) found that attention is involuntarily drawn away from a given task when a visual stimulus interferes. In this study, participants were presented with four boxes; they were told that an image would precede a letter that they were to memorize. The conditions were either to attend to the same box, a different one, all four, or to focus on the center. However, even though they were told to not attend to a certain box, the participant was consistently drawn to the image before the letter in all cases, resulting in a longer response time in all conditions except for the same.
The research in visual capture does not all work in the favor of vision being constantly dominant, as Shams, Kamitani, & Shimojo in 2000 found that visual illusion can be induced by sound in a controlled environment. When a flash of light is accompanied by a series of auditory beeps, the results show that the participant views the flash to be a series of flashes corresponding with the beeps.
A phantom limb is the sensation that an amputated limb is still attached. This can cause pain and distress amongst many amputees, and was thought to be incurable. However, in 1998, Vilayanur S. Ramachandran created a mirror box, which allows for an amputee to place their intact limb on one side of the box, and observe their amputated limb by looking at the mirror image of their actual limb. Through visual capture, the visual system is able to override the somatosensory system and send feedback to the brain that the arm is actually okay and not in any specific pain.
The McGurk effect is a phenomenon that occurs when the reception of an auditory stimulus is determined by the visual system. For example, when the syllable “ba” is repeated over and over, and one sees an individual saying this, then the individual is perceived to be saying “ba”. However, when the same audio is played over a person saying the word “fa”, the fact that the utterance is completely forgotten, and the person will hear the word “fa”. This is once again because vision is able to dominate the auditory system and produce a response that is guided strictly by vision.
Understanding visual capture has the potential to lead to numerous benefits in the future. Beyond solving people's pain in phantom limb syndrome, there are numerous potential applications for visual capture. Already, there have been surround-sound systems built to provide unique listening experiences, that “put you right in the middle of the action”.
Here is a table summarizing key aspects of visual capture:
| Aspect | Description |
|---|---|
| Definition | Dominance of vision over other senses in creating a percept. |
| Mechanism | Visual cues distract from other sensations, causing the origin of the stimulus to appear visual. |
| Brain Areas | Thalamus, LGN, MGN. |
| Examples | Ventriloquism effect, watching movies, phantom limb treatment. |
| Applications | Surround-sound systems, virtual reality, treating phantom limb pain. |
Tips for Aspiring Ventriloquists
Learning to throw your voice is a skill that takes time. With dedication and consistent practice, anyone can learn the techniques required to throw their voice. The time it takes varies from person to person. Some may pick it up in a few weeks, while others might take months of practice. All you need is your voice and a willingness to practice. Ventriloquism involves creating the illusion of life by speaking without moving your lips, and often paired with a puppet or character. Throwing your voice is an impressive skill that adds depth and realism to ventriloquism performances.
One of the most frequent mistakes is allowing lip movement while speaking.
How to Improve Your Technique:
- Practice directing attention: Practice directing attention toward this target and pretending it’s speaking instead of you.
- Monitor lip movement: Practice speaking in front of a mirror to monitor your lip movement.
- Diaphragm control: Focus on using your diaphragm instead of your throat. You can also try squeezing your voice.
- Vocal Experimentation: Experiment with different tones and volumes.
- Voice Recording for review Use a voice recorder or your phone to capture your practice sessions. Ask a friend or family member to listen to your practice.
- Start Simple: Begin with simple sounds or single words before attempting full sentences.
- Be patient: Be patient and practice each step thoroughly before moving on to the next.
Building confidence and consistency takes regular practice. Consistency is more effective than occasional long sessions.
A skilled ventriloquist can create a convincing illusion of speech originating from the puppet.
A Practice Exercise
- Record the Line with a Pause: Start by recording a long pause, followed by the line you want the audience to hear.
- Start the Recording and Move Into Position: Play the recording and quickly move to your performance spot.
If you’re ready to take your skills to the next level, explore Maher Studios’ courses and resources for in-depth training and expert advice. Maher Studios offers a variety of courses and resources tailored to ventriloquists of all levels.