The Ventriloquism Effect: A Deep Dive into the Psychology of Illusion
Most of the time, the human brain works like a brilliant machine, seamlessly piecing together the disparate sensory impressions that bombard us from every angle during our waking hours. Our five main senses-sight, hearing, touch, smell and taste-take cues from our environment via specialized cells, sending a constant stream of signals to the brain. The brain usually does a good job of making sense of this cacophony, sometimes too good. When the brain comes across a “glitch” in the information it’s getting from its surroundings, it can work a bit too hard to make sense of it.
The "ventriloquism effect" describes an illusory phenomenon where the perceived location of an auditory stimulus is pulled toward the location of a visual stimulus. Ventriloquism is more than just a fun performance-it’s a fascinating psychological illusion. This article will dig into the psychology behind ventriloquism.
Ventriloquism works by tricking your brain into connecting sound with a visual cue. When you watch a ventriloquist perform, your brain focuses on the puppet’s moving mouth and assumes the sound is coming from it, even though the ventriloquist is speaking. A simple example of this effect is watching a movie. When characters speak, you don’t focus on the speakers in the theater. Ventriloquism uses the same psychological principle but takes it a step further by adding humor, personality, and timing.
The psychology of ventriloquism is partly based in the history of the art. Ventriloquists use the expression and suppression of their own and the puppet's mouth movements as well the direction of their respective eye gaze to maximize the illusion. While the puppet's often exaggerated mouth movements have been demonstrated to enhance the ventriloquism effect, the contribution of direct eye gaze remains unknown.
In Experiment 1, participants viewed an image of a person's face while hearing a temporally synchronous recording of a voice originating from different locations on the azimuthal plane. The eyes of the facial stimuli were either looking directly at participants or were closed. Participants were more likely to misperceive the location of a range of voice locations as coming from a central position when the eye gaze of the facial stimuli were directed toward them. Thus, direct gaze enhances the ventriloquist effect by attracting participants' perception of the voice locations toward the location of the face. In an exploratory analysis, we furthermore found no evidence for an other-race effect between White vs Asian listeners.
In Experiment 2, we replicated the effect of direct eye gaze on the ventriloquism effect, also showing that faces per se attract perceived sound locations compared with audio-only sound localization.
The classic example of this illusion is a performer on a stage with a puppet sitting on their knee. The performer talks while moving their lips as little as possible, while making much more visible movements with the puppet’s mouth. The common misconception is that this trick involves the performer somehow “throwing” their voice through a clever trick of the voice box.
“Imagine you hear a loud sound, and at exactly the same time, there is an abrupt appearance of something. Then, automatically-because of the coincidence in time-you would tend to associate these two events as originating from the same cause,” says Salvador Soto-Faraco, ICREA Research Professor at the Universitat Pompeu Fabra in Spain, who researches perception and ventriloquism. In ordinary life, the assumption that sounds and movements go together is a very reasonable one. A loud bang and a flash of light that happen at the same time usually are the consequence of a single event-like a fizzle and spark from a plug or an ambulance’s siren and flashing lights.
“The ventriloquist illusion can be made more or less powerful, depending on the timing between flashing light and sound,” says Soto-Faraco. But in the case of ventriloquism, the stumbling block that results in the illusion comes from the different ways that the senses transmit information. Our sense of hearing, for example, reports information about the location of a sound in a very different way than our sense of sight. For people with good vision, locating an object in space is extremely easy, but trying to pin down exactly where a sound originates from hearing alone-i.e., with eyes closed-is much, much harder.
“Here we have two different sensory modalities giving information about spatial location, but in very different ways,” says Soto-Faraco. “We could think of different currencies. They could not be directly used together. In this conversion calculation, the brain places much more weight on vision than hearing. This is because vision tends to be much more accurate. But this is where the ventriloquist illusion trips the brain up: The sensory information from vision is not reliable. The puppet’s mouth is moving, but there’s no sound coming out. The sense of hearing is overruled by vision.
There is in fact a remarkably simple law for how the brain weighs the information it receives from different sensory sources. It appears to work by a model called “optimal integration theory.” This theory can be described in a mathematical formula for deciding how much reliability to place on any particular bit of sensory information at a given time.
The brain is amazing at combining information from different senses. The puppet becomes the focal point because its movements match the sound, even if the sound is coming from somewhere else. If the puppet’s mouth moves perfectly in sync with the ventriloquist’s words, while the ventriloquist keeps their lips still, the brain has no reason to question the illusion. When a ventriloquist tilts the puppet’s head or gestures as it “talks,” it mimics natural body language. Ventriloquists use synchronized movements, eye contact, and distinct voices to create a believable and engaging character.
The quirks of the ventriloquism illusion don’t end there. Our brains find ventriloquism so convincing that the sensation of misdirected sound can persist up to half an hour after we stop seeing the trick. Consider the waterfall illusion.
“This is explained by neural fatigue. After a while of the neurons for direction of motion being excited, they get tired,” says Soto-Faraco. “So they stop inhibiting the neurons that compete with them. A similar thing happens with the ventriloquism effect. When someone is exposed to a sound and a flash that have different spatial origins for a while, the brain adapts to this difference. Picture this: First, you see a person at the center of your gaze moving their lips, but the sound is actually coming from your right. Your brain compensates for this, and it seems like the person moving their lips is actually the one speaking. So far, so simple. But then the person in front of you actually starts to speak. Your brain will continue to compensate for the spatial discrepancy. This illusory aftereffect happens only very briefly, so you have to move fast to measure it in the lab, Soto-Faraco says. “It’s also not clear whether this happens because of neural fatigue, like in the waterfall illusion.
Stranger still, scientists have now found that ventriloquism works even when there’s no puppet (or other similar object) involved, a recent study in the journal Psychological Science has found. In the lab, researchers tend to use a slightly more banal set-up than a ventriloquist with a puppet, instead opting for the simpler cues of a tone and a flashing circle. In this latest study, participants were trained to associate a circle on a screen with the tone. Then researchers recreated the ventriloquism effect by shifting the sound to come from a source away from the circle.
“We were surprised to find that the effects on participants’ perception of acoustic space were almost as strong for imagined stimuli as they were for real visual stimuli,” says study author Christopher Berger of the Karolinska Institute in Sweden in a 2018 press release.
As well as shedding light on how our brains process sensory information, researchers believe that studying ventriloquism could help in a number of practical fields. Understanding this quirk of perception could help to train brain-computer interfaces, and to help stroke patients regain their neural function, the Swedish researchers hope. Similar illusions in different senses are of particular interest in developing virtual reality technology. More closely linking the sensory experiences that define a virtual world is key to advancing the sense of how real the virtual world feels.
“It’s beautiful but very complex,” says Soto-Faraco.
Far from a seemingly trivial party trick, the ventriloquism illusion actually says a lot about the fundamental ways our brains make sense of the sensory information we receive from the world around us. The psychology of ventriloquism is a fascinating mix of science and art.
Fig. a Illustration of the nine different voice locations simulated in the current experiment. b A reproduction of the face stimuli that were used in the current experiment. The top row shows the male and female face with their eyes closed; the bottom row shows the two faces looking directly at the camera (DeBruine & Jones, 2017).
Fig. a Plot showing the functions fitted for average “left,” “right,” and “middle” responses in the “Eyes Closed” (red) and “Direct Gaze” (blue) conditions. The dashed lines mark the intersections between the functions fitted to the “left” and “right” responses respectively and the functions fitted to the “middle” responses per gaze condition. The arrows show the cone width per gaze condition. b Violin plots showing the data by eye gaze condition and participant ethnicity. *p < .05.
Fig. a Plot showing the functions fitted for average “left,” “right,” and “middle” responses in the “Eyes Closed” (red), “Direct Gaze” (blue) and “Baseline” (i.e., audio-only; black) conditions. The dashed lines mark the intersections between the functions fitted to the “left” and “right” responses, respectively, and the functions fitted to the “middle” responses per gaze condition. The arrows show the cone width per gaze condition. b Violin plots showing the data for the three visual conditions *p < .05.