How to Create Auditory Illusions
Optical illusions have long been the inspiration for and integrated into visual arts. M.C. Esher’s work, for instance, presents the viewer with impossible objects and perceptual confusions. Due to the historical ease of reproducing and distributing visual material, as opposed to auditory material, visual illusions have long been widely encountered, studied, and applied in artistic works.
Auditory illusions, much like visual illusions, reveal the deficiencies and oddities of our perceptual processes, but the auditory and visual systems have their own unique attributes. In psychology and neurology, the study of optical illusions has played a large role in understanding the visual perception apparatus. Auditory illusions provide key information in unpacking our auditory processes for psychologists and neurologists.
The field of psychoacoustics examines how the brain processes sound, music, and speech. Hearing is not strictly mechanical but involves significant neural processing and is influenced by our anatomy, physiology, and cognition. Researches have even found that how we unconsciously interpret sounds is influenced by our individual environments, backgrounds, and dialects.
Dr. Diana Deutsch, a psychologist at the University of California, San Diego is at the forefront of psychoacoustic research and her work has utilized countless auditory illusions and sonic paradoxes. Due largely to her research, there has been increasing understanding of the cognitive factors in the auditory system and how it has evolved over time to help us interpret our sonic environments effectively. Psychoacoustic research has been applied in myriad contexts including modeling compression codecs like mp3s, software development, audio system design, drone flying, car manufacturing, and even, terrifyingly, in acoustic weapon development.
A few years ago, I attended a talk on wave field synthesis, and to say I was captivated feels like a sorry understatement. Wave field synthesis, if you are unfamiliar with it as I was, is a spatial auditory illusion and rendering technique that produces a holophone, or auditory hologram, using many individually driven loudspeakers. The effect is that sounds appear to be coming from a virtual source and a listener’s perception of the source remains the same regardless of their position in the room. This introduction to wave field synthesis, in addition to being quite exciting, pointed me towards a categorical lack of knowledge about auditory illusions. Since then, I’ve been filling in the gaps and adding these illusions to my sonic toolbox.
Now, quite a bit of theatrical sound design could be considered spatial illusions like, for example, when we recreate actual physical phenomena like the doppler effect.
There are a number of types of illusions that can be roughly categorized as spatial illusions, perpetual motion, and non-linear perceptual effects. More auditory illusions continue to be uncovered and understood, so these categories aren’t rigid.
Spatial Illusions
Spatial illusions are already a mainstay of theatrical sound design. We frequently manipulate spatialization to make it seem as though sounds are coming from a particular source or direction other than the loudspeaker producing the sound. Holophones can be created in a number of ways including wave field synthesis as I’ve mentioned. Binaural recording is another example of spatial manipulation, reproducing interaural features and anatomical influences of the head and ear.
Unreal sounds created in the inner ear or brain are a part of our daily lives that we typically don’t notice, and there are several auditory illusions that mirror common visual illusions. A Zwicker Tone, for example, is the sonic equivalent of an after image. Illusions of Auditory Continuity show us that when an acoustic sound signal is momentarily cut off and replaced by another sound, listeners perceive the original signal to continue through the interruption. Through the familiar Precedence or Haas Effect, we perceive a singular sonic event when one sound is followed by another with a short delay time, and that we ascribe directionality based on the first arriving sound.
Here's a basic overview of some spatial auditory illusions:
| Illusion Type | Description |
|---|---|
| Wave Field Synthesis | Creates a holophone effect using multiple loudspeakers to make sounds appear to originate from a virtual source, maintaining the listener's perception regardless of their position. |
| Binaural Recording | Reproduces interaural features and anatomical influences of the head and ear to create a spatial auditory experience. |
| Zwicker Tone | The auditory equivalent of a visual afterimage. |
| Illusions of Auditory Continuity | Listeners perceive an interrupted sound as continuous when it is briefly replaced by another sound. |
| Precedence (Haas) Effect | When one sound is followed by another with a short delay, listeners perceive a single sonic event and ascribe directionality based on the first sound. |
These are just a few examples of how spatial auditory illusions can manipulate our perception of sound and space.
Perpetual Motion Illusions
Less subtle are perpetual motion illusions. Pitch and tempo circularity is roughly analogous to the barber pole illusion in which a sound seems to be endlessly ascending or descending or a rhythm seems to be endlessly increasing or decreasing in tempo. Both pitch and tempo circularity encapsulate a number of techniques and effects. The Risset Rhythm and Shepard Tone are complex versions. The Shepard Tone most notably influenced the film score for Dunkirk and created a palpable sense of anxiety.
Shepard Tone Illusion
The Shepard Tone is a fascinating example of an auditory illusion, creating the perception of a tone that continually ascends or descends in pitch, yet never seems to reach a final destination. This effect is achieved by layering a series of tones that span different octaves, with the highest and lowest frequencies gradually fading in and out to create the illusion of endless motion.
Non-Linear Perceptual Effects
There are a number of speech-related auditory illusions. Most famously, the Laurel/Yanny internet phenomenon of 2018 brought speech interpretation illusions into the spotlight. It also demonstrated the incredible subjectivity of our hearing. Similarly, The McGurk Effect presents a puzzling phenomenon in the interaction of vision and speech.
McGurk Effect
The McGurk Effect is a fascinating phenomenon that highlights the interaction between auditory and visual perception in speech. It occurs when the auditory component of one sound is paired with the visual component of another sound, leading to the perception of a third, different sound. For example, if you hear the sound "ba" while watching a video of someone articulating "ga," you may perceive the sound as "da." This illusion demonstrates how our brains integrate information from different senses to interpret speech, and how visual cues can override what we actually hear.
Dr. Deutsch has amassed an immense number of Stereophonic Illusions including Phantom Words, Binaural Beats, the Glissando Illusion, the Octave Illusion, the Scale Illusion, the Tritone Paradox, and more. Her work shows us how differently people perceive the same sounds. When we listen to speech the words we perceive are influenced by our expectations, knowledge, dialect, and culture, in addition to the physical sounds we hear. Much of her work has also demonstrated how left and right-handedness influences how complex sounds are synthesized and localized in our heads. In the Tritone Paradox, utilizing sequentially played Shepard tones a tritone apart, some listeners hear the tone ascending while others hear it descending.
Phantom Words
Some years ago I discovered a way to produce a large number of ‘phantom words’ and phrases within a short period of time.
To obtain the best effect, find a time when you will not be disturbed, and sit in front of two loudspeakers, with one to your left and the other to your right. (Headphones don’t work so well for this illusion.) Make sure that your sound system is set for stereo, and that the two loudspeakers are balanced for loudness.
In a ‘phantom words’ demonstration, each track contains two words, or a single word composed of two syllables, and these are repeated over and over again. The same sequence is presented through both loudspeakers, but the tracks are offset in time so that when the first sound (word or syllable) is coming from the speaker on the left the second sound is coming from the speaker on the right; and vice versa.
It works well to have a pen and paper in front of you, so that you can write down the words and phrases that you hear. Often people initially hear a jumble of meaningless sounds, but after a while distinct words and phrases suddenly emerge. It often seems that the left and right loudspeakers are producing different words, which sometimes appear to be spoken by different voices. After a while, you will probably find that new words and phrases appear to be coming from one or both of the loudspeakers. When this happens, write down the new ones also.
In addition, it’s not unusual to hear a third stream of words or phrases, apparently coming from some location between the loudspeakers. Nonsense words, and musical sounds such as percussive sounds or tones, sometimes appear to be mixed in with the meaningful words.
If English is your second language, you may find that you hear some words and phrases in your native language.
In courses on illusions that I taught at UCSD, I generally played some ‘phantom words’ to my class. The students at our university are linguistically very diverse, and taken together I’ve received reports of ‘phantom words’ in Spanish, Mandarin, Cantonese, Korean, Japanese, Vietnamese, Tagalog, French, German, Italian, Hebrew, and Russian - to name just a few.
People appear to hear words and phrases that reflect what is on their minds - rather as in a Rorschach test, though it’s my impression that the present effect is stronger. I can guess who is likely to be on a diet, as they report words like ‘I’m hungry’. ‘diet coke’ or ‘feel fat’. And students who are stressed tend to report words that are related to stress - if I play these sounds close to exam time, students may well hear phrases like ‘I’m tired’, ‘no brain’, or ‘no time’.
While this is brief overview is the tip of the ever-expanding metaphorical iceberg of auditory illusions, I have found that looking into psychoacoustics and auditory neurology provides incredible design techniques and ideas that are not always at our disposal. The potential here that I’m so excited about, is to create audience experiences that rouse questions about the subjectivity of their perceptions of the world around them. Audiences can leave the theater not believing their ears.
It also illuminates a greater need for interdisciplinary collaboration and cooperation between fields that often feel disparate: psychology, neurology, audiology, engineering, music, sound design, etc. In my own work, I have yet to utilize almost any of this material (with the exception of spatialization techniques, of course), but it is leading me to think about designing for the whole head, the ear, the brain, and the mind.