Timbre Perception: Definition, Measurement, and Representation
Timbre, the sound’s unique "color," is fundamental to how we perceive and appreciate music. This review explores the multifaceted world of timbre perception and representation. It begins by tracing the word’s origin, offering an intuitive grasp of the concept. Building upon this foundation, the article delves into the complexities of defining and measuring timbre. It then explores the concept and techniques of timbre space, a powerful tool for visualizing how we perceive different timbres. The review further examines recent advancements in timbre manipulation and representation, including the increasingly utilized machine learning techniques. While the underlying neural mechanisms remain partially understood, the article discusses current neuroimaging techniques used to investigate this aspect of perception.
For a long time, the conceptual question of what timbre is has been answered with what timbre is not (Bregman, 1994). Timbre, distinct from pitch and loudness, allows us to distinguish between sound sources (ANSI, 1960). The word itself, originally French, entered English with three meanings reflecting its evolution in French (Merriam-Webster Dictionary, n.d.). The first two, referring to a specific drum type and a heraldic crest, are now obsolete. Interestingly, the French origin hints at its perceptual nature. Cadoz (1987) notes that "timbre" initially referred to a drum with a characteristic "color" sound. When discussing musical timbre, we’re essentially talking about perception. Cadoz (1987) reinforces this by stating that timbre can only be understood through human perception. Our auditory system and cognitive abilities play a crucial role in shaping musical expression and conveying emotions through timbre. This article delves into the multifaceted world of timbre perception and representation.
Section 2 explores how the concept of timbre manifests across various fields, highlighting its significance beyond the realm of music. Section 3 examines the journey from our initial perception of timbre to its representation in different forms. Section 4 ventures beyond acoustics to explore the emerging role of neuroscience in understanding the neural mechanisms underlying our perception of timbre.
The question "Whose timbre?" (Smalley, 1994) highlights the inherent subjectivity of timbre, making a universally accepted definition challenging (Grey, 1975). This complexity leads to diverse interpretations across various fields. Musicians, for instance, focus on the expressive potential of timbre, employing terms like "bright" or "warm" to describe a violin’s sound or the "breathy" quality of a flute (Kakegawa & Asakura, 2023). Psychoacoustics, on the other hand, takes a more scientific approach, analyzing the acoustic properties of sound waves to understand how different frequencies and temporal features contribute to our perception of timbre. Cultural backgrounds also influence how we perceive timbre (Kim et al., 2016).
Timbre: The Elusive Sound Quality
In music, timbre, also known as tone color or tone quality (from psychoacoustics), is the perceived sound of a musical note, sound, or tone. Timbre distinguishes sounds according to their source, such as choir voices and musical instruments. In simple terms, timbre is what makes a particular musical instrument or human voice have a different sound from another, even when they play or sing the same note.
For instance, it is the difference in sound between a guitar and a piano playing the same note at the same volume. Both instruments can sound equally tuned in relation to each other as they play the same note, and while playing at the same amplitude level each instrument will still sound distinctive with its own unique tone color. The physical characteristics that govern timbre include frequency spectrum and envelope. Musicians can change timbre by modifying their singing/playing techniques. For example, a violinist can use different bowing styles or bow on different parts of the string. E.g., playing sul tasto produces a light, airy timbre, whereas sul ponticello produces a harsh, even, and aggressive timbre.
Tone quality and tone color are synonyms for timbre, as well as the "texture attributed to a single instrument". However, the word texture can also refer to the arrangement/composition, such as multiple, interweaving melody lines versus a singable melody accompanied by subordinate chords. Hermann von Helmholtz used the German Klangfarbe (tone color), and John Tyndall proposed an English translation, clangtint, but both terms were disapproved of by Alexander Ellis, who also discredits register and color for their pre-existing English meanings.[1] Determined by its frequency composition, the sound of a musical instrument may be described with words such as bright, dark, warm, harsh, and other terms. There are also colors of noise, such as pink and white.
Many commentators have decomposed timbre into component attributes. The richness of a sound or note a musical instrument produces is sometimes described in terms of a sum of a number of distinct frequencies. The lowest frequency is called the fundamental frequency, and the pitch it produces is used to name the note, but the fundamental frequency is not always the dominant frequency. The dominant frequency is the frequency that is most heard, and it is always a multiple of the fundamental frequency. For example, the dominant frequency for the transverse flute is double the fundamental frequency. Other significant frequencies are called overtones of the fundamental frequency, which may include harmonics and partials. Harmonics are whole number multiples of the fundamental frequency, such as ×2, ×3, ×4, etc. Partials are other overtones. There are also sometimes subharmonics at whole number divisions of the fundamental frequency. When the tuning note in an orchestra or concert band is played, the sound is a combination of 440 Hz, 880 Hz, 1320 Hz, 1760 Hz and so on. Each instrument in the orchestra or concert band produces a different combination of these frequencies, as well as harmonics and overtones.
William Sethares wrote that just intonation and the western equal tempered scale are related to the harmonic spectra/timbre of many western instruments in an analogous way that the inharmonic timbre of the Thai renat (a xylophone-like instrument) is related to the seven-tone near-equal tempered pelog scale in which they are tuned.
The timbre of a sound is also greatly affected by the following aspects of its envelope: attack time and characteristics, decay, sustain, release (ADSR envelope) and transients. Thus these are all common controls on professional synthesizers. For instance, if one takes away the attack from the sound of a piano or trumpet, it becomes more difficult to identify the sound correctly, since the sound of the hammer hitting the strings or the first blast of the player's lips on the trumpet mouthpiece are highly characteristic of those instruments.
Instrumental timbre played an increasing role in the practice of orchestration during the eighteenth and nineteenth centuries. Berlioz[7] and Wagner[8] made significant contributions to its development during the nineteenth century. For example, Wagner's "Sleep motif" from Act 3 of his opera Die Walküre, features a descending chromatic scale that passes through a gamut of orchestral timbres. First the woodwind (flute, followed by oboe), then the massed sound of strings with the violins carrying the melody, and finally the brass (French horns).
Debussy, who composed during the last decades of the nineteenth and the first decades of the twentieth centuries, has been credited with elevating further the role of timbre: "To a marked degree the music of Debussy elevates timbre to an unprecedented structural status; already in Prélude à l'après-midi d'un faune the color of flute and harp functions referentially".[9] Mahler's approach to orchestration illustrates the increasing role of differentiated timbres in music of the early twentieth century. "a seven-bar link to the trio consisting of an extension in diminuendo of the repeated As ... During these bars, Mahler passes the repeated notes through a gamut of instrumental colors, mixed and single: starting with horns and pizzicato strings, progressing through trumpet, clarinet, flute, piccolo and finally, oboe:
In rock music from the late 1960s to the 2000s, the timbre of specific sounds is important to a song. Often, listeners can identify an instrument, even at different pitches and loudness, in different environments, and with different players. In the case of the clarinet, acoustic analysis shows waveforms irregular enough to suggest three instruments rather than one.
Psychoacoustic experiments from the 1960s onwards tried to elucidate the nature of timbre. One method involves playing pairs of sounds to listeners, then using a multidimensional scaling algorithm to aggregate their dissimilarity judgments into a timbre space.
The concept of tristimulus originates in the world of color, describing the way three primary colors can be mixed together to create a given color. By analogy, the musical tristimulus measures the mixture of harmonics in a given sound, grouped into three sections. It is basically a proposal of reducing a huge number of sound partials, which can amount to dozens or hundreds in some cases, down to only three values.
The term "brightness" is also used in discussions of sound timbres, in a rough analogy with visual brightness.
Timbre Perception in Different Fields
As mentioned, for musicians, pitch, loudness, timbre, and duration are the fundamental pillars of sound. However, timbre is often the most challenging element to describe explicitly. It frequently overlaps with concepts like sound quality and tone color (Slawson, 1981), and musicians often employ emotional descriptors. Musicians use timbre as a powerful tool for expressive nuance in musical composition and performance (Bernays & Traube, 2013; De Paula et al., 2000; McAdams, 2019).
The conventional definition struggles when applied beyond traditional instruments, where the focus is often on replicating the sounds of the real world (Risset & Wessel 1999; Smalley 1994; Krumhansl 1989). In computer music, this limitation is addressed through computer modeling and signal processing techniques. These techniques allow for the creation of entirely new sounds by manipulating and combining acoustic properties in novel ways. This opens up a vast sonic palette for composers and sound designers, pushing the boundaries of musical expression. For instance, a sound designer might create a unique, otherworldly texture by combining the attack envelope of a plucked string with the high-frequency components of a cymbal.
Evolution of Timbre Research
The 19th century saw the foundation laid for understanding timbre perception. Pioneering researchers like Hermann von Helmholtz (Helmholtz & Ellis, 1875) proposed that our perception hinges on the prominence of different frequencies within a sound, while Carl Stumpf (1883) emphasized its subjective nature. The early 20th century brought further advancements.
The latter half of the 20th century witnessed a surge in research on timbre recognition and discrimination. Studies by Preis (1984), Wedin & Goude (1972), Miller & Carterette (1975), and Samson et al. (1997) established a strong link between spectral features (the distribution of frequencies in a sound) and perceived musical timbre similarity. The concept of categorical perception, where listeners perceive sounds as belonging to distinct categories rather than a continuum, became a focus. Studies by Cutting and Rosner (1974) and Grey (1975) explored this in musical sounds.
A significant limitation of these studies was their reliance on isolated tones, devoid of any musical context. Grey’s (1978) groundbreaking research addressed this by testing three instrumental timbres (clarinet, trumpet, and bassoon) in various musical contexts. His findings revealed that the context significantly impacted the types of timbre differences listeners perceived. Musical contexts amplified spectral differences, while isolated contexts allowed for clearer comparison of temporal details (how sound unfolds over time).
Grey’s (1975) work employed computer analysis and synthesis to delve deeper into the complexity of musical timbre. He manipulated sound properties to synthesize natural instrument tones from simplified components. These simplified tones, valuable for further psychoacoustic studies, served to reduce the number of factors influencing perception. Notably, Grey identified the attack segment as a crucial cue for instrument recognition, highlighting the multifaceted nature of timbre perception, which involves both steady-state characteristics and transient features.
The intricate nature of timbre demanded novel approaches to its exploration and analysis. Progress relied heavily on advancements in digital technology, data analysis techniques, and computer algorithms. Pioneering this application in 1973, Wessel revealed the crucial role of both spectral and temporal features in timbre perception. He examined the concept of timbre space which initially introduced by Plomp (1970), and explored its use with Multidimensional Scaling (MDS) (Shepard, 1962) techniques.
Timbre Space and Multidimensional Scaling (MDS)
Timbre space serves as a geometric model, with similar tones positioned closer together, representing their perceptual dissimilarities. Wessel explored its potential as a navigational tool for composers and conducted an MDS experiment on perceived dissimilarities of orchestral instruments.
Multidimensional Scaling is a powerful statistical technique that has been instrumental in mapping out the hidden dimensions of timbre space. Prior to MDS, timbre research was limited by its reliance on predefined acoustic features, potentially overlooking aspects of human perception. MDS offered a paradigm shift. Developed by Shepard in 1962, it leverages direct dissimilarity ratings from listeners. Instead of imposing pre-conceived notions about relevant features, listeners judge how different pairs of sounds seem to them.
The core of MDS lies in its ability to translate these dissimilarity judgments into a spatial representation. By analyzing the ratings, MDS creates a timbre space, a multidimensional map where each point represents a specific sound. Sounds perceived as similar by listeners are positioned closer together in this space, while highly dissimilar sounds occupy more distant locations.
In essence, MDS offers significant advantages for timbre research. First, it’s data-driven, relying on listener judgments rather than pre-determined acoustic features. This captures the full spectrum of perceived timbral differences. Second, MDS provides a visual representation (timbre space) of sound relationships.
Early explorations of timbre space, as seen in Wessel’s (1973) work employing MDS identified two key perceptual dimensions. Listeners’ judgments are analyzed by a program called INDSCAL that creates a two-dimensional space where the location of a sound reflects its perceived similarity to others (see figure 1 in Wessel, 1973). The vertical dimension likely relates to the steady-state portion of a sound’s spectrum, while the horizontal dimension might capture the influence of the initial attack.
Building on these findings, Grey (1975) employed synthetic orchestral tones and listener judgments analyzed via MDS. This approach yielded a three-dimensional timbre space (see figure 11 in Grey, 1975). Similar to Wessel’s results, the first dimension again reflected spectral energy distribution. The second dimension captured attack rapidity, but also interestingly correlated with spectral fluctuation (changes in spectral content over time). The third dimension represented the spectral balance during the attack transient, highlighting how the initial energy distribution varied across frequencies.
Traditional MDS techniques aim for lower-dimensional representations while preserving distances between data points. However, given the complexities of timbre, the adequacy of a two- or three-dimensional space as a comprehensive representation model is brought into question. Researchers like Krumhansl (1989) and McAdams et al. (1995) investigated the concept of "timbre specificity," unique perceptual characteristics that contribute to identifying individual timbres.
McAdams et al. (1995) employed an extended version of MDS in a study using synthesized musical tones. Their analysis aimed to estimate several factors, including the number of listener groups, the location of each timbre on common dimensions, and the specificities of each timbre. They also attempted to qualitatively interpret the specificities, identifying both continuous features (e.g. raspiness of attack) and discrete features (e.g. high-frequency transients). These specificities could account for additional continuous dimensions and discrete features of varying perceptual salience. The study’s psychophysical interpretation of the common dimensions, informed by the acoustic parameters outlined by Krimphoff et al. (1994), identified them as quantifiable through log-rise time, spectral centroid, and spectral flux. The results suggested that musical timbres possess specific attributes not captured by these shared perceptual dimensions.
While adding more dimensions might seem intuitive for representing the intricacies of musical sound, Pollard and Jansson’s (1982) study presented a distinct approach. Recognizing the limitations of one-dimensional scales for multidimensional timbre, they proposed a "tristimulus" method focusing on the graphical representation of a sound’s time-dependent timbre behavior. This method analyzes the sound through filters, capturing the loudness of specific frequency bands and their evolution over time.
Pollard and Jansson’s tristimulus method, inspired by color vision, offers a novel way to visualize dynamic timbre changes. While intuitive for understanding attack-to-steady-state transitions, it focuses on limited spectral features, potentially neglecting other key aspects of timbre like inharmonicity and temporal envelopes. This selectivity may limit its effectiveness, particularly for complex sounds or subtle variations. Additionally, interpreting the tristimulus diagram requires expertise, introducing potential subjectivity and variability in analysis. Despite these limitations, tristimulus method offers a valuable contribution to the field by highlighting the importance of considering dynamic changes in timbre.
High-dimensional audio representations have become a staple in the field of speech processing, with ongoing discussions centered around the most effective degree of detail required for accurate modeling. In Dau et al’s (1997) study, they employed a temporal modulation filter bank model, which, while effective, did not explicitly incorporate spectrotemporal modulations. Elhilali et al.
Within the realm of music information retrieval, the trend observed is the application of a vast array of meticulously crafted audio descriptors, as exemplified by the work of Siedenburg et al. (2016). The abundance of descriptors in music information retrieval led Peeters et al. (2011) to identify redundancies, prompting a reevaluation of their necessity. In alignment with this, Patil et al. (2012) and Thoret et al. (2017) showcased the efficacy of spectrotemporal modulation features in achieving robust automatic classification of musical instruments, underscoring the viability of these features in timbre analysis.
McDermott and Simoncelli (2011) delved into the nuances of sound texture perception, employing an analysis-resynthesis methodology to assess the perceptual fidelity of re-created sounds.
Advancements in Timbre Manipulation and Representation
The 21st century has witnessed significant strides in the understanding and manipulation of musical timbre, leading to transformative approaches in sound synthesis and musical expression. The work of Bitton et al. (2020) stands out in this context, presenting an auto-encoder architecture that disentangles loudness from other sound features, thereby enabling more precise control over timbre during sound synthesis. This novel approach has opened new avenues for transforming sounds between instruments and vocals, offering musicians a greater palette of sonic possibilities.
Complementing this practical development, Vahidi et al. (2020) investigated how specific acoustic properties of synthesized sounds from subtractive synthesizers influence their perceived timbre, guiding the design of more intuitive synthesizers. In a similar vein, Sköld (2022) has tackled the challenge of visually representing timbre in musical notation. He innovates by integrating perception-based symbols into a staff notation system to depict spectral qualities visually (width, centroid, density). This system, tested on electroacoustic works (see figure 7-9 in Sköld, 2022), is particularly effective for pieces with multiple sounds.
Building on the potential for practical application, Sköld acknowledges the inherent complexity of timbre. However, Sköld argues that his system goes beyond a simple notation tool. The creative potential of timbre is further explored by Caillon et al. (2020), who have utilized machine learning models to learn and control loudness-independent sound features. Their work aligns with the concept of multidimensional timbre spaces, underscoring the importance of training models for specific functionalities.