The Definition and Perception of Rhythm: An In-depth Exploration
Music is inherently a temporal experience, with its temporal structures playing a crucial role in listeners' aesthetic, emotional, and behavioral responses. Music is perceived across multiple interconnected timescales, from individual notes to measures and phrases.
In this context, understanding the nuances of rhythm perception becomes essential. Rhythm, beat, and meter are fundamental components of music. In our usage, rhythm refers to the absolute timing of individual notes or sounds, beat refers to the perceived regular pulse that listeners tend to feel and synchronize their movements with, and meter is the repeating cycle of beats, often a pattern of variable salience (composed of stronger and weaker beats). The beat tends to be steady or theoretically isochronous (evenly spaced), although human performance of music inevitably adds temporal variability, via both musical intention (e.g., rubato, expressively stretching and compressing the beat rate) and natural performance dynamics (e.g., due to the limits of temporal precision of human movements).
The capacity to synchronize movements to the beat in music is a complex, and apparently uniquely human characteristic.
Synchronizing movements to the beat requires beat perception, which entails prediction of future beats in rhythmic sequences of temporal intervals. Absolute timing mechanisms, where patterns of temporal intervals are encoded as a series of absolute durations, cannot fully explain beat perception.
Beat perception seems better accounted for by relative timing mechanisms, where temporal intervals of a pattern are coded relative to a periodic beat interval. Evidence from behavioral, neuroimaging, brain stimulation and neuronal cell recording studies suggests a functional dissociation between the neural substrates of absolute and relative timing. This chapter reviews current findings on relative timing in the context of rhythm and beat perception.
Audio-Tactile Interactions in Rhythm Perception
When we explore a texture with our fingers, the interaction between the skin and the surface produces vibrations that propagate both through the air, up to our ears, and through our skin, down to our mechanoreceptors. Both sensory channels contribute to the perception of the texture properties.
These audio and tactile vibrations emanating from the same source are perceptually merged into a single amodal percept, creating a mental image of the surface. As both stimuli share the same origin, the two modalities greatly influence each other. Altering the frequency content of the touch-produced sound can bias the perception of tactile roughness. This effect, that can be produced when we rub our hands together, is known as the parchment skin illusion.
While psychophysical experiments demonstrate high-level interactions between audio and tactile sensory systems, neuroimaging studies suggest that these interactions also occur in early sensory areas. These experiments reveal strong interactions and common neural processes for vibrotactile perception and pitch perception, for frequencies above 60 Hz. However, audio-tactile interactions with lower frequency content, associated with rhythm, in particular rhythmic changes, are rarely investigated.
In the present paper, we investigated the perception of audio and haptic stimuli in which the rhythm evolves continuously with time. We decided to use the term rhythm that is here considered as the succession of events forming periodic patterns, which elements are distinguishable from each other, sticking to the definition given by Cooper et al.: “to experience rhythm is to group separated sounds into structured patterns”. We use the term for beat rates up to 60 Hz, frequency range that is more commonly characterized as flutter range in tactile perception.
Whether the sensation of accelerating or decelerating rhythms is shared between audio and haptic perception remains unknown. In audio, these evolving stimuli are better known as accelerando or decelerando, in the case of tempo increase or decrease. In touch, it has been shown that a 10% variation in the ridge density can be detected. Here, we generated haptic stimuli whose spatial frequency gradually evolves during exploration by a finger on a glass plate actuated with ultrasonic friction modulation. This method uses ultrasonic levitation to change the friction between the finger and the glass.
Modulating friction in reaction to users’ exploratory motion produces sensations of texture, shape and relief on a flat surface. In addition, the use of synthesized stimuli makes it possible to freely combine auditory and haptic stimuli. A similar setup has already been used to show audio-haptic perception changes with aging.
In the present study, we modulated the friction with respect to the position of the user’ finger. The modulation is a spatial sinusoidal wave, which spatial frequency gradually increases or decreases, becoming finer or coarser. Touching these haptic stimuli produces the sensation of bumps that becomes closer or more distant from each other, like accelerating or decelerating rhythmic patterns.
The perception of these haptic gradients is here investigated by a psychophysical experiment, whose results are compared with the literature on auditory perception. We further explain these observations with a multimodal model of rhythm perception. This model predicts similar auditory and haptic mechanism in the perception of rhythmic gradients, confirmed by a final multimodal experiment that demonstrates interaction between the two modalities.
Haptic Detection of Periodicity Changes
The longer a participant explores a texture, the better they are at discriminating if the frequency is increasing or decreasing. How fast they can detect the trend is a clear indication of their perceptual threshold. The first experiment investigates how the exploration distance influences the detection of gradient g, by constraining the exploration by a window w. The experimental design draws inspiration from studies on auditory perception, which explore the minimal duration needed to perceive a frequency or tempo change at a given rate of change.
Figure 1: Overview of the experiment on haptic gradient detection thresholds.
The detection thresholds were investigated for 4 gradient value conditions (g= 0.015, 0.025, 0.035, 0.045 mm-1) and 2 directions (increasing or decreasing) with 6 window sizes (w= 10, 20, 30, 40, 50, 60 mm). A stimulus corresponding to the increasing direction is presented in Fig. 1a. In each trial, subjects were asked to synchronize their movement with a cursor to ensure a constant finger velocity. After exploring the stimulus once, they had to report if they felt that the stimulus “became finer” or “became coarser”, which corresponded to increasing or decreasing spatial frequencies, respectively.
Subjects’ responses and the related analyses are presented in Fig. 7a,b in the Materials and methods section. The percentages of correct answers for all subjects and for each condition are fitted with psychometric curves to obtain the window size thresholds wT. The minimal exploration distances to perceive a change in the gradient value g = 0.015, 0.025, 0.035 and 0.045 mm-1 were found to be wt = 46.2, 33.1, 28.7 and 25 mm, respectively. The thresholds wT decrease as the gradient value g increases with a linear dependency on a logarithmic scale, as illustrated Fig. 1c. A logarithmic regression reveals a significant correlation (p=0.004) between the window size threshold and the gradient value such that log(wT) = 1.50 - 0.55 log(g), which can also be written as wT × g0.55 = 4.48.
Comparison of Audio and Haptic Thresholds
The exploration distance wT and the gradient value g are not proportional, but follow a power law with an exponent of 0.55. To compare the results of this experiment with data from the literature on tempo and frequency gradients in auditory stimuli, the exploration distances w (in mm) and gradient value g (in mm-1) were converted into stimulus durations ΔT = w / vfinger (in s) and rate of frequency change r=g × vfinger (in s-1), respectively, using the finger velocity vfinger=59.6± 9.7 mm/s. Participants were asked to explore the stimuli with a constant speed by synchronizing their finger movement with a cursor on a visual display.
Figure 2: Comparison of the experimental results with the literature.
Figure 2 provides a comparison between our results and the literature data. The haptic gradient threshold curves strongly resemble the audio tempo gradient threshold curves, with the only difference being that the haptic thresholds are presented for shorter durations. Textures of a few centimeters explored at a velocity of approximately 50 mm/s typically lasted approximately 1 s, which is indeed below the usual tempo durations for audio stimuli. The graph also shows that the slope distribution of the tempo and haptic gradients is close to that obtained for frequency chirps.
To numerically investigate these similarities, we performed logarithmic regressions, which yielded the equations in Fig. 2. We can compare the values of the exponent e and the constant c of the threshold laws ΔT × re = c. This analysis confirms that the 3 exponent values are in the same range and, most importantly, that the haptic and auditory tempo values differ by only 7.3% and 7.8% (relative error) for e and c, respectively.
In summary, haptic gradient thresholds follow the same law as rhythmic gradients. This suggests that similar mechanisms are activated in the two modalities for low-frequency gradient perception.
Perceptual Model of Audio-Haptic Rhythmic Gradients
To investigate these mechanisms, we adapted a model from the literature on the perception of irregular rhythmic patterns based on the work of Schulze. Three theories compete to explain the encoding of tempo perception.
The successive interval discrimination theory proposes that each interval between two beats is compared with the previous interval. When a difference exceeds a given threshold, an irregularity is perceived. Comparison with the internal rhythm theory states that the first beats are internalized and used as a rhythmic reference. When a beat differs by more than a threshold from the reference, an irregularity is perceived.
Finally, the third theory, the internal interval theory, is similar to the successive interval discrimination theory, but uses the interval rather than the rhythmic difference. It postulates that the first interval is internalized and used as a reference. When a duration difference between one interval and the reference exceeds a certain threshold, an irregularity is perceived.
These theories were tested by Schulze on beat sequences that contained carefully chosen irregularities. The results of his study revealed that the internal rhythm theory was a good predictor, but that the results were also in agreement with the internal interval model predictions. The experiment was reproduced by Keele et al., who concluded that the comparison with the internal interval theory was more likely to predict the perceived rhythm. A generalization of Schulze’s model was later proposed to take into account the influence of the initial pace. Investigating the perception of linear tempo gradients, Madison explained his results using models of previous studies with the principle of accretion, in which the accumulation of small differences reinforces the global difference.
Figure 3: Model of tempo perception applied to haptics.
The haptic frequency gradient perception model, derived from its audio counterpart, is illustrated in Fig. 3. First, the haptic signal encoding the friction is converted into a pulse train, where each pulse corresponds to the local maximum of the virtual shape. The pulse train signal mimics the response of the Pacinian channels to sinusoidal stimulation. The duration between two pulses τ is then computed. The probability of perceiving 2 intervals of duration τ1 and τ2 as identical is described by the probability distribution P(α). We assume that the probability of perceiving a difference in successive intervals depends on the duration ratio α = τ2/τ1 and follows a log-normal function as presented in Fig. 3a. The standard deviation of logarithmic values σ is the only fixed parameter in the model.
We compared three theories of tempo perception (see Fig. 8 in the Materials and methods section). Among the three theories, the internal interval with accretion theory is the best predictor of the observed results. In this model, the first interval is internalized and used as a reference, and then each interval duration is compared to the reference duration to calculate the probability P of perceiving no difference. The small, imperceptible variations are compounded using the accretion principle and their accumulation reinforces the global difference. The overall probability of perceiving no change in the stimulus PN is then the product of all the previous probabilities P(αi). The final probability of perceiving the change in frequency is given by Pg,w=1-PN.
In line with the experimental design, this procedure is applied to all gradient magnitude g and window size w conditions. The theoretical thresholds are calculated by performing the same analysis with psychometric curve fittings (see Fig. 9 in the Materials and methods section). These thresholds are defined as the critical window sizes wT that yield a 50% chance of perceiving the irregularity (Pg,wT=0.5). To minimize the error between the four wT values of the model and of the experiment, we optimize the parameter σ of the log-normal distribution. We find that σ = 1.153 leads to the best predictions of the observed data, with a mean relative error of 1.74%. The proposed model can also extrapolate the experimental thresholds to a broader range of values, as presented in Fig. 1c.
Audio-Haptic Interaction
The previous results hint at a shared process between the haptic and the audio perception of rhythmic gradients. To test whether both modalities do indeed influence each other, we measured their influence on the overall detection threshold when both modalities were present. This methodology was successfully used in previous studies to unravel the interaction between haptics and other...