Signal Detection Analysis Explained
Have you ever taken a hearing test? If you haven’t - or if it’s been a while since you have - here’s basically how it goes: you wear big over-the-ear headphones and you hear some fuzzy noise. Every once in a while, a tone is played amidst the noise. The tones in a hearing test and the lights that indicate aircraft on a radar screen are known as signals.
It would be perfectly reasonable to think that understanding when people would be able to hear a tone or see a light in noisy situations is pretty easy: if the tone is louder than the noise, people should be able to hear and identify it; if the light is brighter than the noise, people should be able to see and identify it. But, that would be wrong.
Here’s an example of how factors other than relative strengths of signal and noise can influence whether or not we see - or at least report seeing - signals in noise. Let’s say I am an experimenter and you are a signal-detecting participant and we run two experiments with monetary awards associated with it. In each experiment, I give you a signal detection task: could be hearing tones, seeing lights, feeling temperature changes, or any other kind of perceptual thing of your choosing (it doesn’t matter to me - it can be whatever helps you understand the example).
In Experiment 1, I will give you 1 US Dollar for every time you correctly identify a signal with no penalty for being wrong, and you could earn up to $20 if you get them all. In Experiment 2, I will give you $20 at the start and take away $1 every time you incorrectly identify a signal with no bonus for being right.
Far be it from me to assume how you would behave in each of those experiments, but if I had to guess, I would imagine that you might be more likely to identify signals in Experiment 1 than you would be in Experiment 2. It would only be natural to risk being wrong more easily in Experiment 1 and to be more cautious about being wrong in Experiment 2. From a strictly monetary-reward-maximizing perspective, the best strategy would be to say that you are seeing signals all the time in Experiment 1 and to say that you are never seeing signals in Experiment 2.
Signal Detection Theory: Core Concepts
The central tenet of signal detection theory is that the decisions that are made by operators under different conditions are all products of underlying strength distributions of the signal and of the noise. The strength of a signal and the strength of noise occupy more than single points in an operator’s mind: if they did, detecting signals would be deterministic, and operators would always choose based on the stronger of the signal and the noise.
We know that operators don’t behave like that, and so the basic theory is that the strength of the signal and the strength of the noise are represented by distributions. We assume that both distributions are normal distributions, as depicted in Figure 8.3. We don’t really know what the \(x\) and \(y\) values are for the signal distribution and the noise distribution and it honestly doesn’t matter. All we are interested in is the shapes of the curves relative to each other so that we can learn about how people make decisions based on their relative perceived strength.
Since the placement doesn’t matter, we can set one of those curves wherever we want and then measure the other curve relative to the set one. And since we can set either one of the curves wherever we want, to make our mathematical lives easier, it is a very good idea to set the noise distribution to have a mean of 0 and a standard deviation of 1. The point of signal detection theory is to understand the underlying perception of signal strength with respect to noise and how operators make decisions given that perception.
The contingency table below describes the possible outcomes of a signal-detection test.
If the signal truly is present, an operator can either correctly identify it - a hit - or not identify it - a miss.
| Signal Present | Signal Absent | |
|---|---|---|
| Response: Signal Present | Hit | False Alarm |
| Response: Signal Absent | Miss | Correct Rejection |
Applications of Signal Detection Theory
Since signal detection theory emerged in the psychophysics literature in the years following World War II, the framework has been used metaphorically to model choices under different conditions. Medical diagnosis a natural fit for the framework: a medical condition can be either present or absent and a diagnostician can either make a diagnosis or not.
Another application of the SDT framework to a decision process is one we have already encountered: classical null hypothesis testing. In null hypothesis testing, an effect (either a relationship as in correlation or a difference as in a difference between condition means) can be either present or absent at the population level.
The \((x, y)\) pairs with population-level correlations of 0.95 and 0.15 led to samples that have correlations of 0.94 and 0.06, respectively. The former correlation is statistically significant at the \(\alpha=0.05\) level (\(p<0.001\)); the latter is not (\(p=0.5423\)).
Please recall from correlation and regression that \(R^2\) is the proportion of variance explained by the model. The proportion of the variance that is not explained by the model - that is, \(1-R^2\), is the error. Since \(r^2=0.944^2=0.891\), the model in part A of Figure 8.2 explains 89.1% of the variance in the \(y\) data. It’s an enormous sample size.
In SDT terms, it is a signal so strong that it would be visible despite the strength of any noise (or: when the correlation is that strong on the population level, it is nearly impossible to sample 100 observations at random that wouldn’t lead to rejecting the null hypothesis). The error associated with the model is \(1-r^2=0.109\). If we take the error to be the noise, then the signal-to-noise ratio is nearly 9:1.
The model in part B of Figure 8.2, by contrast, represents an \(r^2\) of \(0.062^2=0.0038\), therefore explaining less than four-tenths of one percent of the variance. Thus, 99.6% of the observed data is explained by error - or noise - and the signal-to-noise ratio is about 4:996.
Key Measurements in Signal Detection Theory
Of those three, the \(\beta\) statistic is the least frequently used - it’s more in the bailiwick of hardcore SDT devotees - but we’ll talk about it anyway. As noted above, \(d'\) is a measure of the discriminability between noise and signal, and represents the difference between the peaks of the noise and the signal plus noise distributions.
It’s relatively easy to pick out a signal when it is on average much stronger than the noise. In a hearing test, it’s easy to pick out the tones if those tones are consistently much louder than the background noise; on a radar screen, it’s easier to pick out the planes when those lights are consistently much brighter than the atmospheric noise.
The \(d'\) statistic has no theoretical upper limit and is theoretically always positive. However, it’s pretty much impossible to see a \(d'\) value much greater than 3: since the standard deviation of the noise curve is assumed to be 1, a \(d'\) of about 3 indicates as much of a difference as can be determined (whether the peaks of the distributions are 3 standard deviations apart or 30, no overlap is no overlap). It’s also possible but really unlikely to observe a negative \(d'\).
There is one scenario that would consistently produce a negative value of \(d'\). As Miles’s teacher points out, somebody with no knowledge of the test material would have an expected score of 50%, which she takes as evidence that Miles had to know all of the answers in order to get each one wrong.
The \(\beta\) statistic (not to be confused with the \(\beta\) distribution or any of the other uses of the letter \(\beta\) in this course) is a measure of response bias: whether an operator’s response is more likely to indicate signal or to indicate noise at any point.
The \(\beta\) statistic - of which there can be one or there can be many depending on the experiment - is the ratio of the probability density of the signal plus noise curve to the density of the noise curve at a criterion point. If a criterion is relatively strict, then an operator is likely to declare that they perceive a signal only when they have strong evidence to believe so: there will be few hits at a strict criterion point and few false alarms as well.
If a criterion is relatively lenient, then an operator is generally more likely to declare that they perceive signals: there will be relatively many hits at a lenient criterion point and many false alarms as well. The term response bias may imply that it describes a feature of a given operator, but that is not necessarily the case. The response bias is largely a feature of the criterion that the operator has adopted, which in turn can vary based on circumstances.
For example, in a signal-detection experiment where an operator receives a reward (monetary or otherwise) for each hit that they register with no penalty for false alarms, the operator has motivation to adopt a more lenient criteria - they really should say that the signal is present all the time. Conversely, in a situation where an operator is penalized for false alarms and not rewarded for hits, then the operator may be motivated to adopt a more stringent criterion - they might say that the signal is never present.
The C-statistic is a measure of the predictive power of an operator. It has an hard upper limit of 1 (indicating perfect predictions) and a soft lower limit of 0.5 (indicating indifference between predicting correctly and incorrectly). The C-statistic is equal to the area under the Receiver Operator Characteristic (ROC) Curve. It is, equivalently, for that reason also known as the area under the ROC, the AUC (Area Under Curve), or the AUROC (Area Under ROC Curve).
Receiver Operator Characteristic (ROC) Curve
The Receiver Operator Characteristic (ROC) is a description of the responses made by the operator in a signal detection context. The ROC curve is a plot of the hit rate (on the \(y\)-axis) against the false alarm rate (on the \(x\)-axis). Figure 8.8 is an illustration of what an empirical ROC curve looks like: at different measured points (for example, different decision criteria), the hit rate and the false-alarm rate are plotted.
Figure 8.9 is an illustration of theoretical ROC curves that have been smoothed according to models (sort of like curvy versions of the least-squares regression line, as if this page needed more analogies).
Calculations in Signal Detection Theory
Having covered the conceptual bases of SDT and the measurements it produces, we turn now to the actual calculations. The hit rate at each point is the frequency at which the individual correctly identified a signal. The false alarm rate at each point is the frequency at which the individual misidentifies noise as a signal. Thus, the hit rate and the false alarm rate give us the probabilities that an operator is responding to their perception of signal and noise, respectively, under each condition.
Because both the signal and the noise distributions are normal distributions, based on the area in the upper part of the curve, we can calculate \(z\)-scores that mark the point on those curves that define those areas. Additionally, because the noise distribution is assumed to be a standard normal distribution, the values of \(z_{FA}\) are also \(x\)-values on the strength axis.
At this point, our analyses hit a fork in the road. The observed ROC curve is based on data: it does not change based on how we choose to analyze the data, and the C-statistic does not change either. There is domain-specific debate - as in this recent example paper regarding recognition memory - over which model is more appropriate and why. I have have no strong opinions on the matter with regard to psychological processes.
From an analytic standpoint, it seems to make more sense to start from the unequal variance assumption because it includes the possibility of equal variance (it may be another case of a poorly-named stats thing: the unequal variance assumption might be more aptly called the not assuming the variances are equal but they might be assumption, but that’s really not as catchy). That is, if we start off assuming unequal variance and the variances end up being exactly equal, then that’s ok.
This page will cover both. We know the mean and the standard deviation of the noise distribution because we decided what they would be: 0 and 1, respectively. That leaves the mean and standard deviation of the signal+noise distribution to find out.
Linearized ROC
Our sample data come from an experiment with five different conditions, each one eliciting a different decision criterion (and, in turn, a different response bias). To find the overall \(d'\) (and \(\beta\), which will depend on first finding \(d'\)), we will use a tool known as the linearized ROC: a tranformation of the ROC curve mapped on the Cartesian \((x, y)\) plane. The linearized ROC plots \(z_{hit}\) on the \(x\)-axis and \(z_{FA}\) on the \(y\)-axis (see Figure 8.13 below). That’s potentially confusing since the ROC curve has the hit rate on the \(y\)-axis and the FA rate on the \(x\)-axis.
The intercept of the Linearized ROC is an estimate of \(d'\). We know that \(d'\) is the distance between the mean of the noise distribution and the mean of the signal+noise distribution in terms of the standard deviation of the noise distribution. Because we have assumed that the noise distribution is represented by a standard normal distribution, which by definition has a standard deviation of 1 (again, we could have picked any normal distribution, and now we are glad that we picked the one with a mean of 0 and a standard deviation of 1), our estimate of the standard deviation of the signal+noise distribution is also 1.
We also know that since \(d'\) is a measure of the distance between the mean of the noise distribution and the mean of the signal+noise distribution in terms of the standard deviation of the noise distribution, that the distance between the means is equal to \(d'/1=d'\) and thus that the mean of the signal+noise distribution is also equal to \(d'\). We can use the noise distribution to locate points on the strength axis: we assumed the noise distribution is a standard normal so it is centered at 0 and its standard deviation is 1 so the \(z\)-values for the noise distribution are also \(x\) values. Our next step is to find what those \(x\) values represent in terms of the signal plus noise distribution.
We will calculate \(z\)-scores for the signal distribution that correspond to each of the criterion points based on the estimates of the mean and standard deviation of the signal distribution we got from the linearized ROC, which are separate from the \(z_{Hit}\) values we got from the hit rates in the experimental data. The mean of the signal distribution is equal to \(d'\): \(\mu_{Signal}=0.817\) and the standard deviation of the signal distribution is estimated by the slope of the linearized ROC: \(\sigma_{Signal}=1.17\).
Criterion in Signal Detection Theory
Criterion is a measure of the willingness of a respondent to say ‘Signal Present’ in an ambiguous situation. The choice of a criterion may depend on perceived consequences of outcomes. Researchers may be interested in comparing response bias (i.e., criterion) for groups of individuals who differ in various ways. Setting a criterion is equivalent to setting alpha error in a hypothesis testing situation. Alpha error is the chance a researcher is willing to take that a test will be found ‘statistically significant’ when there is no real effect.
We define the Criterion as the z score on the Signal Absent distribution. In the SDT applet, Move the “d’ =” box in the Normal Distributions panel to obtain a value of d’ near 1.8.
Slowly move the Criterion box in the Normal Distributions panel from the far right to the far left, and back again. Explain what happens to the hit and false alarm rates as you shift criterion. How do the rates of change in hits and false alarms differ? When criterion is at the far right, the respondent will never say ‘Signal Present.’ Consequently, the hit and the false alarm rates are both zero. When the criterion is at the far left, the respondent will always say ‘Signal Present,’ and the hit and false alarm rates are both 1.00. As the criterion moves from the far right to the left, the respondent becomes more willing to say ‘Signal Present.’ When d’ is positive, as it is here, the hit rate increases more rapidly than the false alarm rate until the criterion reaches the point where the two normal curves intersect. From there on, the false alarm rate grows more quickly.
When the signal is absent, the stimulation on any given trial may be more or less like a signal. The criterion represents the minimum level of stimulation needed for the respondent to say ‘Signal Present.’ The computed Criterion is the z-score of this stimulation value on the Signal Absent distribution.
If false alarms are very costly, the higher criterion is better.