One of the most curious features of Bayesian inference is the non-intuitive conclusions that can result from innocent looking observations. A case in point is the well-known issue with mandatory drug tests being administered in a population that is mostly clean.
For the sake of this post, let’s assume that there is a drug, called Drugg, that is the new addiction on the block and that we know from arrest records and related materials that about 7 percent of the population uses it. We want to develop a test that will detect the residuals in a person’s bloodstream thus indicating that the subject has used Drugg within some period time (e.g. two weeks) prior to the administration of the test. The test will return a binary result with a ‘+’ indicating that the subject has used Drugg and a ‘-‘ indicating that the subject is clean.
Of course, since no test will be infallible, one of our requirements is that the test will provide an acceptably low percentage of cases that are either miss detections or false alarms. A missed detection occurs when the subject uses Drugg but the test fails to return a ‘+’. Likewise, a false alarm occurs when the test returns a ‘-‘ but the subject is clean. Both situations present substantial risk and potentially high costs, so the lower both percentages can be made the better.
In order to develop the test, we gather 200 subjects for clinical trials; 100 of them are known Drugg users (e.g. they were caught in the act or are seeking help with their addiction) and the remaining 100 of them are known to be clean. After some experimentation, we have reached the stage where the 99 percent of the time, the test correctly returns a ‘+’ when administered to a Drugg user and 95 percent of the time, it correctly returns a ‘-‘ when administered to someone who is clean. What are the false alarm and missed detection rates?
This is where Bayes theorem allows us to make a statistically based inference and one that is usually surprising. To apply the theorem, we need to be a bit careful with notation so let’s first define some additional notation. A person who belongs to the population that uses Drugg will be denoted by ‘D’. A person who belongs to the population that is clean will be denoted by ‘C’. Let’s summarize what we know in the following table.
| Description | Symbol | Value |
|---|---|---|
| Probability of a ‘+’ given that the person is C | P(+|C) | 0.05 |
| Probability of a ‘-’ given that the person is C | P(-|C) | 0.95 |
| Probability of a ‘+’ given that the person is D | P(+|D) | 0.99 |
| Probability of a ‘-’ given that the person is D | P(-|D) | 0.01 |
| Probability that a person is C | P(C) | 0.93 |
| Probability that a person is D | P(D) | 0.07 |
There are two things to note. First the results of our clinical trials are all expressed as conditional probabilities. Second, the conditional probabilities for disjoint events sum to 1 (e.g. P(+|D) + P(-|D) = 1 since a member of D, when tested, must result in either a ‘+’ or a ‘-‘).
In the population as a whole, we won’t know to which group the subject belongs. Instead, we will administer the test and get back either a ‘+’ or a ‘-‘ and from that observation we need to infer to what group the subject is most likely to belong.
For example, let’s use Bayes theorem to infer what the missed detection probability, P(D|-) (note the role-reversal between ‘D’ and ‘-‘). Applying the theorem we get
\[ P(D|-) = \frac{ P(-|D) P(D) }{ P(-) } \; . \]
Values for P(-|D) and P(D) are already listed above, so all we need is to get P(-) and we are in business. This probability comes is obtained from the formula
\[ P(-) = P(-|C) P(C) + P(-|D) P(D) \; . \]
Note that this relationship can be derived from $P(-) = P(- \cap C ) + P(- \cap D)$ and $P(A \cap B) = P(A|B) P(B)$. The first formula says, in words, that the probability of getting a negative from the test is the probability of either getting a negative and the subject is clean or getting a negative and the subject uses Drugg. The second formula is essentially the definition of conditional probability.
Since we’ll be needing them P(+) as well, let’s compute them both now and note their values.
| Description | Formula | Symbol | Value |
|---|---|---|---|
| Probability of a ‘+’ given that the person either is in C or D | \[ P(+) = P(+|C) P(C) + P(+|D) P(D) \] | P(+) | 0.1158 |
| Probability of a ‘-’ given that the person either is in C or D | \[ P(-) = P(-|C) P(C) + P(-|D) P(D) \] | P(-) | 0.8842 |
The missed detection probability is
\[ P(D|-) = \frac{ P(-|D) P(D) }{ P(-) } = \frac{ 0.01 \cdot 0.07 }{ 0.8842 } = 0.0008 \; . \]
So things are looking good and we are happy. But our joy soon turns to perplexity when we compute the false alarm probability
\[ P(C|+) = \frac{ P(+|C) P(C) }{ P(+) } = \frac{ 0.05 \cdot 0.93 }{ 0.1158 } = 0.4016 \; . \]
This result says that around 40 percent of the time, our test is going to incorrectly point a finger at a clean person.
Suppose we went back to our clinical trials and came out with the second version of the test where nothing had changed except P(-|C) had now risen from 0.95 to 0.99. As the figure below shows, the false alarm rate does decrease but still remains very high (surprisingly high) when the percentage of the population using Drugg is low.
The reason for this is that when the percentage of users in the population is small in order to get the missed detection rate low we have to do it at the expense of a greater percentage of false alarms. In other words, our diligence in finding Drugg users has made us overly suspicious.