In the last column, the basic inner workings of Bayes theorem were demonstrated in the case where two different random variable realizations (the attributes of the Christmas tree bulbs) occurred together in a joint probability function. The theorem holds whether the probability functions for the two events are independent or are correlated. In addition, it can be generalized in an obvious way to cases where there are more than two variables and where one some or all of them are continuous rather than discrete random variables.
If that were all there was to it – a mechanical demonstration between conditional and joint probabilities – Bayes theorem would make a curious footnote in probability and statistics textbooks and would hold little practical interest and no controversy. However, the real power of Bayes theorem comes in ability to link one statistical event with another and to allow inferences to be made about cause and effect.
Before looking at how inferences (sometimes very subtle and non-intuitive) can be drawn, let’s take a moment to step back and consider why Bayes theorem works.
The key insight come from examining the meaning contained in the joint probability that two events, $A$ and $B$, will both occur. This probability is written as
\[ P( A \cap B ) \; , \]
where the operator $\cap$ is the logical ‘and’ requiring both $A$ and $B$ to be true. It is at this point that the philosophically interesting implications can be made.
Suppose that we believe that $A$ is a cause of $B$. This causal link could take the form of something like: $A$ = ‘it was raining’ and $B$ = ‘the ground is wet’. Then it is obvious that the joint probability takes the form
\[ P( A \cap B ) = P(B|A) P(A) \; , \]
which in words says that the probability that ‘it was raining and the ground is wet’ = the probability that ‘the ground is wet given that it was raining’ times the probability that ‘it was raining’.
Sometimes, the link between cause and effect is obvious and no probabilistic reasoning is required. For example, if the event is changed from ‘it was raining’ to ‘it is raining’, it becomes clear that ‘the ground is wet’ due to the rain. (Of course even in this case, another factor may also be contributing to how wet the ground is but that complication is naturally handled with the conditional probability).
Often, however, we don’t observe the direct connection between the cause and the effect. Maybe we woke up after the rain had stopped and the clouds had moved on and all we observe is that the ground is wet. What can we then infer? If we lived somewhere without running water (natural or man-made), then the conditional probability ‘that the ground is wet given that is was raining’ would be 1 and we would infer that ‘it was raining’. There would be no way for the ground to be wet other than to have had rain fall from the sky. In general, such a clear indication between cause and effect doesn’t happen and the conditional probability describes the likelihood that some other cause has led to the same event. In the case of the ‘ground is wet’ event perhaps a water main had burst or a neighbor had watered their lawn.
In order to infer anything about the cause from the observed effect, we want to reverse the roles of $A$ and $B$ and argue backwards, as it were. The joint probability can be written with the mathematical roles of $A$ and $B$ reversed to yield
\[ P( A \cap B ) = P(A|B) P(B) \; , \]
Equating the two expressions for the joint probability gives Bayes theorem and also a way of statistically inferring the likelihood that a particular cause $A$ gave the observed effect $B$.
Of course any inference obtained in this fashion is open to a great deal of doubt and scrutiny due to the fact that the link backwards from observation to proposed or inferred origin is one built on probabilities. Without some overriding philosophical principle (e.g. a conservation law) it is easy to confuse coincidence or correlation with causation. Inductive reasoning can then lead to probabilistically support but untrue conclusions like all swans are white – so we have to be on our guard.
Next week’s column will showcase one such trap within the context of mandatory drug testing.