Smoking or Barking?

Section VI of Hume’s Enquiries Concerning Human Understanding, entitled Of Probability, opens with a note on Locke:

Mr. Locke divides all arguments into demonstrative and probable. In this view, we must say, that it is only probable that all men must die, or that the sun will rise to-morrow. But to conform our language more to common use, we ought to divide arguments into demonstrations, proofs, and probabilities. By proofs meaning such arguments from experience as leave no room for doubt or opposition (p. 56).

Demonstrations are based on what I call Faith, a prior belief in the truth (BR=1) or falsity (BR=0) of a hypothesis on the grounds of pure reason. Faith requires no evidence, and no evidence can change it.

Probabilities are the result of a tug of war between confirmative and disconfirmative evidence, when none of the two sides manages to prevail on the other.

Proofs occur when the tug of war has a winner. This can happen in two ways:

1. Accumulation of overwhelming confirmative or disconfirmative evidence.

   Evidence accumulates multiplicatively:

   PO = LR₁ ∙ LR₂ ∙ … ∙ LR_N ∙ BO

   If Likelihood Ratios are consistently confirmative (LR>1) or consistently disconfirmative (LR<1), Posterior Odds tend to infinity or to zero. Hence, posterior probabilities converge towards certainty, although they never reach it. We cannot demonstrate that all men must die, or that the sun will rise tomorrow, or – to use another famous analogy – that all swans are white. We can only expect it, based on an overwhelming accumulation of confirmative evidence. As they converge to one of the two boundaries of the probability spectrum, posterior probabilities no longer depend on Base Rates. Whatever the initial priors (except Faith), convergence proves that the hypothesis is true or false. This happens to everyone’s satisfaction, leaving no room for doubt or opposition. The evidence is de facto conclusive.

2. Discovery of a piece of conclusive evidence.

   Multiplicative accumulation implies that even a single piece of conclusive evidence can immediately drive Posterior Odds all the way to infinity or to zero. One black swan is sufficient to prove that the hypothesis “All swans are white” must be false. A Smoking Gun is sufficient to prove that the hypothesis “The suspect is guilty” must be true.

The allure of conclusive evidence is this: it provides Certainty. Not just as the limit of a convergent accumulation, which remains open to refutation, but as an inescapable logical consequence. Not just de facto, but de jure.

Certainty is a desirable and legitimate goal, but it leaves us exposed to the danger of confusing conclusive evidence with perfect evidence. A smoking gun proves that the suspect must be guilty, irrespective of our priors. But it would be wrong to conclude that, if no smoking gun is found, the suspect must be innocent. Whether we believe he is innocent or not continues to depend on our priors.

Chief Inspector Hubbard runs the door test on Margot and Tony. The test is a Smoking Gun: positive evidence proves them guilty; negative evidence does not prove them innocent. However, as he runs the test on Margot, the Inspector believes that, if she opens the door, she conclusively proves herself guilty; if she doesn’t, she is almost certainly innocent. Why? Because, before the test, he believes she is most likely innocent: the Base Rate of her guilt is low. On the other hand, as the Inspector runs the same test on Tony, if Tony fails to open the door (as he was about to), the Inspector would still suspect him. Why? Because, before the test, he believes Tony is most likely guilty: the Base Rate of his guilt is high.

So, while Margot’s test is near perfect – if she opens the door she is certainly guilty; if she doesn’t she is almost certainly innocent – Tony’s test is distinctly imperfect: if he opens the door he is certainly guilty; if he doesn’t… he is less probably, but still quite likely, guilty. How likely depends on BR and TPR. Let’s assume BR=90%: before the test, the Inspector believes Tony is 90% guilty. As the following figure shows, if TPR – the probability that Tony opens the door if he is guilty – is high, say 95%, then Tony’s probability of guilt, if he does not open the door, drops to 31%. If TPR is lower, say 80%, the probability only drops to 64%. Therefore, while he is not as convinced as he was before the test, the Inspector still believes there is quite a high chance that Tony is guilty.

In comparison, assuming Margot’s Base Rate of guilt is 10%, her probability of guilt after failing to open the door is only 1% with TPR=95% and 2% with TPR=80%.

Given the Inspector’s priors, a Smoking Gun is near perfect evidence about Margot, but quite imperfect evidence about Tony. If the Inspector wants to be as sure as possible about Margot, a Smoking Gun is the right test: if it is positive, she is certainly guilty; if it is negative, she is almost certainly innocent.

Of course, the Inspector knows that what ultimately matters is not what he thinks, but what the Court decides. In the eyes of the Court, who sentenced her to death, Margot is almost certainly guilty. She is to the Court what Tony is to the Inspector. Hence, the most appropriate test for the Court would be a Barking Dog. As shown in the following figure, a Barking Dog is the mirror image of a Smoking Gun: it is near perfect evidence if the Base Rate is high, but quite imperfect if it is low.

For example, a meeting with Swann would be a Barking Dog: if no meeting took place, the suspect is certainly innocent; if there was a meeting, the suspect is not necessarily guilty. However, with a high Base Rate, guilt is almost certain; with a low Base Rate, the suspect is less probably, but still quite likely, innocent. For the Court, a Barking Dog, rather than a Smoking Gun, would be near perfect evidence about Margot. If she didn’t meet Swann, she is certainly innocent; if she did, she is almost certainly guilty – in fact, the main reason why Margot was convicted was precisely that the Court was convinced she had met Swann. But a Barking Dog would be quite imperfect for the Inspector who, even if Margot had met Swann, would still believe she is 31% innocent if TNR – the probability that Margot did not meet Swann if she is innocent – is 95%, and 64% innocent if TNR is 80%.

From the Inspector’s point of view, a Barking Dog runs the risk of getting Margot convicted even if he thinks there is a sizeable probability that she is innocent. So he rightly prefers a Smoking Gun: if Margot opens the door, everybody agrees she is guilty; if she doesn’t, the Inspector is almost certain of her innocence, while the Court is still unconvinced. However, even according to the Court, the probability of her guilt will have fallen sufficiently as to grant Margot the benefit of the doubt. She is safe.

If he believes that Margot is probably innocent, the Inspector should resist the natural temptation to prove her innocence. He should try instead to prove her guilt.