Sunday Sherlocks

Nate Silver‘s second example to illustrate Bayes’ Theorem is:

Hypothesis: A terrorist attack is under way. Evidence: A plane has crashed into the Twin Towers.

1. What is the prior probability of a terrorist attack? Virtually zero in most places, but it is greater than zero in big metropolitan areas like Manhattan. Even there, it is small, which is why most people get on with their lives without worrying about it. Silver sets it at 1/20,000: BR=0.005%.
2. What is the probability of a plane crashing into the Twin Towers, in case of a terrorist attack? It is high. Silver calls it a certainty: TPR=100%.
3. What is the probability of a plane crashing into the Twin Towers, in case there is no terrorist attack? Silver cites records showing that in the previous 25,000 days there had been two instances of planes hitting a tall building in New York City: one in 1945 on the Empire State Building and the other in 1946 on 40 Wall Street. So he sets the chances at 1 in 12,500: FPR=0.008%.

Result: The probability of a terrorist attack in Manhattan, after a plane crashes into the Twin Towers, is 38%. Pretty high, but – as most of us were thinking at the time – it was still possible that it might have been some sort of freak accident. But as soon as the second plane hit, a terrorist attack became a virtual certainty: BR=38%, together with TPR=100% and FPR=0.008%, implies PP=99.99%.

The result is quite insensitive to the three assumptions. For example, with BR=1/200,000 the posterior probability becomes 6% after the first crash and 99.87% after the second. If TPR=60%, PP is 27% and then 99.96%. And even if there had been ten times as many plane accidents in the past, the numbers are 6% and 98.74%. A large Likelihood Ratio LR=TPR/FPR leads very quickly to the truth. Nobody in his right mind could doubt that 9/11 was a planned act of violence. In fact, nobody does.

But then the next question is: Who did it? Al-Qaeda, of course. The evidence supporting the hypothesis that al-Qaeda organized and carried out the attack is as flagrant as seeing Desdemona in bed with Cassio. Hence the question becomes: Why do millions of people believe otherwise? Let’s see:

1. What is the prior probability that al-Qaeda (or some other group with similar goals) was behind 9/11? This is the question everybody asked right after the second crash proved it was no accident. It is the probability before seeing any 9/11-related evidence. The sensible answer is: high. Let’s say BR=80%.
2. What is the probability of the accumulated evidence on 9/11 (let’s call it E), in case al-Qaeda conducted the attack? For example, just to pick one item from E: what is the probability that, if al-Qaeda did it, Osama Bin Laden would claim responsibility? The sensible answer is: very high. TPR≈100%.
3. What is the probability of E, in case al-Qaeda did not conduct the attack? How likely is it, for example, that Bin Laden would feign responsibility? The sensible answer is: very low. FPR≈0%.

Result: the probability that al-Qaeda orchestrated the attack, given the accumulated evidence about 9/11, is a virtual certainty.

Not so, however, for 36% of Germans, 43% of Britons and 61% of Turks, according to a survey conducted in 2008. Another survey, completed in 2011, showed that 73% of Turkish Muslims believe that the attackers were not even Arabs. Five years earlier it was 59%. So much for convergence to the truth.

How can this be? Part of the answer is: ignorance. Many people don’t know the full extent of E, and it is reasonable to assume that, if they did, they would change their mind. But the most interesting part of the answer is: distrust.

Let’s start with the prior probability BR. The alternative to the hypothesis that al-Qaeda conducted the attack is that someone else did it. There is no consensus among 9/11 sceptics on who that someone may be: the US government, Israel, some secret service, others unknown. So let’s give it a collective name: Iago. “It must have been Iago!” we can imagine many people exclaiming after the second plane. Nothing is logically wrong with it. If BR=0, ‘It was certainly not al-Qaeda’, then that’s it: Faith is impermeable to any amount of evidence. It is like Eugene Fama believing in Market Efficiency. But if BR is positive, a change of mind is possible. The smaller it is, however, the stronger the evidence it requires. For example, if the prior probability that Arabs were involved is 4%, then TPR=90% and FPR=10% produce PP=27% – which is what Turkish Muslims believe on average. But stronger evidence – say TPR=99% and FPR=1% – would force even them to admit PP=80%. As long as BR is non zero, sufficiently strong evidence must eventually convince the staunchest sceptics. The accumulation of evidence leads to the truth. Hence, apart from absolute Faith, the reason why millions believe that Iago did it needs to be found in the way they evaluate evidence.

Think of E as a collection of N independent pieces of evidence, E=(E₁, E₂, …, E_N), each with its own Likelihood Ratio LR=TPR/FPR. Bayes’ Theorem in odds form is PO=LR∙BO. Starting from any level of prior odds BO (except Faith: infinite or zero BO), each piece of evidence increases posterior odds PO if LR>1 (confirmative evidence) or decreases it if LR<1 (disconfirmative evidence). Evidence is cumulative: PO=LR₁∙LR₂∙…∙LR_N∙BO. For example, if E₁ has TPR=90% and FPR=10%, it will produce a nine-fold increase in prior odds. If E₂ has TPR=20% and FPR=80%, it will cut the ensuing odds by 4; and so on. The final PO will be the result of a tug of war between confirmative and disconfirmative evidence.

9/11 sceptics base their theories on evidence which they regard as disconfirmative. The most popular theory contends that the collapse of the Twin Towers was the result of a controlled demolition. The theory is dumb at best, and has been widely discredited. But let’s assume for a second that the sceptics are right. The point is that, even so, evidence is cumulative: one piece of disconfirmative evidence is worth next to nothing when set against a long string of near Smoking Guns. Therefore, to maintain their scepticism, the demolition theorists not only have to believe that their LR is lower than 1 (which most clearly is not), but also need to discredit all other evidence. How? Easy: Iago planted it. He wanted the world to believe that al-Qaeda did it. So he planted handkerchief E into their lodgings. Not only he orchestrated the attack, but also fabricated evidence to blame it on Al-Qaeda. Fabricated evidence is disconfirmative by definition: it is more likely if the hypothesis is false than if it is true. Hence its accumulation weakens the al-Qaeda hypothesis and strengthens its alternative.

Somehow cognisant of the absurdity of such a flight of fantasy, the Truthers maintain not only that LR<1, but that LR=0, i.e. TPR=0%. In that case, PO=0 irrespective of all other evidence: a Perfect Alibi. According to them, the hypothesis that the planes caused the collapse of the towers is not only improbable: it is impossible. With a Perfect Alibi, all other evidence, no matter how confirmative, becomes irrelevant. There is no need to ask how Iago did it: “When you have eliminated the impossible, whatever remains, however improbable, must be the truth.” A dispiritingly amusing anthology of such wacky Sherlockesque lucubrations can be found here.