can be written in odds form as:
Odds are the ratio between the probability that a hypothesis is true and the probability that it is false. PP/(1-PP) are the Posterior Odds, PO. BR/(1-BR) are the Prior (or Base) Odds, BO. TPR/FPR is the Likelihood Ratio, LR. Hence we can write:
PO = LR ∙ BO
Posterior Odds are a linear function of Prior Odds, with slope LR. The Likelihood Ratio transforms Prior Odds into Posterior Odds. When Posterior Odds are greater than Prior Odds, we can say that evidence E is confirmative with respect to hypothesis H, as the probability that H is true increases in the light of E. Evidence is confirmative if the Likelihood Ratio is greater than 1, i.e. if the True Positive Rate is greater than the False Positive Rate. TPR>FPR means that E is more likely when H is true than when it is false.
In our child footballer example, TPR=100% and FPR=5%, hence LR=20. The coach is very accurate: if a child is a champion, the coach is infallible at spotting him; and if he is not a champion, the coach is wrong only 5% of the times. A Likelihood Ratio of 20 transforms Prior Odds of 0.001 into Posterior Odds of 0.02. Through the update, the probability that the child is a champion increases 20 fold. However, it remains very low. What can the father do in order to gain more comfort? He can ask a second coach. Assuming that the second coach is independent of the first, and that he is as accurate, his opinion will shed more light on the child’s future. Starting from the new Prior Odds of 0.02, if the coach says that child is not a champion, then that’s it: the father should abandon all hopes. A 100% TPR means that the coach never misses a champion. So, if he says that a child is not a champion, he is always right. But if he says the child is a champion, then Posterior Odds increase to 0.4, i.e. the Posterior Probability increases to 29%. With a second positive opinion, the father can keep his dream alive. But for real support he needs a third coach. Assuming once more that the third coach is as independent and as accurate as the other two, a negative opinion will again kill all dreams, but a positive one will increase PP to 89%. Now we’re talking. True, the father can never be 100% sure: a fourth positive opinion would increase PP to 99.4%, a fifth to 99.97%, and so on. But the updating process converges to the truth.
Convergence occurs if evidence is confirmative: LR>1. The size of the Likelihood Ratio determines the speed of convergence but, as long as TPR>FPR, convergence is assured. If LR=1, there is no convergence: evidence is as likely when the hypothesis is true as when it is false. Imagine, for instance, that the coach gives his opinion based on a coin toss. Evidence provided by such an “opinion” would clearly be useless. With a coin toss, TPR=FPR=50%, hence PO=BO. But TPR and FPR don’t need to be 50%: as long as they are equal, there is no convergence. Overall Accuracy is defined as A=50%+(TPR-FPR)/2. LR=1 implies A=50%.
What happens if LR<1? In this case, evidence E is disconfirmative with respect to hypothesis H. E is more likely when H is false than when it is true. Imagine the coach was not just useless but utterly misleading. He is more likely to say that the child is a champion if the child is not a champion than if he is. If TPR is lower than FPR, Accuracy is lower than 50% and the updating process diverges from the truth.
The coach’s opinion represents soft evidence. His accuracy has not been properly measured through a controlled, replicable experiment. Hence it is in the eye of the beholder: it is accuracy as perceived by the observer, i.e. the observer’s confidence in using it as a sign for evaluating the probability that the hypothesis is true. It is the father who decides whether the coach’s Accuracy is higher, equal, or lower than 50%, i.e. whether the coach is confirmative, unconfirmative or disconfirmative. This is ultimately a matter of trust.
The idea that the accumulation of evidence leads to the truth is a powerful engine of progress. People may start from different priors, but as long as they look at the same evidence they should converge to the same truth. This assumes, however, that the evidence is trusted. Without trust there is no convergence. The ultimate reason why people believe weird things is not that they ignore the evidence but that they distrust it.
(Robert Matthews makes the same point here. But what he calls “hard facts” are not the same as hard evidence. They are soft evidence, which can be distrusted by otherwise rational people. Hard evidence is much more difficult to distrust – but, as with homeopathy, it can be done).
Distrust of evidence is why otherwise rational and knowledgeable people believe that lunar landings were fakes, that some secret powers killed JFK and destroyed the twin towers, that “alternative” medicine works, that Darwinian evolution is wrong, and hundreds of other weird things.
By the same token, trust can be manipulated by unscrupulous experts, who artificially boost their TPR and hide the consequent increase in FPR. But the most important source of experts’ power is the Prior Indifference Fallacy. If BR=50%, hence BO=1, then PO=LR: support equals accuracy. Under prior indifference, even a useless expert, worth as much as a coin toss, can produce a massive shift in probability estimates.
