We accumulate evidence in order to test hypotheses. Think of evidence as a sign consistent with a hypothesis being true. For example, if the hypothesis is: There is a fire, the evidence can be: There is smoke.
Evidence can give two right responses: True Positives (Smoke, Fire) and True Negatives (no smoke, no fire) and two wrong responses: False Positives (smoke, no fire) and False Negatives (no smoke, fire). False Negatives, i.e. wrongful rejections of the hypothesis, are known as Type I errors, or Misses; False Positives, i.e. wrongful acceptances of the hypothesis, are known as Type II errors, or False Alarms.
Ideally, we would like both errors to have the smallest probabilities. Perfectly accurate evidence is one that produces no Misses and no False Alarms. But it is rare. Typically, there is a trade-off between the two. If we mostly care about avoiding Misses (as is the case with fire), we need to accept a higher probability of False Alarms. If we mostly care about avoiding False Alarms, we must accept a higher probability of Misses. At the extremes, never rejecting the hypothesis would entirely avoid Type I errors, but would likely lead to a larger probability of Type II errors. Vice versa, always rejecting the hypothesis would eliminate Type II errors, but entail a higher probability of Type I errors.
In general, there is no reason for the two errors to have equal probabilities. For instance, in our child footballer example, the test is perfectly accurate at avoiding Type I errors (it is infallible at spotting top players: in our notation, FNR=0%) but has a small probability of Type II errors (FPR=5%). When the two error probabilities happen to be equal (FPR=FNR), evidence is symmetric. With symmetric evidence, Bayes’ Theorem becomes:
Hence, under Prior Indifference (BR=50%), we have PP=TPR, which is the Inverse Fallacy exactly.
The following figure gives a graphic depiction of the relationship between the Posterior Probability and the True Positive Rate for different levels of the Base Rate, for the particular case of symmetric evidence. The relationship is positive: the higher the level of accuracy, measured by TPR, the higher the level of support, measured by PP, for any given level of BR. However, the relationship is concave if BR>50%, and increasingly so as BR tends to 100%. Conversely, the relationship is convex if BR<50%, and increasingly so as BR tends to zero. Only if BR=50% the relationship is 45° linear, and PP=TPR.
The figure makes clear that the Inverse Fallacy is due to a failure to appreciate the increasing non linearity of the relationship between accuracy and support as the Base Rate BR departs from the 50% indifference level. The higher the Base Rate, the larger is the underestimation of the Posterior Probability. The lower the Base Rate, the larger is its overestimation. In particular, a small Base Rate (as in our child footballer example) implies a very convex relationship between PP and TPR, such that even a small departure from perfect accuracy implies a large drop of PP. For instance, with BR=1% (as in the figure) even a 1% drop from perfect accuracy (TPR=99%) implies a massive drop of PP all the way to 50%, as the probability of error FPR equals the Base Rate. If the probability of error is higher than the Base Rate, PP falls below 50%. For instance, with TPR=95%, PP drops to 16%, and with TPR=90% it drops to 8%. The Prior Indifference Fallacy hides the implications of convexity. Under prior indifference, a 1% drop in the level of accuracy translates into a 1% drop in the level of support: “Since the test is 99% accurate at spotting top players, and the test is saying that my child will be a top player, then he will be a top player with 99% probability”. Under prior indifference, accuracy equals support: all it takes for a symmetric test to be supportive is to be more accurate than a coin toss. In fact, imagine the test was only 50% accurate. This would be a useless test, and the correct conclusion, following Bayes’ Theorem, would be PP=BR: the probability that your child will be a top player after a positive test result should not move from the Base Rate. But under Prior Indifference, a positive test result – however worthless the test – would pull you towards the very wrong conclusion that your child’s chance of stardom is 50%.
How about that.
