Hand, The Improbability Principle

I would like to say that if you’ve taken a statistics and probability course you don’t need to read David J. Hand’s The Improbability Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day (Scientific American / Farrar, Straus and Giroux, 2014). But even sophisticated quants often seem not to understand why black swans may not be so uncommon after all.

Hand, emeritus professor of mathematics at Imperial College London, former president of the Royal Statistical Society, and the chief scientific adviser to Winton Capital Management, has written this book for the layman. No complicated equations here, just easy to follow prose.

What Hand calls the Improbability Principle “asserts that extremely improbable events are commonplace. It’s a consequence of a collection of more fundamental laws, which all tie together to lead inevitably and inexorably to the occurrence of such extraordinarily unlikely events. These laws, this principle, tell us that the universe is in fact constructed so that these coincidences are unavoidable: the extraordinarily unlikely must happen; events of vanishingly small probability will occur. The Improbability Principle resolves the apparent contradiction between the sheer unlikeliness of such events, and the fact that they nevertheless keep on happening.” (pp. 11-12)

Braided together to form the Improbability Principle are several laws. First, the law of inevitability, which says that of all possible outcomes, one of them must occur. Second, the law of selection, which says that you can make probabilities as high as you like if you choose after the event. Think of the classic example of a person painting bull’s eyes and targets around arrows shot into the side of a barn. Third, the law of truly large numbers, not to be confused with the law of large numbers. This one says that with a large enough number of opportunities, any outrageous thing is likely to happen. Fourth, the law of the probability lever, according to which a slight change in circumstances can have a huge impact on probabilities. And fifth, the law of near enough, which says that events that are sufficiently similar are regarded as identical.

On the face of it, these five laws may seem a peculiar way to think about highly unlikely events, and those expecting a rigorous argument in their favor will be disappointed. But I think this book serves two functions. It offers the novice an excellent overview of probability and statistics, and it challenges self-satisfied quants to reexamine their assumptions. For instance, it’s common knowledge that the distribution of market fluctuations isn’t normal, but even if one were to substitute the not-so-different Cauchy distribution for a bell curve the probabilities would be staggeringly different. In a normal distribution the probability of a 5-sigma event is 1 in 3.5 million; in a Cauchy distribution it’s 1 in 16. Ten-sigma, 20-sigma, and 30-sigma events in a normal distribution are highly unlikely (a 30-sigma event has a probability of 1 in 2.0 x 10¹⁹⁷), but in a Cauchy distribution they happen 1 in 32, 1 in 63, and 1 in 94 times. Fortunately, price fluctuations can’t be captured by a Cauchy distribution, but despite all the work done by statisticians, we still are shocked by market movements that, if price changes followed a normal distribution, are supposed to happen only about once every 100,000 years.

The Improbability Principle explains both how our Hardyesque world works and how we do and, more important, should think about it.