Once in a thousand years

This is from my book for bright young people, Playwiths. Read the fine print here, and you can have it for free.

Consider the number of years between events described as “once in a thousand years”, such as floods. To the layperson, this immediately raises the question: how can the authorities access data, covering several thousand years? The answer is that they can’t, but they have what is usually referred to as the Poisson distribution to fall back on, and to understand that, we need to consider an old tale of Prussian cavalrymen who were kicked in the head by their horses.

Just in case you know any French, the Poisson distribution has nothing to do with handing out fishes. It was developed by (and named after) Siméon-Denis Poisson. It describes the probability of clusters in random events, given nothing more than the average occurrence of such events. (If you have no French, their word for fish is poisson, leading to dreadful puns about one man's meat being another man's poisson, but that is irrelevant.

This on the right is not irrelevant, but it is, instead, an elephant, which is a horse of a different colour, as we say in the writing trade. Now let's get back to the horses...

Poisson died in 1840, before the Prussians were kicked. Ladislaus Bortkiewicz published a book in 1898 in which he tried out the distribution of head kicks in each of the 14 corps of Prussian cavalry over a 20-year period, to see if it matched Poisson’s predictions.

Basically, the Poisson distribution works like this: given a sample average (or better, a population average), you can predict the probability of clusters of, say, breast cancer cases in a workplace, the number of calls to a call centre in a given minute, power failures on a grid, some types of traffic accident, the number of typographical errors on a page and the failure of light bulbs. And given some flood data for a few inundations, the Poisson distribution can predict about how often there would be a flood of a certain level.

Let us consider the Prussian data: there were several cases where a significant number of kicks had happened, and many more where no kicks had happened, so Bortkiewicz got hold of the data for 200 corps-years. In 109 cases, there were no injuries, but there were 65 instances of one injury, 22 cases of two, three cases of three head-kicks and one unfortunate corps, in one year, had four instances, a total of 122 cases. That meant the probability of a case in any given corps in any given year was about 6/10, or if you want precision, 0.61.

Bortkiewicz triumphantly showed that the known distribution was an almost perfect fit to the theoretical prediction. After that, people everywhere took up Poisson’s idea enthusiastically.

This story was popular, because most of the world liked the idea of Prussian cavalry being kicked in the head, but the main point was to say that there would be variation, and a high “score” did not necessarily imply carelessness or anything else. Ask anybody who has done some basic statistics, and they will all know about the Prussian head-kicks. It’s the example that is always mentioned.

What is less-mentioned is that you can calculate the flood height that, based on prior data, would happen once in a thousand years. This figure would be approximate, and the estimates would be refined after each flood, and they would be slightly invalidated if the risk is increasing rather than steady, but it’s better than nothing if you need a predictor.

I actually began looking into this issue, revisiting it after several decades, because somebody was questioning the science behind climate change and global warming, and as a throw-away line, poked fun at councils in Australia which have maps showing the limits of one-in-a-thousand-year floods. How, the idiot asked, could anybody know what has happened in the past?

Those who know my historical interests will not be surprised to learn that I point to 1859 as the year when scientists in unrelated disciplines began to be unable to understand one another. The public had started to feel lost around science a few years earlier, but after the 1860s, a great deal of science was either counter-intuitive or it relied on obscure methods. One way and another, science all got progressively more complicated.

Counter-intuitive science is in some ways the worst source of dissent and confusion, but if we know that mathematicians have a clever wrinkle that lets them estimate what a one-in-a-thousand-year flood would be like, we can accept that. The science that flies in the face of uninformed ‘common sense’, and the science that causes fears to arise, these are the sorts of science that cause trouble.

Even if the ancient Greeks knew that the world was a sphere, peasant minds were happy to say that the world they saw was clearly flat. In the same way, other equally simple and fearful peasant-quality minds attack the idea of evolution, misrepresenting what evolution is, even as they deny it. 

Climate is another case: the modern peasants who watch the weather on TV thinks they understand climate, but that, in fact, is a very different kettle of poissons.