Once in a thousand years

Yes, I've been busy again: there's some teaching happening (I am a volunteer scientific visitor in a local school), and I have been cleaning up two new books.

One of these is called Playwiths, and it is based on a website of mine that has pulled in around 4 million hits over 20 years. The following is a small sample from the completed text, which as the first link reveals, is about STEAM: Science, Technology, Engineering, Arts and Mathematics.

Consider the number of years between events designated as “once in a thousand years”, such as floods. To the lay person, this immediately raises the question: how can the authorities access data, covering several thousand years. The answer is that they can’t, but they have what is usually referred to as the Poisson distribution to fall back on, and to understand that, we need to consider an old tale of Prussian cavalrymen who were kicked in the head by their horses.

If you know any French, the Poisson distribution has nothing to do with handing out fishes. It was developed by (and named after) Siméon-Denis Poisson. It describes the probability of clusters in random events, given nothing more than the average occurrence of such events. Poisson died in 1840, before the kicked Prussians died. Ladislaus Bortkiewicz published a book in 1898 in which he tried out the distribution of head kicks in each of the 14 corps of Prussian cavalry over a 20-year period, to see if it conformed to Poisson’s predictions.

Basically, the Poisson distribution works like this given a sample average (or better, a population average), you can predict the probability of clusters of, say, breast cancer cases in a workplace, the number of calls to a call centre in a given minute, power failures on a grid, some types of traffic accident, the number of typos on a page and the failure of light bulbs. And given some flood data for a few inundations, the Poisson distribution can predict about how often there would be a flood of a certain level.

Let us consider the Prussian data: there were several cases where a significant number of kicks had happened, and many more where no kicks had happened, and Bortkiewicz got hold of the data for 200 corps-years. In 109 cases, there were no injuries, 65 instances of one injury, 22 cases of two, 3 cases of three head-kicks and one unfortunate corps, in one year, had four instances, a total of 122 cases. That meant the probability of a case in any given corps in any given year was about 6/10, or if you want precision, 0.61.

Bortkiewicz triumphantly showed that the known distribution was an almost perfect fit to the theoretical prediction. After that, people everywhere took up Poisson’s idea enthusiastically.

This sort of thing always catches on with the mob, because most of the world favoured the idea of Prussian cavalry being kicked in the head, but the main point was to say that there would be variation, and a high “score” did not necessarily imply carelessness or anything else. Ask anybody who has done some basic stats, and they will all know about the Prussian head-kicks.

It’s the one that is always mentioned. What is less-mentioned is that you can calculate the flood height that, based on prior data, would happen once in a thousand years. This figure would be approximate, and the estimates would be refined after each flood, and they would be slightly invalidated if the risk is increasing rather than steady, but it’s better than nothing as a predictor.

I actually began looking into this issue, revisiting it after several decades, because somebody was questioning the science behind global warming, and as a throw-away line, poked fun at councils in Australia which have maps showing the limits of one-in-a-thousand-year floods.

Those who know my interests will not be surprised to learn that I point to 1859 as the year when scientists in unrelated disciplines began to be unable to understand one another. The public had started to be lost a few years earlier, but after the 1860s, a great deal of science was either counter-intuitive or relied on obscure methods, and it all got progressively more complicated.

Counter-intuitive science is in some ways the worst source of dissent and confusion: if we know that mathematicians have a clever wrinkle that lets them estimate what a one-in-a-thousand-year flood would be like, we can accept that. Science that flies in the face of uninformed “common sense”, science that causes fears to arise, these are the sorts of science that cause trouble. Even if the ancient Greeks knew that the world was a sphere, peasant minds were happy to say that the world they saw was clearly flat.

In the same way, other equally simple and fearful minds attack the idea of evolution, misrepresenting what evolution is, even as they deny it. Climate is another case: everybody who watches the weather on TV thinks he or she understands climate, which is a very different kettle of poissons.