Can you trust statistics?

This was a set of two talks, recorded somewhere around 1991 -- it ended up gaining a 'highly commended' in the Michael Daley awards.



As this speaks of health matters, John Snow's pump in Broad Street gains a few mentions in my talks, because it is my favourite example for lots of reasons.

I found where Broad Street used to be when I was in London -- though the Soho people I asked, mentioning a cholera outbreak were horrified, stating vehemently but in a low and urgent voice that there had never been cholera there, while looking around furtively, to make sure no tourists were in earshot.

Just around the corner, I found a replica of the pump, a plaque on which the name of a parochial politician was in larger print than Snow's, and also the John Snow pub. I went in and drank a beer to the great man (Snow, NOT the Councillor) , and bought a souvenir t-shirt. Now read on . . .


* * * * *

I'm a reformed smoker. I gave up many years ago, but I did it for political reasons. I just wasn't prepared to support the government of the day by paying tax on both liquor and tobacco, so I gave up the demon weed. Soon after that, the government changed, but I didn't like the other mob very much either, so I kept on not smoking.

Now it goes without saying that reformed smokers are tiresome people. At least if you're a smoker they are. They will keep on at you, trying to get you to stop as well, so they can hang another scalp on their thoroughly smug and sanctimonious belts.

To all non-smokers, those who still puff smoke are tiresome people, who can't see the carcinoma for the smoke clouds. Stupid fools who deny any possibility of any link between smoking and anything.

Like the tobacco pushers, the smokers dismiss the figures contemptuously as "only statistics". The really tiresome smoker will even say a few unkind things about the statisticians who stand behind the figures. Or about the statisticians who lie behind the figures.

But the smokers don't just settle for simple attacks, like alleging the statisticians are secret non-smokers. It's all very much nastier than that. So much nastier, there must be a deeply ingrained cultural hatred of statisticians in our society, and probably an equal contempt or detestation for statistics as well.

At one stage in my infamous career, I freely confess to having quite a lot to do with gathering statistics, and messing about with numbers, an honourable and harmless activity, I would have thought. But it was then I discovered that such people, while sometimes accused gently of being "mathematicians", more often suffer the far heavier opprobrium of being called "statisticians".

I certainly encountered this all-too-human tendency, and all too often at that. The things people used to say about statistics and the users of statistics offended me greatly.

If you've never suffered from being called a statistician, you may think it's a minor inconvenience to suffer. That's only because you've never heard the jokes which go with the label: there are more jokes about statistics than I know about the dismal science of economics, even if you let me throw in all of the many jokes I know about the economists as well.

Take, for example, the definition of a statistician as "somebody who's rather good around figures, but who lacks the personality to be an accountant". Or the story about the statistician who drowned in a lake with an average depth of six inches.

These days of course, we really ought to say fifteen centimetres, rather than six inches, but I'd like to stress the hoary agedness of so many of these witticisms about statistics: my reasons will become clear soon enough.

Then there are other clever-clogses who earnestly assure and advise us that a statistician collects data and draws confusions. We are also told statisticians are people who draw mathematically precise lines from an unwarranted assumption to a foregone conclusion.

With this sort of bias floating around, it's little wonder the great physicist, Lord Rutherford, once harumphed, "If your experiment needs statistics, then you ought to have done a better experiment".

Then there was the cruel and cutting comment, attributed to various witty people, and said to be about various people, that "X uses statistics much as a drunkard uses a lamp-post: rather more for support than for illumination".

On a slightly different tack, but still a debunking one, people will sometimes assert that statistics show how the vast majority of people have more than the average number of legs. Which is a bit like the common discovery, popular with conservatives, that tests reveal half our nation's school leavers to be below average.

Or the mildly sexist one-liner that statistics are like bikinis: what they reveal is interesting, but what they conceal is vital. And politicians like to get into the act as well, so we find Fiorello La Guardia, one-time mayor of New York saying statistics are like psychiatrists -- alienists he calls them -- statistics are like psychiatrists because they'll testify for either side.

Finally, there's the grand-daddy of them all, the famous line about "Lies, damned lies, and statistics". Now quickly answer out loud, so there's no cheating: who was it who first said that?

The odds are if there are two or three know-alls in your house, you'll now be locked in bitter dispute. At least, I hope you are. It may be hard on you, but it will help me prove my point.

The official version is that this line was first voiced by Mr. Disraeli, the well-known politician, but many quite reputable and reliable reference books attribute it to no less a personage than the author Mark Twain.

Now you can see why I expect to have started a few arguments by asking you to say your answers out loud. Even the experts can't agree on who it was said it! So if the authorities can't answer with one clear voice, how could you and your neighbours?

Well, the true facts of the case are fairly simple. That catchy snippet about "Lies, Damned Lies" et cetera was first published by Mark Twain all right, this can be proven to anybody's satisfaction, but Twain attributed the line to Disraeli.

The only problem is this: search as hard as you like, you won't find the story in any earlier publication than Twain's "Autobiography". In short, Mark Twain made the whole thing up! Disraeli never spoke those words: Twain invented them all, but he wanted the joke to have a greater force, and so gave the credit to an English politician.

Twain wasn't only well-known for his admiration of a good "Stretcher" (of the truth, that is), he even lied when he was talking about lies, and his name wasn't even Mark Twain, but Samuel Clemens! Now would you buy a used statistic from this man?

Come to think of it, the yarn's pedigree should have been enough in itself to cast doubt on the its veracity, with an arch-liar like Twain quoting, of all things, a politician! Yes, sad but true, there are more jokes about politicians than there are about statisticians, but only by a short head. It must be because so many politicians are trained originally as economists.

When you look to the background of the "Damned Lies" story as I did recently, there's an even stronger link between statistics and politics. Last century, when Disraeli is supposed to have made the remark, statistics were just numbers about the State. The state of the State, all summed up in a few simple numbers, as it were.

Now governments being what they are, or were, there was more than a slight tendency in the nineteenth century to twist things just a little, to bend the figures a bit, to bump up the birth rate, or smooth out the death rate, to fudge here, to massage there, to adjust for the number you first thought of, to add a small conjecture or maybe to slip in the odd hypothetical inference.

It was all too easy to tell a few small extravagances about one's armaments capacity, or to spread the occasional minor numerical inexactitude about whatever it was rival nations wanted to know about, and people did just that.

By the end of the last century, though, statistics were no longer the mere playthings of statesmen, and we find Francis Galton explaining that the object of statistics "...is to discover methods of condensing information concerning large groups of allied facts into brief and compendious expressions suitable for discussion".

So while you can go on sniping or objecting about being reduced to a mere statistic, those poor old statistics are still doing a fine job.

It's a pity, though, they've such a bad image problem, especially if you're trying to convert a diehard smoker from his or her evil ways.

As somebody once observed, or should have done if they didn't, figures don't lie, it's just that liars can figure. Presumably we don't set out to deceive ourselves deliberately: but could we use statistical information in such a way as to be unintentionally misled? I think it's very possible. Like fire, statistics make a good servant, but a bad master.

From Galton's time on, his meaning of statistics as some sort of numerical summary has become generally accepted, and the addition of "tests of significance" has added hugely to the number of statistics we can use.

So if the word "statistics" no longer means what it did when Mr. Disraeli didn't really make his comment, then it hardly seems fair to keep on giving statistics such a cruel and unusual treatment.

But as I implied before, I won't rush to the defence of your average number-abuser. If somebody does a Little Jack Horner with a pie that's absolutely bristling with statistical thingummies and they produce just one statistical plum, I won't be impressed at all: the plum's rather more likely to be a lemon, anyhow.

There are several handy little tests I apply to any figures and statistics which come my way: either the figures pass or they fail. These tests let me decide whether I'll take any notice of the figures or not. Statistical tests are very useful, especially if somebody is trying to prove by statistics that X causes Y.

In the first place, I want to know if there is a plausible reason why X might cause Y. If there isn't, then it's all very interesting, and I'll keep a look-out, just in case a plausible reason pops up later, but I won't rush to any conclusion. Not just yet, I won't.

Secondly, I want to know how likely it is that the result could have been obtained by chance. After all, if somebody claims to be able to tell butter from margarine, you wouldn't be too convinced by a single successful demonstration, would you?

Well, perhaps you would: certain advertising agencies think so, anyway. So let's take another tack: if you tossed a coin five times, you wouldn't think it very significant if you got three heads and two tails. Not unless you were using a double-headed coin, maybe.

If somebody guessed right three, or even four, times out of five, on a fifty-fifty bet, you might still want more proof. You should, you know, for there's a fair probability it was still just a fluke, a higher probability than most people realise. There's just about one chance in six of correctly guessing four out of five fifty-fifty events.

Now back to the butter/margarine dichotomy. Getting one right out of one is a fifty-fifty chance, while getting two right out of two is a twenty five per cent chance, still a bit too easy, maybe. So you ought to say "No, that's still not enough. I want to see you do it again!".

Statistical tests work in much the same way. They keep on asking for more proof until there's less than one chance in twenty of any result being just a chance fluctuation. The thing to remember is this: if you toss a coin often enough, sooner or later you'll get a run of five of a kind, and much more often than you'll fill an inside straight at poker.

As a group, scientists have agreed to be impressed by anything rarer than a one in twenty chance, quite impressed by something better than one in a hundred, and generally they're over the moon about anything which gets up to the one in a thousand level. That's really strong medicine when you get something that significant.

There. Did you spot the wool being pulled down over your eyes, did you notice how the speed of the word deceives the eye, the ear, the brain and various other senses? Did you feel the deceptive stiletto, slipping between your ribs? We test statistics to see how "significant" they are, and now, hey presto, I'm asserting that they really are significant. A bit of semantic jiggery-pokery, in fact.

And that's almost as bad as the sort of skullduggery people get up to when they're bad-mouthing statistics. Even though something may be statistically significant, it's a long way away from the thing really being scientifically significant, or significant as a cause, or significant as anything else, for that matter.

As I said earlier, statistics make good servants but bad masters. We need to keep them in their places. But we oughtn't to refuse to use statistics, for they can serve us well.

Some little time ago, I talked about Dr. John Snow, the man who solved a cholera epidemic in London in 1853. He did it by having the handle taken off a pump in Broad Street which was supplying polluted water. The story interested me, and I ended up researching it rather more deeply than I needed to, and I learned about some interesting side issues. Let me share one of them with you now.

During that same cholera epidemic in 1853, not ten minutes' walk from the Broad Street pump, in London's Middlesex Hospital, an unknown woman of thirty-three was helping to look after some of Snow's patients, and many other victims of the epidemic as well. It offered her some relief from the tedium of middle-class Victorian era spinster life, but her decision was a world-shaking one, nonetheless.

To us, she's no mere unknown, for that quiet spinster was Florence Nightingale. And while most people know her as the woman who founded the modern profession of nursing, there are just a few of us who know of her other claim to fame: as a founder of the art and science of statistics.

I'll come back to her in my next talk, and to whether the ABC is secretly driving you insane, and why all the podiatrists in New South Wales seem to be turning into public telephone boxes in South Australia. Or why I think that's what is happening.

My grandmother was one Florence Evans. Not an unusual name, they told me, lots of Evanses in Wales they said, so when I visited her native village of Manorbier, there was some doubt as to just which of several Florence Evanses I was talking about.

Still, after old Mrs Ogmore-Pritchard had eliminated the one who died at seventeen, and the one who died an old maid, she recalled the one who emigrated to wild colonial parts, and that was my Flo Evans.  From there, we found surviving relatives and said g'day.

Anyhow, Florence is a common enough name these days, but in 1820, it wasn't at all common. Only Florence Nightingale carried the name back then, and that was because she was born in the city of Florence, in a room with, by the sounds of it, a truly marvellous view.

It was only later, when Miss Nightingale became world-famous as the founder of modern nursing, that other young girls were also named Florence, in honour of the Lady with the Lamp. Aside, that is, from Percy Florence Shelley, the son of the poet, who was also born in Firenze. But for Ms Nightingale, the name might have remained non-gender-specific.

And yet, Florence the First, Florence Nightingale, could quite easily have turned into a fairly good mathematician: anybody with the steely resolve to break into nursing as it was in those days, when it was peopled by drunks and retired prostitutes, anybody game enough to take on all of that, could have done just about anything at all.

And certainly Florence had the interest in mathematics, and she had the ability. Unfortunately, she bowed to her father's wishes, and abandoned her interests. Or did she? After her name was made famous in the Crimea, Florence Nightingale returned to London in 1857, and started to look at statistics, and the way they were used.

First, she prepared a pamphlet, based on the report of a Royal Commission, studying the Crimean campaign. This little work, named "Mortality in the British Army", is generally believed to feature the first-ever use of pictorial charts to present statistical facts. Graphs, in fact, the origin of all those rinky-dinky little diagrams, beloved of geography teachers, you know the ones, with wheat bags, or oil barrels or human figures lined up like little paper dolls, or skittles, or whatever.

In the following year, 1858, Miss Nightingale was elected to the newly formed Statistical Society, just as she turned her attention to hospital statistics on disease and mortality.

In essence, she said, you could never discover trends in the data if everybody went happily around, concocting their own special data in their own sweet ways. You had to make everybody keep their figures in the same way. And so she prepared her scheme, published in 1859, for uniform hospital statistics. Her aim? No less than to compare the mortality figures for each disease in different hospitals, a thing which just could not be done under the old methods.

As in other spheres, Florence Nightingale was a success here, too, so the Statistical Congress of 1860 had, as its principal topic, her scheme for uniform hospital statistics.

These days, we use rather more sophisticated methods. It won't be sufficient just to say Hospital X loses more patients than Hospital Y does, so therefore Hospital X is doing the wrong thing. We need to look at the patients at the two hospitals, and make allowances for other possible causes. We have to study the things, the variables, which change together.

Earlier, I suggested that statistics are best regarded as convenient ways of wrapping a large amount of information up into a small volume. A sort of short-hand condensation of an unwieldy mess of bits and pieces.

And one of the handiest of these short-hand describers is the correlation coefficient, a measure of how two variables change at the same time, the one with the other.

Now here I'll have to get technical for a moment. You can calculate a correlation coefficient for any two variables, things like number of cigarettes smoked, and probability of getting cancer. The correlation coefficient is a simple number which can suggest how closely related two sets of measurements really are.

It works like this: if the variables match perfectly, rising and falling in perfect step, the correlation coefficient comes in with a value of one. But if there's a perfect mismatch, where the more you smoke, the smaller your chance of surviving, then you get a value of minus one.

With no match at all, no relationship, you get a value somewhere around zero. But consider this: if you have a whole lot of tennis balls bouncing around together, quite randomly, some of them will move together, just by chance. No cause, nothing in it at all, just a chance matching up. And random variables can match up in the same way, just by chance. And sometimes, that matching-up may have no meaning at all.

So this is why we have tests of significance. We calculate the probability of getting a given correlation by chance, and we only accept the fairly improbable values, the ones that are unlikely to be caused by mere chance.

The trouble is, all sorts of improbable things do happen by chance. Winning the lottery is improbable, although the lotteries people won't like me saying that. But though it's highly improbable, it happens every day, to somebody. With enough tries, even the most improbable things happen.

So here's why you should look around for some plausible link between the variables, some reason why one of the variables might cause the other. But even then, the lack of a link proves very little either way. There may be an independent linking variable.

Suppose smoking was a habit which most beer drinkers had, suppose most beer drinkers ate beer nuts, and just suppose that some beer nuts were infected with a fungus which produces aflatoxins that cause slow cancers which can, some time later, cause secondary lung cancers.

In this case, we'd get a correlation between smoking and lung cancer which still didn't mean smoking actually caused lung cancer. And that's the sort of grim hope which keeps those drug pushers, the tobacco czars going, anyhow. It also keeps the smokers puffing away at their cancer sticks.

It shouldn't, of course, for people have thrown huge stacks of variables into computers before this. The only answer which keeps coming out is a direct and incontrovertible link between smoking and cancer. The logic is there, when you consider what the cigarettes contain, and how the amount of smoking correlates with the incidence of cancer. It's an open and shut case.

I'm convinced, and I hope you are too. Still, just to tantalise the smokers, I'd like to tell you about some of the improbable things I've been getting out of the computer lately. They aren't really what you might call damned lies, and they are only marginally describable as statistics, but they show you what can happen if you let the computer out for a run without a tight lead.

Now anybody who's been around statistics for any time at all knows the folk-lore of the trade, the old faithful standbys, like the price of rum in Havana being highly correlated with the salaries of Presbyterian ministers in Massachusetts, and the Dutch (or sometimes it's Danish) family size which correlates very well with the number of storks' nests on the roof.

More kids in the house, more storks on the roof. Funny, isn't it? Not really. We just haven't sorted through all of the factors yet.

The Presbyterian rum example is the result of correlating two variables which have increased with inflation over many years. You could probably do the same with the cost of meat and the average salary of a vegetarian, but that wouldn't prove anything much either.

In the case of the storks on the roof, large families have larger houses, and larger houses in cold climates usually have more chimneys, and chimneys are what storks nest on. So naturally enough, larger families have more storks on the roof. With this information, the observed effect is easy to explain, isn't it?

There are others, though, where the explanation is less easy. Did you know, for example, that Hungarian coal gas production correlates very highly with Albanian phosphate usage? Or that South African paperboard production matches the value of Chilean exports, almost exactly?

Or did you know the number of iron ingots shipped annually from Pennsylvania to California between 1900 and 1970 correlates almost perfectly with the number of registered prostitutes in Buenos Aires in the same period? No, I thought you mightn't.

These examples are probably just a few more cases of two items with similar natural growth, linked in some way to the world economy, or else they must be simple coincidences. There are some cases, though, where, no matter how you try to explain it, there doesn't seem to be any conceivable causal link. Not a direct one, anyhow.

There might be indirect causes linking two things, like my hypothetical beer nuts. These cases are worth exploring, if only as sources of ideas for further investigation, or as cures for insomnia. It beats the hell out of calculating the cube root of 17 to three decimal places in the wee small hours, my own favourite soporific.

Now let's see if I can frighten you off listening to the radio, that insomniac's stand-by. Many years ago, in a now-forgotten source, I read there was a very high correlation between the number of wireless receiver licences in Britain, and the number of admissions to British mental institutions.

At the time, I noted this with a wan smile, and turned to the next taxing calculation exercise, for in those far-off days, all correlation coefficients had to be laboriously hand-calculated. It really was a long time ago when I read about this effect.

It struck me, just recently, that radio stations pump a lot of energy into the atmosphere. In America, the average five-year-old lives in a house which, over the child's life to the age of five, has received enough radio energy to lift the family car a kilometre into the air. That's a lot of energy.

Suppose, just suppose, that all this radiation caused some kind of brain damage in some people. Not all of them necessarily, just a susceptible few. Then, as you get more licences for wireless receivers in Britain, so the BBC builds more transmitters and more powerful transmitters, and more people will be affected. And so it is my sad duty to ask you all: are the electronic media really out to rot your brains? Will cable TV save us all?

Presented in this form, it's a contrived and, I hope, unconvincing argument. Not that it matters much, even switching off right now won't stop the radiation coming into your home, so lie back and enjoy it while you can! My purpose in citing these examples is to show you how statistics can be misused to spread alarm and despondency. But why bother?

Well, just a few years ago, problems like this were rare. As I mentioned, calculating just one correlation coefficient was hard yakka in the bad old days. Calculating the several hundred correlation coefficients you would need to get one really improbable lulu was virtually impossible, so fear and alarm seldom arose.

That was before the day of the personal computer and the hand calculator. Now you can churn out the correlation coefficients faster than you can cram the figures in, with absolutely no cerebral process being involved.

As never before, we need to be warned to approach statistics with, not a grain, but a shovelful, of salt. The statistic which can be generated without cerebration is likely also to be considered without cerebration. Which brings me, slowly but inexorably to the strange matter of the podiatrists, the public telephones, and the births.

Seated one night at the keyboard, I was weary and ill at ease. I had lost one of those essential connectors which link the parts of one's computer. Then I found the lost cord, connected up my computer, and fed it a huge dose of random data.

Well, not completely random, just deliberately different. I told it about the rattiest things I could dredge up, all sorts of odds and sods from a statistical year-book that just happened to be lying around. In all, I found twenty ridiculously and obviously unrelated things, so there were one hundred and ninety correlation coefficients to sift through. That seemed about right for what I was trying to do.

When I was done, I pressed button B, switched on the printer, and sat back to wait for the computer to churn out the results of its labours. The first few lines of print-out gave me no comfort, then I got a good'n, then nothing again, then a real beauty, and so it went.

At the end, I scanned the results.

I saw that NSW podiatrists' registrations showed a correlation of minus point nine eight with the number of South Australian public telephones, and minus point nine six with the Tasmanian birth rate. The Tasmanian birth rate in turn correlated plus point nine four with the South Australian public phones.

Well of course the podiatrists and phones part is easy. Quite clearly, New South Wales podiatrists are moving to South Australia and metamorphosing into public phone boxes. Or maybe they're going to Tasmania to have their babies, or maybe Tasmanians can only fall pregnant in South Australian public phone booths.


Or maybe codswallop grows in computers which are treated unkindly. As I said, figures can't lie, but liars can figure. I would trust statistics any day, so long as I can find out where they came from, and I'd even trust statisticians, so long as I knew they knew their own limitations. Most of the professional ones do know their limitations: it's the amateurs who are dangerous.

I'd even use statistics to choose the safest hospital to go to, if I had to go. But I'd still rather not go to hospital in the first place. After all, statistics show clearly that more people die in the average hospital than in the average home.