Occasional reflections on Life, the World, and Mathematics

The tyranny of the 95%


The president of the National Academy of Science is being quoted spouting dangerous nonsense. Well, maybe not so dangerous, but really nonsense.

I found this by way of Jonathan Chait, a generally insightful and well-informed political journalist, who weighed in recently on the political response to the IPCC report on climate change. US Republican Party big shot Paul Ryan, asked whether he believes that human activity has contributed to global warming, replied recently “I don’t know the answer to that question. I don’t think science does, either.” Chait rightly takes him to task for this ridiculous dodge (though he ignores the fact that Ryan was asked about his beliefs, so that his skepticism may reflect a commendable awareness of the cognitive theories of Stephen Stich, and his need to reflect upon the impossibility of speaking scientifically, or introspecting coherently, about the contents of beliefs), but the form of his criticism left me troubled:

In fact, science does know the answer. Climate scientists believe with a 95 percent level of certainty (the same level of certainty as their belief in the dangers of cigarette smoking) that human activity is contributing to climate change.

Tracking through his links, I found that he’d copied this comparison between climate change and the hazards of smoking pretty much verbatim from another blog, and that it ultimately derived from this “explanation” from the AP:

Some climate-change deniers have looked at 95 percent and scoffed. After all, most people wouldn’t get on a plane that had only a 95 percent certainty of landing safely, risk experts say.

But in science, 95 percent certainty is often considered the gold standard for certainty.

[…]

The president of the prestigious National Academy of Sciences, Ralph Cicerone, and more than a dozen other scientists contacted by the AP said the 95 percent certainty regarding climate change is most similar to the confidence scientists have in the decades’ worth of evidence that cigarettes are deadly.

Far be it from me to challenge the president of the National Academy of Sciences, particularly if it’s the “prestigious” National Academy of Sciences, or more than a dozen other scientists, but the technical term for this is “bollocks”. First of all, 95 percent is not a “gold standard” of anything. It’s a minimum threshold of the likelihood of an apparent effect not having occurred due to chance at which it is labelled “statistically significant”. It’s a pretty arbitrary threshold, mainly for a combination of historical reasons and computational convenience.

Second, while it is nearly impossible to conceive of putting a number on the probability that the entire consensus that cigarettes are deadly (or, in Chait’s even milder version, at the end of the game of Chinese Whispers, that they have “dangers”), if you did it would certainly be much MUCH smaller than 5%. Pretty much every individual study that was published excluded chance at the 5% level. There have now been about seven decades of such studies. For them to be consistently wrong there would have to be a fundamental error extending through all of them. Back in the early 1960s, at the time of the first Surgeon General’s report, most of the evidence was still observational and epidemiological, so there could have been some basic confounding going on. It should have seemed unlikely, and the probability couldn’t formally be measured, but calling it 5% at that time wouldn’t have been unreasonable. Since then, not only has the observational evidence been extended to different populations at different times, to different diseases, and including secondhand smoke (which should have completely different confounding effects, if that were at work), but it includes converging evidence from experimental studies and theoretical understanding of physiological mechanisms, and the direct causal results of large-scale manipulation of smoking through public-health measures. At this point, the claim that tobacco use harms health must be judged about as certain as the germ theory of infectious disease, or the theory of evolution by natural selection. (Okay, the latter is perhaps an inopportune comparison, for the American audience.)

Compared with that, it’s easy to see why the climate scientists would strike a more cautious note. There’s a lot of converging evidence, empirical and theoretical, but it’s still about one anomalous time period for a single planet, unlike the medical evidence about smoking we do not have empirical observations of many millions of individual cases under many different conditions; and the theoretical analysis is still pretty novel and complicated. I think they’ve presented the case fairly by suggesting a 5% residual uncertainty, and their cause is not being well served by suggesting that, well, scientists always hedge their bets, and if they say 95% sure, that’s about as sure as anyone could reasonably be.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Tag Cloud

%d bloggers like this: