Of hockey sticks and statistics

[Updated at bottom] I recently read two books on climate science — or rather, two books on the controversies around climate science. One book was Michael Mann’s book The Hockey Stick and the Climate Wars; the other The Hockey Stick Illusion by Andrew Montford.

Now, I am, by inclination and tribal allegiance, of the party of Michael Mann, one of the world’s leading climate scientists. He and his colleagues have been subject to beastly treatment by political opponents, some of which is detailed in his book. And I only picked up the Montford book out of a sense of obligation to see what the opposing side was saying. And yet…

Montford’s book makes a pretty persuasive case. Not that climate science is bunk, or a conspiracy, or that anthropogenic global warming is a fiction — there is far too much converging evidence from different fields to plausibly make that claim (and indeed, Montford never makes such a claim) — but that a combination of egotism and back-scratching has seriously slowed down the process of evaluation and correction of sometimes sloppy statistical procedures, and tarnished the reputation of the scientific community generally.

I admit to a certain bias here: The attacks on Mann’s work that Montford describes are statistical in nature, and Mann’s response reminds me of the tone that is all too common when statisticians raise questions about published scientific work. Montford has a remarkable amount of technical detail — so much that I found myself wondering who his intended audience is — and the critiques he describes (mainly due to the Canadian mining engineer Steve McIntyre) seem eminently sensible. In the end, I don’t think they panned out, put they were genuine shortcomings in the early work, and McIntyre seems to have done the right things in demonstrating the failure of a statistical method, at least in principle, and to have earned for his trouble only incomprehension and abuse from Mann and his colleagues.

In particular, like many empirical scientists, Mann seems not to have understood — or not to have wanted to understand — the importance of making it possible for people wanting to check the methodology to reproduce exactly the calculations that led to the reported results, by making data and code available. Without that, the best anyone can do who is suspicious of the method is try to redo the calculations with new code; but if the results come out differently, you have no good way to establish where the error is, making a formal criticism and correction extremely challenging and time-consuming.

Montford reports that Mann refused to share the computer code with McIntyre, saying that to do so

would be giving in to the intimidation tactics that these people are engaged in.

When I read this in Montford’s book I thought, this can’t be true. This sounds like a libellous invention. But then, Mann repeated the comment in his own book, and seemed rather proud of it. His argument for why he wouldn’t release his source code, in full, was

(1) Our source code wasn’t necessary to reproduce and verify our findings. [Other scientists…] had independently implemented our algorithm without access to our source code. (2) Our source code was our intellectual property. [NSF], which had funded our work, had already established that we had more than met the standards of disclosure of data and methods expected of NSF-funded scientists and that the specific source code we had written was our intellectual property. (3) While I was happy to provide source code — and had — to colleagues (including competitors) who were engaged in good faith attempts to assess our methods, important precedents were at issue here. Did we really want to head down the slippery slope of releasing proprietary materials indiscriminately? What other vexatious demands might be made of us and others?

Was it Samuel Johnson who said that intellectual property is the last refuge of a scoundrel. It’s hard to see why this would be considered a vexatious demand at all by someone with nothing to hide. Now, it could be that what he had to hide was that the code was actually a mess. But since he had already shared it with some others, it shouldn’t have been too difficult to whip it into shape to share with McIntyre.

Professor Mann’s comments remind me of the remark made by historian John Lewis Gaddis in a 1983 article on “The Emerging Post-Revisionist Synthesis on the Origins of the Cold War”:

A decade ago this subject was capable of eliciting torrents of impassioned prose, of inducing normally placid professors to behave like gladiators at scholarly meetings, of provoking calls for the suppression of unpopular points of view, threats of lawsuits, and, most shocking of all, the checking of footnotes.

Gaddis remark was made tongue-in-cheek, of course, but Michael Mann would presumably have argued, “Do we want to head down the slippery slope of allowing people to check our footnotes? What will they want to check next.

Another palaeoclimatologist, Phil Jones, managed to top Mann’s chutzpah by replying to a request for the data from a publication by writing

We have 25 years invested in the work. Why should I make the data available to you, when your aim is to find something wrong with it?

Mann’s book has some technical defences of his work, and some of them are reasonably convincing, but his method is mostly ad hominem: His opponents are bad people, arguing in bad faith. “Their intent wasn’t so much to contribute to the scientific conversation as it was to influence the public discourse.” Everyone who criticises him is shown to have not quite the right credentials (and/or biased funding sources). They publish on web sites rather than in peer-reviewed journals. But when they do publish in peer-reviewed journals, they’re not the right journals, and peer review isn’t that useful anyway. (Something I agree with, see here.) He quotes another scientist saying of his critics McIntyre and McKitrick, “As far as I know they’re quacks.” (Rhetorical tip: Playground insults don’t become cogent arguments as soon as you quote someone else making them.)

On the other hand, Mann gets the vapours when McIntyre “titled a post about a highly respected NASA climate scientist with the rhetorical question ‘Is Gavin Schmidt Honest?'” Throughout his book, Mann alternates snide aspersions and ad hominem attacks on his opponents, while expressing shock at the much milder critical tone that comes from the global-warming sceptic camp. One particularly silly passage is his dismay at an economist Hu McCulloch, who alleged that “he could not reproduce a tropical ice core record produced by Lonnie Thompson and colleagues, implicitly claiming either ineptitude or, worse, malfeasance.” Is that really what someone is claiming when he fails to reproduce the results? If Thompson is anything like Mann in his attitude toward helping other researchers to reproduce his work, I can think of a lot of other reasons why he might have failed, and decided he just needed to let other researchers sort it out.

Although Montford’s book preceded Mann’s, Mann seems to take no cognizance of its claims and arguments. In one particularly telling non-exchange, Montford describes, with detailed correspondence, how their technical comment on Mann’s original hockey-stick paper was rejected, after several rounds of refereeing, “principally because the discussion cannot be condensed into our 500-word/1-figure format”. Mann reports that the comment was rejected “for lacking merit”, but provides no evidence, simply dismissing the claim that “lack of space” was an issue by citing Nature‘s published policy which does not treat length as a primary criterion. Indeed. I would say this rather strengthens Montford’s argument, that the rejection was, at least, suspicious.

Update 31-1-2014: The paragraphs below referred to a lawsuit that Michael Mann has brought against a blogger. Commenter Rob Honeycutt has pointed out that I misinterpreted the news reports, which prominently referred to the blogger comparing Mann’s abuse of data to a pedophile’s abuse of children. The libel suit is, apparently, based on the allegations of fraud in the article.

I should also cite prominently the critical book review of Montfort on the climate change science blog realscience.org. I certainly don’t mean to praise Montfort’s book, which is excessively tendentious, to the point of paranoia at times. I was just dismayed that Mann’s book, with the weight of scientific respectability on its side, isn’t any better.

Interestingly, Professor Mann has filed a libel suit, but not over Montford’s accusation that he has been, at least, uncooperative and high-handed with his critics, and at worst has worked behind the scenes to manipulate peer review of work in climate science. Rather, he has sued a journalist for  comparing his abuse of data to a pedophile’s abuse of children. So his failure to sue Montford cannot be attributed to any hesitancy to hoist the weapon of lawsuit in an intellectual battle, and must therefore be taken as tacit confirmation that the scandalous statements and at least the broad outlines of the behaviour attributed to him in the book are indeed accurate.

In my opinion, his suit against the blogger is ridiculous. That sort of grotesque language demeans the speaker, but it is still political satire. If the blog were attempting to spread the rumour that Mann is a child molester, that should certainly be libel. Whether the law supposes it libellous merely to compare him to a child molester I don’t know, since the fine points of libel law are so variable, and follow no principles that I understand. But I am sure that I think the law is a ass if it does suppose that. I hope the law won’t sue me for saying that… According to this article, the lawsuit could destroy the august conservative magazine National Review.

6 thoughts on “Of hockey sticks and statistics”

  1. What you seem to fail to grasp here is that, much of what’s in Montford’s book has been shown to be wrong. To read the two books side-by-side, without also evaluating the relative merits of each book’s claims would present you with a false balance.

    It’s absolutely necessary for scientists to challenge each other’s work. That is the core of what makes scientific discovery robust. Even Keith Briffa, a colleague of Mann’s, privately expressed contempt, early on, for Mann’s original Hockey Stick graph. But what did he do? Well, he certainly didn’t waste time trying to attack Mann’s methods. Why bother? Just do the research the way you think it should be done to see if you get similar conclusions! This is at the heart of the problem. McIntyre doesn’t do this.

    MBH98/99 was (I believe) the first full scale multiproxy temperature reconstruction ever produced. It was ground breaking work at that time. Do we ever expect first attempts to be perfect? That is why there have been over a dozen other multiproxy reconstructions done since MBH98/99.

    The question in scientific research is not, “Did you perform your research exactly as it should have been?” (Though critical responses to papers can be important.) The key question is, “Do your conclusions stand the test of time?” So far, every reconstruction done since MBH98/99 has produced concurring conclusions. That being, the warming of the past century is likely unprecedented in the last ~1000 years.

    What I find problematic about McIntyre’s work is,
    1) It’s one-sided. He never applies the same level of skepticism to any research that challenges man-made climate change.
    2) He has never, though he’s clearly capable, produced a multiproxy reconstruction the way he thinks it should be done.
    3) He primarily seems interested in creating material for the internet, for the purposes of attacking climate scientists, rather than engaging in actual scientific research on climate change. In this point, Montford was a perfect partner to create a misleading book about one scientist’s work.

    I would suggest further research into the claims made in Montford’s book. If you go a bit deeper I think you’ll find that much of what is claimed is actually baseless. And I think you’ll also find that the errors that have been exposed in Montford’s book have never been corrected. (As opposed to corrections Mann has made to his research.) Montford is perfectly happy to leave the errors in his book as they are because they delivers the message (however incorrect) that he wants.

    Here is one piece pointing out errors in Montford’s book.

    You also present an error here, when you say, “Rather, he has sued a journalist for comparing his abuse of data to a pedophile’s abuse of children.”

    This is not the subject of the libel case. The case is about Steyn saying that Mann’s work is “fraudulent.” The case centers around the idea that there was a “gross disregard for facts.” Mann’s research has been investigated numerous times and it’s been clearly shown that his work was not fraudulent. There is nothing about the case that references to Sandusky or pedophilia. You should read some of the court documents related to this case. They’re quite fascinating.

    In my own conclusion, I would suggest that McIntyre’s work went out of its way to try to cast doubt on Mann’s research, and in that process created as many errors of its own. Montford’s book takes that dubious effort and magnifies it for the purposes of attacking climate change science in general by vilifying a single piece of research by a single researcher.

    I have to say, Montford’s effort has been highly effective. In one lecture I saw, given by Dr Richard Alley, he recounted being in Washington speaking to a science committee where one high level member stated, “Well, we know all this climate change stuff is based on a fraudulent Hockey Stick graph.”

    I’m sure Don Montford appreciates your piece here perpetuating that position.

    1. Thank you for taking the time for such a long and detailed reply. I should begin by thanking you for correcting my comments on Mann’s libel suit. I should perhaps have mistrusted the news reports and gone to the primary sources, but it wasn’t of great interest to me. I merely wanted to make the point that Mann is not opposed to libel suits in principle, so I take the lack of a suit against Montford as evidence that the quotes are basically correct.

      For the rest, I am mystified by your summation that “Montford appreciates your piece here perpetuating that position,” that “this climate change stuff is based on a fraudulent Hockey Stick graph”. I’ve gone and reread my post, and I still cannot see how anyone can find any suggestion of fraud, or even that the Hockey Stick is essentially wrong. (This is a particularly vexatious accusation, since, as you have pointed out, Mann is prepared to sue people who accuse him of fraud.) I also state very clearly that the substance of Mann’s results has been pretty reliably established, and that McIntyre’s critiques seem, in the end, not to have been borne out. (I probably should have linked to the realclimate post, which you helpfully cite; I did read it, and found it persuasive in most points.)

      I fundamentally disagree with your claim that attempts to reproduce other people’s work are a waste of time. “Do the research the way you think it should be done…” No. You need to engage with the methodology that produced what you think is a dubious result. It’s no use everyone producing their own results, with no basis for comparing them. Monomaniacs may be no fun, but they can be very useful critics, since they’ll look at what you think is a plausible result and hunt for the methodological flaw.

      Your arguments that MBH98/99 was novel, hence inevitably flawed, are beside the point. I never (and as far as I can see, no one ever) said he was a bad person or a bad scientist because his paper wasn’t perfect. What I criticised was what seemed to be a sense of entitlement, to decide who is allowed to criticise his work. As a statistician who has often been on the receiving end of contemptuous statistics-schmatistics dismissals of methodological queries that sound a lot like what McIntyre was getting, I had a basic sympathy for his position.

      Basically, I finished reading Montford’s book with the feeling that he had caricatured Mann and his palaeoclimatology colleagues as a sort of cult. Turning to the Mann book for a corrective, I almost felt like he was committed to intensifying the impression.

      1. I apologize, I may have overstepped in my ending there. I spend a lot of time defending climate science from people far less reasonable than you (an understatement). Sometimes I can get carried away.

        What I would contend, though, is that you’re giving Montford and McIntyre far more credit than they deserve. I watched both these guys operate over many years now and I do not believe they are operating on a level that is the least bit reasonable.

        I fully understand Dr Mann’s reaction to not provide code to these guys, because they have not acted in good faith. They have consistently acted deliberately to inhibit the work of researchers rather than attempt to further scientific knowledge. With the case of the CRU at East Anglia, they barraged scientists with FOI requests for information that was already readily available to them. And, as with my example of Dr Alley, they’ve been quite effective in terms of perpetuating completely erroneous notions of climate science.

        What I think you’re interpreting as Mann’s perpetuation of the caricaturization from Montford’s book actually has more basis in reality than you might be aware of. Nor could you know this unless you were intimately involved in this issue. In his book Mann is attempting to demonstrate the levels to which “skeptics” are willing to go. It may seem absurd, but I think that is only because the levels they go to are so outrageous.

        I have to note that I didn’t say that attempts to reproduce scientific work are a waste of time. Done in good faith, it’s very important. My point was that what is most important are the results. Does the work stand the test of time.

        Still, to this day, after a dozen investigations showing Mann’s work was not fraudulent, and after a dozen other reconstructions showing the same results as Mann’s original work, each and every day I find people repeating the errors of McIntyre and Montford as if they were absolute truth.

        It’s frustrating, to say the least.

  2. Just for a reference, here is the Richard Alley lecture I was talking about. Go to minute 20:00.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: