The peer-review fetish: Let’s abolish the gold standard!

I’ve just been reading two books on the climate-change debate, both focusing on the so-called “hockey stick graph”: Michael Mann’s The Hockey Stick and the Climate Wars: Dispatches from the Front Lines, and A. W. Montford’s The Hockey Stick Illusion: Climategate and the Corruption of Science. I’ll comment on these in a later post, but right now I want to comment on the totemic role that the strange ritual of anonymous peer review plays for the gatekeepers of science.

One commonly hears that anonymous peer review (henceforth APR) is the “gold standard” for scientific papers. Now, this is a reasonable description, in that the gold standard was a system that long outlived its usefulness, constraining growth and innovation by attempting to measure something that is inherently fluid and abstract by an arbitrary concrete criterion, and persisting through the vested interests of a few and deficient imagination of the many.

That’s not usually what people mean, though.

An article is submitted to a journal. An editor has read it and decided to include it. It appears in print. What does APR add to this? It means that the editor also solicited the opinion of at least one other person (the “referee(s)”). That’s it. The opinion may have been expressed in three lines or less. She may have ignored the opinion.

Furthermore, to drain away any incentive for the referee(s) to be conscientious about their work,

• They are unpaid.
• They are anonymous. We know how well that works for raising the tone of blog comments.
• Anonymity implies: Their contributions will never be acknowledged. If they contribute important insights to the paper, they may be recognised in the acknowledgement section: “We are grateful for the helpful suggestions of an anonymous referee.” Very occasionally an author will suggest, through the editor, that a referee who has made important contributions be invited to join the paper as a co-author. More commonly, a paper will be sent from journal to journal, collecting useful suggestions until it has actually become worth publishing.*
• No one will ever take issue with any positive remarks the referee makes, as no one but the authors (and the editor) will ever see them. Negative comments, on the other hand, may get pushback from the author, and thus need to be justified, requiring far more work.
• Normally, the author will be forced to demonstrate that she has taken the referee’s criticism to heart, no matter how petty or subjective. This encourages the referee to adopt an Olympian stance, passing judgement on what by rights ought to be the author’s prerogative.

Of course, I don’t mean to say that most referees most of the time don’t do a very conscientious job. I take refereeing seriously, and make a good-faith effort to be fair, judicious, and helpful. But I’m sure that I’m not the only one who feels that the incentives are pushing in other directions, and to the extent that I do a careful job, it is mainly out of some abstract sense of duty. I am particularly irritated when I find myself forced to put original insights into my report, to explain why the paper is deficient. I would much rather the paper be published as is, and then I could make my criticism publicly, and then, if I’m right, be recognised for my contribution.

So, what is the advantage of APR? Well, it certainly helps the reliability of  to collect outside expertise on specialist material, though I can’t see why they should be anonymous. Sure, someone might feel intimidated about giving an honest opinion publicly, particularly on a very senior and respected researcher. On the other hand, a masked reviewer is more likely to act like an asshole, and the editor will surely be able to find someone who wants a platform to comment publicly on said senior respected researcher’s newest work.

In any case, the real peer review is the open discussion that comes after the paper is published. What is the value of one or two positive appraisals by parties unknown in comparison with that? (And if no one thinks it’s worth commenting on the work, then no one cares, so why force the issue?)

I don’t understand why, in this day and age, we should make a big fuss about getting a paper published. These days, journals mainly serve to impede scientific communication. This is particularly true of the big media trolls Science and Nature, which dominate the prestige market in certain fields (not mine, fortunately), and will never let accuracy get in the way of a good story. They “embargo” research, prohibiting authors from reporting on their results ahead of publication, to make sure that the journal can control the news cycle.

In mathematics, and many other fields, you write a paper and post it to the web. Journal publication is just a sort of acknowledgement. Nowadays we have much better technology for aggregating opinions to . Comments, blogs, numbers of downloads. Journals could become blogs that link to the papers, with public evaluations, explaining why this researcher thinks the paper is worth your time, and public critiques that will quickly make the community’s cumulative errata accessible and searchable.

Instead of me needing to choose a single journal to submit my paper to, balancing the proper audience against the prestige against the speed (and accuracy) of the reviewing process, the journals will seek my paper out. Or not. If no one cares it will just sink, unnoticed. The point is, it’s a mistake to think of scientific validity as a yes/no question, and it’s a mistake to think of scientific importance as something that can be ranked by a number like the “impact factor” of a journal.

The only losers in this transition would be the publishers. And good riddance to them. But, of course, that’s why, while no one would design our current system of journals with APR today, we may be stuck with it. The powerful publishers will see to it that we can’t get there from here. They’ll keep collecting their rents as the gatekeepers of science.

And science will stay in recession, as long as we can’t abandon the gold standard.

* I quote, in this context, a comment that appeared on the blog Crooked Timber, in response to Eszter Haggitai’s post on wanting to reveal publicly the comments she had made on a paper when refereeing it for a journal. The commenter, Chris Armstrong, writes

we understand that in publishing our work, we’re opening ourselves up for public criticism. But someone who submits their work to a journal can be taken to have opened themselves up to PRIVATE criticism and not necessarily to public criticism. They might be mortified to know that the weaknesses of this private version of their papers are then going to be aired before the public. If there are mistakes in the paper, that’s between the author and the editor and the reviewers, they might feel, and they should get a chance to correct them before publication.

(He or she states explicitly that this is expressing a social norm, not his or her own sentiment about journal submissions.) This is certainly not the norm in mathematics, where papers are usually posted to a preprint server, and otherwise made available to colleagues around the time that they are submitted for publication. And the idea of submitting papers for private criticism seems to me an abuse of the time and effort of the anonymous referees. If you want private criticism, ask your friends. They might ask you for a favour in return some day.