The Silver standard

Who speaks for statistics?

Ace forensic psephologist Nate Silver has attracted quite a bit of attention lately, with his 4+-year-old blog devoted to his statistical model that is intended to provide a synoptic view of the entire range of public data to produce a single probabilistic prediction of the outcome. Now, there are some clear criticisms that could be made of his approach, and of his results — in particular, the obvious failure of his successive predictions to be martingales, as they would have to be if they were appropriately using all current information — but he has been remarkably clear and open about his procedures and principles, and his reasoning on matters large and small seems generally sound, if not necessarily compelling. It’s funny that his conclusions should arouse any controversy at all, given that they are hardly different (as Silver himself is quick to acknowledge) from the conclusions one would draw from a simplistic combination of poll results. His main contribution is in giving careful answers to the obvious critiques that could be proposed: What’s a reasonable estimate for the difference between state poll results and the actual election result? How correlated are polling errors? What’s the best way to average polls of varying qualities done over multiple days? And so on. In the end, the answer doesn’t differ much from what anyone with number sense would come up with in a few hours, but you don’t know that for sure until you do it. And Silver’s reputation derives from the sense and good care that he takes in posing these questions and resolving them.

(The failure of the martingale property is actually evidence of his honesty in following the model that he set up back in the spring. He clearly would have been capable of recognising the trends that other people can see in the predictions, and introducing an ad hoc correction. He didn’t do that.)
While it has been mainly the right wing attacking Silver — following that tribe’s propensity for magical thinking, as I have described before — there is also the left-wing lines of what I grew up with is this critique by Yale literature fellow Natalia Cecire. She uses Nate Silver as a foil to attack statistics in general. Her rhetoric really needs to be read in full, but here is a sample:

Statistics always pulls back from the claims it makes; if it did not do so, it would not be statistics. Statistics is an inherently puerile discipline, not because it is dominated by men but because its principles concord so strongly with the way we have constructed boyhood—an unrelenting commitment to the play of abstract forms above all else: above wishes, above belief, above ethics, its only ethics being a commitment to the rules of the game. It presumes being unable to really know “the answer,” except as defined and bounded by the game.

I have great sympathy for critiques of moral obtuseness of mathematics and mathematicians (something I have emphasised in attacking mathematical finance), but I am genuinely inclined to write this off as simple posturing. All the talk of “the game” seems like a cover for ignorance, since she never begins to explain what game statisticians are playing.  (I recognise that the word puerile is meant here in a technical sense, not as the epithet of everyday language. But then you read further and realise it is being used as a term of abuse after all.) This non-ethical “commitment to the rules of the game” sounds a lot like what other people might call “commitment to standards of intellectual rigor”, which don’t sound quite so puerile when put that way. And statistics (as opposed to mathematics) is distinguished precisely by the tension between formal rules and practical problem-solving, the need to accommodate statistical principles to the peculiarities of the data.

Perhaps what she has in mind is something like a conversation I once had with a fairly respected economist at a small meeting on ageing in London. This fellow had spoken against government plans for introducing new medical criteria for reëvaluating people’s entitlement to disability pensions, claiming that the benchmark for health status should have been benchmarked against general rising life expectancies. The general point seemed reasonable, but the numbers he was using to make his argument were all based on comparing two different averages that seemed incommensurable — I don’t remember the details, but it was something like comparing average years of life after diagnosis and average years of life lost. I wasn’t at all sure, since the presentation was rushed, so I buttonholed him at lunch to ask him to explain the point. My expectation was that he would just clear up my misunderstanding. What I was not prepared for was that he simply refused to discuss the technical point until “you explain to me where this is leading”. It took me a while to even understand what he meant, which was basically that he considered it unethical to work on getting the technical computations right, if that could lead to people having their benefits withdrawn. “I talk to these people. They’re scared. What are you going to do for them?” Statistics for him were a tool of propaganda, not a tool for helping to make well-informed decisions. And not the least of the problems with that approach is that you can only do the trick once. Once people get used to thinking of scientific analysis as propaganda, it’s no longer even very useful as propaganda. Perhaps this reflects a rot primarily in the field of economics, but it casts a shadow on the rest of the scientific world (and encourages the contempt for objective truth) that we see in NC’s critique above, and in the much-discussed Republican war on science.

Of course, in suggesting that she has no idea what she’s talking about I am just taking on the role of those she has already assigned to me, as a would-be castrator of the (sexually) innumerate:

[Ezra Klein writes: “Lots of pundits don’t like Nate Silver because he makes them feel innumerate. Then they criticize him and prove it.] Klein packs a slight dig at the Backlashers’ masculinity in that phrase, “makes them feel innumerate.” “Innumerate” is code for “inadequate,” but a particular kind of inadequate; it’s a castration complex built on an ignorance of statistics. Silver, as a methodologist and as a person, is “threatening” to traditional journalists (Ferenstein).

I definitely thought that this whole idea of treating all criticism as though it were necessarily a sublimation of the only genuine criticism, which is criticism of male genital potency hadn’t survived the 1970s. It’s telling that she considers the true weight of an accusation of innumeracy not that it is a deplorable intellectual failing, but it could only be a cover for an accusation of a really important failing, namely an insufficiently rigid sperm injector. Who could really be ashamed of a mere incapacity to reason where numbers are involved?

Cecire is the perfect pendant to the right-winger who attacked Silver for being gay and/or effeminate. There’s a lot of ad hominem vague opprobrium: It’s not that Silver is actually a talented thinker, not that people admire his careful work on a difficult and interesting problem, only that “Silver’s Wunderkind image creates kind of persona from whom we are prepared to receive statistical models”. Slightly bizarre, actually: I never thought of statistical models as being associated with a boyish image; I thought of them as being associated with these sorts of people:

I realise that I am not a member of the “we” to whom she refers, but I am skeptical of the claim that the general public expects valid statistical models to come from playful young men (as opposed to serious greybeards or international consortia).

I shouldn’t be too harsh on Dr. Cecire. There can be no doubt that she is right to lament the emphasis on “horserace” reporting in US political journalism. Feeding back predictions of who will win to the voting public is vacuous, and does not help voters make better decisions. On the other hand, it’s not clear that people would gobble down the more wholesome informational diet if only the journalists offered it to them. What she fails to appreciate is that there is a positive value, given that people are working to predict the election outcome, to doing the predictions properly, and that clearing away the nonsense is always a good thing. If done properly, having a single optimal prediction that requires less discussion can be precisely what you need to open up space for discussing more important matters — just as astronomers’ early obsession with predicting the motions of heavenly bodies was not exacerbated by the improved precision and simplicity of the Copernican system.

Furthermore, predicting elections may seem like a game, or mere entertainment, but prediction plays an important role in science, both as a test of core understanding of a process, and as a tool for generating hypotheses. Accepting that the formation of public opinion is more important than the measurement of public opinion, you can’t really examine the former without first having a grip on the the latter. Analysing the political process with only election returns as data would be like trying to develop an AIDS drug where the only observation you could make was when the patient dies. And that’s not to mention the crucial function of independent polling in ensuring fair elections, by serving as an independent check on the official vote tallies.

I’m also intrigued by her remark

The Silver backlash wants an answer, a position; it wants Silver to stop playing around. In other words, it reads statistics itself as waffling and double-tonguery. It’s not wrong in that sense. It just fails to appreciate that that is more or less the entire point of statistics: to measure what is irreducibly uncertain.

On the one hand, she seems to diagnose well the basic discomfort that most people have with randomness and probability. I had an acquaintance once who claimed to believe that the probability of winning the lottery is 1/2, since there are only two possible outcomes: Either I win, or I don’t. Most people understand that there is a difference between the likelihood of a coin coming up heads and the likelihood of winning a state lottery, even if the only prediction you can reliably make is “I don’t know what will happen”. On the other hand, I’m not sure what it means to say that the point of statistics is “playing around”, is “waffling and double-tonguery” because it measures what is irreducibly uncertain. Really, I’m not sure whether there’s a genuine insight here or not. It’s not unreasonable to say that an important component of statistics is to measure uncertainty, hence the famous dictum of statistician C. R. Rao “Uncertain knowledge + knowledge about the extent of uncertainty in it = useable knowledge”. But then she’s so excessive in her rhetoric that it’s hard to know if she’s just tossing together buzzwords and flowery phrases in equal measure. It’s not that the critics object to Silver’s “waffling”; if anything, they object to his specificity, assigning a probability rather than calling the election “a tossup” and leaving it at that.

One other point that occurs to me: As a person blind to the appeal of spectator sports, I’ve long been intrigued by a comment that Noam Chomsky made about the role of sport in a modern democracy. It’s very much in line with Marx’s “opium of the masses” analysis. He remarks on the exceptionally sophisticated analysis — observational and quantitative — that he heard average people producing on radio call-in programs, and compared them with the crude analysis that is typical in the political realm. He concludes that sport serves (whether by design or by accident) to soak up people’s analytical acumen, that might otherwise attach itself to the situation of people’s live, and thus find an outlet in politics.

So how are Silver and his ilk to be interpreted in this framework? I suppose Dr Cecire would say that they’re throwing up another smokescreen for those who are interested in politics, to help them think of politics as just another sporting event, so that they won’t apply their analytical intelligence to the substance of politics. I’m inclined to agree. But it could be just the opposite. Maybe people (like Silver) who were wasting their intelligence on understanding sport, are being drawn into politics. And maybe other people who were obsessing over the details of the sporting aspects of politics will now take a broader view, once the details of who’s winning have been adequately mastered by experts.