Category: Technical

Horse thieves and inverse probabilities

Reading Ron Chernow’s magisterial new biography of Ulysses Grant, I came across this very correct statistical inverse reasoning from the celebrated journalist Horace Greeley (whose role in the high school history curriculum has been reduced to the phrase, “Go West, young man” — that he denied having invented):

All Democrats are not horse thieves, but all horse thieves are Democrats.

This seems like an ironic bon mot, but after he became the Democratic candidate for president against Grant in 1872 he tried to use a milder version unironically as a defence of his new party colleagues:

I never said all Democrats were saloon keepers. What I said was all saloon keepers are Democrats.

Presumably he meant to add that if we knew the base rate of saloonkeeping (or horse thievery) in the population at large, we could calculate from the Democratic vote share the exact fraction of Democrats (and of Republicans) who are saloonkeepers (or horse thieves).

Why people hate statisticians

Andrew Dilnot, former head of the UK Statistics Authority and current warden (no really!) of Nuffield College, gave a talk here last week, at our annual event honouring Florence Nightingale qua statistician. The ostensible title was “Numbers and Public policy: Why statistics really matter”, but the title should have been “Why people hate statisticians”. This was one of the most extreme versions I’ve ever seen of a speaker shopping trite (mostly right-wing) political talking points by dressing them up in statistics to make the dubious assertions seem irrefutable, and to make the trivially obvious look ingenious.

I don’t have the slides from the talk, but video of a similar talk is available here. He spent quite a bit of his talk trying to debunk the Occupy Movement’s slogan that inequality has been increasing. The 90:10 ratio bounced along near 3 for a while, then rose to 4 during the 1980s (the Thatcher years… who knew?!), and hasn’t moved much since. Case closed. Oh, but wait, what about other measures of inequality, you may ask. And since you might ask, he had to set up some straw men to knock down. He showed the same pattern for five other measures of inequality. Case really closed.

Except that these five were all measuring the same thing, more or less. The argument people like Piketty have been making is not that the 90th percentile has been doing so much better than the 10th percentile, but that increases in wealth have been concentrated in ever smaller fractions of the population. None of the measures he looked was designed capture that process. The Gini coefficient, which looks like it measures the whole distribution, because it is a population average is actually extremely insensitive to extreme concentration at the high end. Suppose the top 1% has 20% of the income. Changes of distribution within the top 1% cannot shift the Gini coefficient by more than about 3% of its current value. He also showed the 95:5 ratio, and low-and-behold, that kept rising through the 90s, then stopped. All consistent with the main critique of rising income inequality.

Since he’s obviously not stupid, and obviously understands economics much better than I do, it’s hard to avoid thinking that this was all smoke and mirrors, intended to lull people to sleep about rising inequality, under the cover of technocratic expertise. It’s a well-known trick: Ignore the strongest criticism of your point of view, and give lots of details about weak arguments. Mathematical details are best. “Just do the math” is a nice slogan. Sometimes simple (or complex) calculations can really shed light on a problem that looks to be inextricably bound up with political interests and ideologies. But sometimes not. And sometimes you just have to accept that a political economic argument needs to be melded with statistical reasoning, and you have to be open about the entirety of the argument. Continue reading “Why people hate statisticians”

Small samples

New York Republican Representative Lee Zeldin was asked by reporter Tara Golshan how he felt about the fact that polls seem to show that a large majority of Americans — and even of Republican voters — oppose the Republican plan to reduce corporate tax rates. His response:

What I have come in contact with would reflect different numbers. So it would be interesting to see an accurate poll of 100 million Americans. But sometimes the polls get done of 1,000 [people].

Yes, that does seem suspicious, only asking 1,000 people… The 100 million people he has come in contact with are probably more typical.

Parliamentary mortality

An article in the New Statesman raised the question of whether the Conservatives could lose their hold on power via by-elections over the next few years only to dismiss the possibility because by-elections simply don’t happen frequently enough. The reason? Reduced mortality rates. Quite sensible, but then this strange claim was made:

In 1992-7, the last time that the Conservatives had seven by-elections in a parliament, life expectancy was 15 years lower than it is today.

Ummm… If life expectancy had increased by 15 years over the last 20 years, we’d be getting close to achieving mortality escape velocity. In fact, the increase has been about 5 years for men and 4 years for women.

But that raised for me the somewhat morbid question: How many MPs would be expected to die in the next 5 years? Approximate age distribution of MPs is available here. It’s for the last parliament, but I’ll assume it remains pretty similar. It’s interesting that Labour had twice as large a proportion (25% vs 12%) in the over-60 category. In addition, I’ll make the following assumptions:

  1. Within coarse age categories the distribution is the same between parties. (This is required to deal with the fact that the numbers by party are only divided into three age categories.)
  2. Since I don’t have detailed mortality data by class or occupation, I’ll simply treat them as being 5 years younger than their calendar age, since that’s the difference in median age at death between men in managerial occupations and average occupations.
  3. I assume women to have the same age distribution as men.
  4. I’m using 2013 mortality rates from the Human Mortality Database.

My calculations say that the expected number of deaths over the next 5 years is about 6.4 Conservatives and 6.5 Labour. So we can estimate that the probability of at least 7 by-elections due to deceased Tory MPs is just a shade under 50%.

Learning to count

The US espionage services promised last year to reveal roughly how many Americans were illegally spied upon through “accidents” in the warrantless surveillance law restricted to communications by foreigners overseas.

Last month the promise was retracted.

“The NSA has made Herculean, extensive efforts to devise a counting strategy that would be accurate,” Dan Coats, a career Republican politician appointed by Republican President Donald Trump as the top U.S. intelligence official, testified to a Senate panel on Wednesday.

Coats said “it remains infeasible to generate an exact, accurate, meaningful, and responsive methodology that can count how often a U.S. person’s communications may be collected” under the law known as Section 702 of the Foreign Intelligence Surveillance Act.

So we’re supposed to believe that the NSA is capable of making brilliant use of the full depth of private communications to map out threats to US national security… but isn’t capable of counting them. Presumably these “Herculean” efforts involve a strong helping of Cretan Bull—-.

I am reminded of this passage in Der Mann ohne Eigenschaften [The Man without Qualities]:

Es gibt also in Wirklichkeit zwei Geistesverfassungen, die einander nicht nur bekämpfen, sondern die gewöhnlich, was schlimmer ist, nebeneinander bestehen, ohne ein Wort zu wechseln, außer daß sie sich gegenseitig versichern, sie seien beide wünschenswert, jede auf ihrem Platz. Die eine begnügt sich nicht damit, genau zu sein, und hält sich an die Tatsachen; die andere begnügt sich nicht damit, sondern schaut immer auf das Ganze und leitet ihre Erkenntnisse von sogenannten ewigen und großen Wahrheiten her. Die eine gewinnt dabei an Erfolg, und die andere an Umfang und Würde. Es ist klar, daß ein Pessimist auch sagen könnte, die Ergebnisse der einen seien nichts wert und die der anderen nicht wahr. Denn was fängt man am Jüngsten Tag, wenn die menschlichen Werke gewogen werden, mit drei Abhandlungen über die Ameisensäure an, und wenn es ihrer dreißig wären?! Andererseits, was weiß man vom Jüngsten Tag, wenn man nicht einmal weiß, was alles bis dahin aus der Ameisensäure werden kann?!

Thus there are in fact two distinct mental types, which not only battle each other, but which, even worse, generally coexist side by side without ever exchanging a word, other than that they assure each other that they are each desirable in their own place. The one contents itself with being exact and keeping to the facts; the other is not content with that, always looking for the big picture, and derives its knowledge from so-called great eternal truths. The one gains success thereby, the other gains scope and value. Of course, a pessimist could always say, the results of the one have no value, while those of the other have no truth. After all, when the Last Judgment comes, what are we going to do with three treatises on formic acid, and so what if we even had thirty of them?! On the other hand, what could we possibly know about the Last Judgment, if we don’t even know what could happen by then with formic acid?!

The NSA is focusing on the big picture…

Worst mathematics metaphor ever?

I’ve come to accept “growing exponentially” — though I once had to bite my tongue at a cancer researcher claiming that “exponential growth” of cancer rates began at age 50, because earlier the rates were just generally low — and didn’t say anything when someone recently referred to having “lots of circles to square”. But here’s a really new bad mathematics metaphor: the Guardian editorialises that after Brexit

Europe will be less than the sum of its remaining parts.

“More than the sum of its parts” or “less than” is something you say when you’re adding things together, and pointing out either that you don’t actually get as much extra as you’d think or, on the contrary, that you get more. That you get less when you take something away really doesn’t need much explanation and, in any case, it’s not about the sum of the parts. Whether the remains of Europe are more or less than the sum of the other parts seems kind of irrelevant to whatever argument is being suggested.

The unexpected autocracy

One of my favourite logic paradoxes (does everyone have favourite logic paradoxes?) goes by the name of The Unexpected Hanging. There are numerous versions, but a standard story is: A man has been condemned to death for some crime. The judge tells him, “Today is Monday. You are to be hanged at noon some day in the next week, but you will not know until the morning of the day of the hanging which day it will be.” The man then reasons, it can’t be Sunday, because if I haven’t been hanged by Saturday noon, I’ll know it must be Sunday, which would contradict the judge’s order. Since it can’t be Sunday, if we get to Friday afternoon, I’ll know it must be Saturday. Again a contradiction. So it can’t be Saturday. Working backward in this way, he is confident that he cannot be hanged at all. But then Thursday dawns, and he is hanged, and he never anticipated it.

I was thinking about this, particularly in the light of this comment by Josh Marshall:

One thing we can say in Donald Trump’s favor, there was no bait and switch. They told us they would do all of this and more.

It’s true, and I’m not surprised. And yet… Trump did say he would ban Muslims. He would build a wall. He would ban abortion. He would revoke the Affordable Care Act. And yet, at the same time, he was saying over and over again, I’m going to be unpredictable. I won’t say what I’m really going to do. More than that, his whole demeanor suggested that you couldn’t believe the specifics of what he was saying. So, in the end, he does exactly what he said he would do, and it actually is somewhat surprising. Continue reading “The unexpected autocracy”

Being demographic

People have been saying for a long time that the Republican strategy of ethnic nationalism is running out of room, because of increasing proportions of ethnic minorities. I noted during the 2012 election how odd it was that some groups of people were considered to vote “demographically”, while others (white Protestant men) were assumed to vote on the basis of a broad array of concerns. According to the demographic fallacy, minority groups have special interests that are very important to them, but only of peripheral interest to the majority. Too much pandering can piss off the majority, but targeted appeals can motivate the minority, potentially to very high percentages, but there is no way to motivate the majority en bloc. After the 2012 election there were any number of comments of the sort “To win the presidency, Republicans need to make up their deficit among black and hispanic voters. They are losing them at such a level that (with changing povulation composition) a future Republican candidate would need to win the white vote at implausible levels to win a majority.” Now it appears that this argument is exactly wrong, for three reasons:

  1. As Trump correctly intuited, white people are also susceptible to ethnic appeals. And if you can motivate them as an ethnic group, they’re the biggest, baddest one of all. Meanwhile, the Democrats appeal to ethnic minorities was maxed out. The pervasive undercover racism of the Republican party gave Obama a huge edge among hispanics and blacks; naked racism, religious exclusion, and threats of deportation by Trump couldn’t move it any further, but could pull in vast numbers of white voters who share his racist world view and are relieved to hear it expressed openly. Those of us who move in educated circles should have taken more seriously the assertions early on that “Trump says what everyone really thinks”. Obviously, we didn’t know what people were thinking.
  2. Similarly for women. The model of what I called “demographic thinking” in politics is  I’m not the first to notice that women are not actually a minority. The power relations (yay intersectionality!) nonetheless seem to justify seeing the struggle for women’s rights as analogous to the struggle for rights of ethnic minorities.
    Feminists may have gotten suckered by a figure-ground second-sex fallacy with regard to women voters. If you think of males as the default, and women as the “minority”, then an openly misogynist candidate like Trump would seem to turn out the women to vote against him. But most of those women have been having to compromise with and make excuses for Trump-like figures in their lives — in their families — their whole lives. Some will recoil in horror, but most will continue to make excuses. And the women voters lost may be balanced by just as many men gained.
  3. It’s perfectly possible to maintain a semblance of democracy while entrenching the power of a minority to rule over the majority. Many countries have done this. With the single exception of 2004, the Republicans have not won a plurality in a presidential election since 1988. Democrats received a majority of the votes for representatives in 2012 and (probably) 2016. Nonetheless, the Republicans have attained unrestricted control over nearly the entire federal government, and very little stands in the way of further restricting voting rights to maintain their control and civil rights of minorities, expanding the political influence of the wealthy, to maintain their power indefinitely.

The electoral college was designed to leverage the 3/5 compromise to increase the power of southern slave-holding states in presidential election. Now, under very different circumstances, it is still serving this function.

Why were the polls so wrong?

While Tuesdays election result is a global disaster, it is most immediately distressing for three groups: American Latinos, American Muslims, and American pollsters.

First of all, let us dispel with the idea (that I have heard some propound) that they weren’t wrong. Huge numbers of polls done independently in multiple states gave results that were consistently at variance in the same direction with the actual election results. I can see three kinds of explanations:

  1. The pollsters shared a mistaken idea or methodology for correcting their tiny unrepresentative samples for differential turnout.
  2. Subjects lied about their voting intentions.
  3. Subjects changed their minds between the last poll and the election.

3 seems unlikely to account for a lot, as it seems implausible to suppose that many people changed their minds so rapidly. 2 is plausible, but hard to check and difficult  impossible to correct. 1 is a nice technical-sounding explanation, and certainly seems like there must be some truth to it. Except, probably not much. As evidence, I bring the failure of VoteCastr.

Slate magazine teamed up with the big-data firm VoteCastr to trial a system of estimating votes in real time. Ahead of time they did extensive polling to fit an extensive model to predict an individual’s vote (probabilistically) as a function of several publicly-available demographic variables. Then they track records of who actually voted, and update their totals for the number of votes for each candidate accordingly.

Sounds like a perfectly plausible scheme. And it bombed. For instance, their final projection for Florida was 4.9 million (actually, 4,225,249) for Clinton and 4.6 million for Trump, a lead of about 3% for Clinton. The real numbers were 4.5 million and 4.6 million, a lead of 1.3% for Trump. (The difference in the total seems to be mainly due to votes for other candidates, though the total number of Florida votes in VoteCastr is about 100,000 more than in the official tally, which I find suspicious.) They projected a big victory for Clinton in Wisconsin.

The thing is, this removes the uncertainty related to reason 1: They know exactly who came to vote, and they’re matched by age, sex, and party registration. Conclusion: Estimating turnout is not the main problem that undermined this year’s presidential election polls.