Occasional reflections on Life, the World, and Mathematics

Archive for the ‘Technical’ Category

Parliamentary mortality

An article in the New Statesman raised the question of whether the Conservatives could lose their hold on power via by-elections over the next few years only to dismiss the possibility because by-elections simply don’t happen frequently enough. The reason? Reduced mortality rates. Quite sensible, but then this strange claim was made:

In 1992-7, the last time that the Conservatives had seven by-elections in a parliament, life expectancy was 15 years lower than it is today.

Ummm… If life expectancy had increased by 15 years over the last 20 years, we’d be getting close to achieving mortality escape velocity. In fact, the increase has been about 5 years for men and 4 years for women.

But that raised for me the somewhat morbid question: How many MPs would be expected to die in the next 5 years? Approximate age distribution of MPs is available here. It’s for the last parliament, but I’ll assume it remains pretty similar. It’s interesting that Labour had twice as large a proportion (25% vs 12%) in the over-60 category. In addition, I’ll make the following assumptions:

  1. Within coarse age categories the distribution is the same between parties. (This is required to deal with the fact that the numbers by party are only divided into three age categories.)
  2. Since I don’t have detailed mortality data by class or occupation, I’ll simply treat them as being 5 years younger than their calendar age, since that’s the difference in median age at death between men in managerial occupations and average occupations.
  3. I assume women to have the same age distribution as men.
  4. I’m using 2013 mortality rates from the Human Mortality Database.

My calculations say that the expected number of deaths over the next 5 years is about 6.4 Conservatives and 6.5 Labour. So we can estimate that the probability of at least 7 by-elections due to deceased Tory MPs is just a shade under 50%.

Learning to count

The US espionage services promised last year to reveal roughly how many Americans were illegally spied upon through “accidents” in the warrantless surveillance law restricted to communications by foreigners overseas.

Last month the promise was retracted.

“The NSA has made Herculean, extensive efforts to devise a counting strategy that would be accurate,” Dan Coats, a career Republican politician appointed by Republican President Donald Trump as the top U.S. intelligence official, testified to a Senate panel on Wednesday.

Coats said “it remains infeasible to generate an exact, accurate, meaningful, and responsive methodology that can count how often a U.S. person’s communications may be collected” under the law known as Section 702 of the Foreign Intelligence Surveillance Act.

So we’re supposed to believe that the NSA is capable of making brilliant use of the full depth of private communications to map out threats to US national security… but isn’t capable of counting them. Presumably these “Herculean” efforts involve a strong helping of Cretan Bull—-.

I am reminded of this passage in Der Mann ohne Eigenschaften [The Man without Qualities]:

Es gibt also in Wirklichkeit zwei Geistesverfassungen, die einander nicht nur bekämpfen, sondern die gewöhnlich, was schlimmer ist, nebeneinander bestehen, ohne ein Wort zu wechseln, außer daß sie sich gegenseitig versichern, sie seien beide wünschenswert, jede auf ihrem Platz. Die eine begnügt sich nicht damit, genau zu sein, und hält sich an die Tatsachen; die andere begnügt sich nicht damit, sondern schaut immer auf das Ganze und leitet ihre Erkenntnisse von sogenannten ewigen und großen Wahrheiten her. Die eine gewinnt dabei an Erfolg, und die andere an Umfang und Würde. Es ist klar, daß ein Pessimist auch sagen könnte, die Ergebnisse der einen seien nichts wert und die der anderen nicht wahr. Denn was fängt man am Jüngsten Tag, wenn die menschlichen Werke gewogen werden, mit drei Abhandlungen über die Ameisensäure an, und wenn es ihrer dreißig wären?! Andererseits, was weiß man vom Jüngsten Tag, wenn man nicht einmal weiß, was alles bis dahin aus der Ameisensäure werden kann?!

Thus there are in fact two distinct mental types, which not only battle each other, but which, even worse, generally coexist side by side without ever exchanging a word, other than that they assure each other that they are each desirable in their own place. The one contents itself with being exact and keeping to the facts; the other is not content with that, always looking for the big picture, and derives its knowledge from so-called great eternal truths. The one gains success thereby, the other gains scope and value. Of course, a pessimist could always say, the results of the one have no value, while those of the other have no truth. After all, when the Last Judgment comes, what are we going to do with three treatises on formic acid, and so what if we even had thirty of them?! On the other hand, what could we possibly know about the Last Judgment, if we don’t even know what could happen by then with formic acid?!

The NSA is focusing on the big picture…

Worst mathematics metaphor ever?

I’ve come to accept “growing exponentially” — though I once had to bite my tongue at a cancer researcher claiming that “exponential growth” of cancer rates began at age 50, because earlier the rates were just generally low — and didn’t say anything when someone recently referred to having “lots of circles to square”. But here’s a really new bad mathematics metaphor: the Guardian editorialises that after Brexit

Europe will be less than the sum of its remaining parts.

“More than the sum of its parts” or “less than” is something you say when you’re adding things together, and pointing out either that you don’t actually get as much extra as you’d think or, on the contrary, that you get more. That you get less when you take something away really doesn’t need much explanation and, in any case, it’s not about the sum of the parts. Whether the remains of Europe are more or less than the sum of the other parts seems kind of irrelevant to whatever argument is being suggested.

The unexpected autocracy

One of my favourite logic paradoxes (does everyone have favourite logic paradoxes?) goes by the name of The Unexpected Hanging. There are numerous versions, but a standard story is: A man has been condemned to death for some crime. The judge tells him, “Today is Monday. You are to be hanged at noon some day in the next week, but you will not know until the morning of the day of the hanging which day it will be.” The man then reasons, it can’t be Sunday, because if I haven’t been hanged by Saturday noon, I’ll know it must be Sunday, which would contradict the judge’s order. Since it can’t be Sunday, if we get to Friday afternoon, I’ll know it must be Saturday. Again a contradiction. So it can’t be Saturday. Working backward in this way, he is confident that he cannot be hanged at all. But then Thursday dawns, and he is hanged, and he never anticipated it.

I was thinking about this, particularly in the light of this comment by Josh Marshall:

One thing we can say in Donald Trump’s favor, there was no bait and switch. They told us they would do all of this and more.

It’s true, and I’m not surprised. And yet… Trump did say he would ban Muslims. He would build a wall. He would ban abortion. He would revoke the Affordable Care Act. And yet, at the same time, he was saying over and over again, I’m going to be unpredictable. I won’t say what I’m really going to do. More than that, his whole demeanor suggested that you couldn’t believe the specifics of what he was saying. So, in the end, he does exactly what he said he would do, and it actually is somewhat surprising. (more…)

The view from one-in-a-billion

Nautilus just published my new article, a nontechnical introduction to large deviations: The theory of how unlikely rare events are, and which way a rare event chooses to occur, given that it occurs.

Being demographic

People have been saying for a long time that the Republican strategy of ethnic nationalism is running out of room, because of increasing proportions of ethnic minorities. I noted during the 2012 election how odd it was that some groups of people were considered to vote “demographically”, while others (white Protestant men) were assumed to vote on the basis of a broad array of concerns. According to the demographic fallacy, minority groups have special interests that are very important to them, but only of peripheral interest to the majority. Too much pandering can piss off the majority, but targeted appeals can motivate the minority, potentially to very high percentages, but there is no way to motivate the majority en bloc. After the 2012 election there were any number of comments of the sort “To win the presidency, Republicans need to make up their deficit among black and hispanic voters. They are losing them at such a level that (with changing povulation composition) a future Republican candidate would need to win the white vote at implausible levels to win a majority.” Now it appears that this argument is exactly wrong, for three reasons:

  1. As Trump correctly intuited, white people are also susceptible to ethnic appeals. And if you can motivate them as an ethnic group, they’re the biggest, baddest one of all. Meanwhile, the Democrats appeal to ethnic minorities was maxed out. The pervasive undercover racism of the Republican party gave Obama a huge edge among hispanics and blacks; naked racism, religious exclusion, and threats of deportation by Trump couldn’t move it any further, but could pull in vast numbers of white voters who share his racist world view and are relieved to hear it expressed openly. Those of us who move in educated circles should have taken more seriously the assertions early on that “Trump says what everyone really thinks”. Obviously, we didn’t know what people were thinking.
  2. Similarly for women. The model of what I called “demographic thinking” in politics is  I’m not the first to notice that women are not actually a minority. The power relations (yay intersectionality!) nonetheless seem to justify seeing the struggle for women’s rights as analogous to the struggle for rights of ethnic minorities.
    Feminists may have gotten suckered by a figure-ground second-sex fallacy with regard to women voters. If you think of males as the default, and women as the “minority”, then an openly misogynist candidate like Trump would seem to turn out the women to vote against him. But most of those women have been having to compromise with and make excuses for Trump-like figures in their lives — in their families — their whole lives. Some will recoil in horror, but most will continue to make excuses. And the women voters lost may be balanced by just as many men gained.
  3. It’s perfectly possible to maintain a semblance of democracy while entrenching the power of a minority to rule over the majority. Many countries have done this. With the single exception of 2004, the Republicans have not won a plurality in a presidential election since 1988. Democrats received a majority of the votes for representatives in 2012 and (probably) 2016. Nonetheless, the Republicans have attained unrestricted control over nearly the entire federal government, and very little stands in the way of further restricting voting rights to maintain their control and civil rights of minorities, expanding the political influence of the wealthy, to maintain their power indefinitely.

The electoral college was designed to leverage the 3/5 compromise to increase the power of southern slave-holding states in presidential election. Now, under very different circumstances, it is still serving this function.

Why were the polls so wrong?

While Tuesdays election result is a global disaster, it is most immediately distressing for three groups: American Latinos, American Muslims, and American pollsters.

First of all, let us dispel with the idea (that I have heard some propound) that they weren’t wrong. Huge numbers of polls done independently in multiple states gave results that were consistently at variance in the same direction with the actual election results. I can see three kinds of explanations:

  1. The pollsters shared a mistaken idea or methodology for correcting their tiny unrepresentative samples for differential turnout.
  2. Subjects lied about their voting intentions.
  3. Subjects changed their minds between the last poll and the election.

3 seems unlikely to account for a lot, as it seems implausible to suppose that many people changed their minds so rapidly. 2 is plausible, but hard to check and difficult  impossible to correct. 1 is a nice technical-sounding explanation, and certainly seems like there must be some truth to it. Except, probably not much. As evidence, I bring the failure of VoteCastr.

Slate magazine teamed up with the big-data firm VoteCastr to trial a system of estimating votes in real time. Ahead of time they did extensive polling to fit an extensive model to predict an individual’s vote (probabilistically) as a function of several publicly-available demographic variables. Then they track records of who actually voted, and update their totals for the number of votes for each candidate accordingly.

Sounds like a perfectly plausible scheme. And it bombed. For instance, their final projection for Florida was 4.9 million (actually, 4,225,249) for Clinton and 4.6 million for Trump, a lead of about 3% for Clinton. The real numbers were 4.5 million and 4.6 million, a lead of 1.3% for Trump. (The difference in the total seems to be mainly due to votes for other candidates, though the total number of Florida votes in VoteCastr is about 100,000 more than in the official tally, which I find suspicious.) They projected a big victory for Clinton in Wisconsin.

The thing is, this removes the uncertainty related to reason 1: They know exactly who came to vote, and they’re matched by age, sex, and party registration. Conclusion: Estimating turnout is not the main problem that undermined this year’s presidential election polls.

Rehabilitating the single-factor models

Lots of people — myself included — have mocked the penchant of a certain kind of political scientist who like to say that all the superficial busyness of election campaigns is just a distraction, it matters not at all, nor do the candidates really. Presidential elections are decided by the “fundamentals” — usually one or two economic variables. Except that the models work much better for the past than they do for the present or future, and so end up with lots of corrections: So much for an ongoing war, so much for incumbency, or for a party having been in office too long, and so on. They seem kind of ridiculous. Obviously people care who the candidates were. And, of course, these experts agreed that those things weren’t irrelevant, they just tended to cancel each other out, because both major parties choose reasonably competent candidates who run competent campaigns.

And last year they said the fundamentals mildly favoured the Republican to win a modest victory. But the Republicans chose a ridiculous candidate who ran a flagrantly incompetent campaign. So of course this couldn’t be a test of the “fundamentals” theory. But after all that, the Republican won a modest victory. Kind of makes you think…

Making sense of the predictions

I absolutely agree that Sam Wang and the Princeton Election Consortium have a good argument for there being a 99+% chance of Clinton winning. Unfortunately, I think there’s only about a 50% chance of his argument being right. It could also be that Nate Silver is right, that there is a 65% chance. Putting that all together, I come down right about where the NY Times is, at about an 85% chance of escaping apocalypse.

I’ve written a bit more about how I think about the likelihoods here. But a fundamental problem with the PEC estimate is that it clearly puts very little weight on the possibility of model failure. A fundamental problem with the 538 estimates is that they are very clearly not martingales. That is, they are not consistent predictions of the future based on all available information. One way of saying this is to note that a few weeks ago Clinton had close to a 90% estimated victory probability. Now it’s 65%. That seems like a modest change, but if the first estimate was correct, the current estimate reflects an event that had less than a 1 in 3 chance of happening. So we’re more than halfway there. But does anyone really think that the events of the past month have been that unlikely?

Optimal fakes

I put together some ideas I’ve had about the self-limiting nature of lies and fakery into an article for the wonderful online popular science magazine Nautilus.

Some other articles I’ve published in Nautilus on prenatal sex ratio and compression of morbidity (people staying healthier at advanced ages).

Tag Cloud