Occasional reflections on Life, the World, and Mathematics

Archive for the ‘Technical’ Category

Worst mathematics metaphor ever?

I’ve come to accept “growing exponentially” — though I once had to bite my tongue at a cancer researcher claiming that “exponential growth” of cancer rates began at age 50, because earlier the rates were just generally low — and didn’t say anything when someone recently referred to having “lots of circles to square”. But here’s a really new bad mathematics metaphor: the Guardian editorialises that after Brexit

Europe will be less than the sum of its remaining parts.

“More than the sum of its parts” or “less than” is something you say when you’re adding things together, and pointing out either that you don’t actually get as much extra as you’d think or, on the contrary, that you get more. That you get less when you take something away really doesn’t need much explanation and, in any case, it’s not about the sum of the parts. Whether the remains of Europe are more or less than the sum of the other parts seems kind of irrelevant to whatever argument is being suggested.

The unexpected autocracy

One of my favourite logic paradoxes (does everyone have favourite logic paradoxes?) goes by the name of The Unexpected Hanging. There are numerous versions, but a standard story is: A man has been condemned to death for some crime. The judge tells him, “Today is Monday. You are to be hanged at noon some day in the next week, but you will not know until the morning of the day of the hanging which day it will be.” The man then reasons, it can’t be Sunday, because if I haven’t been hanged by Saturday noon, I’ll know it must be Sunday, which would contradict the judge’s order. Since it can’t be Sunday, if we get to Friday afternoon, I’ll know it must be Saturday. Again a contradiction. So it can’t be Saturday. Working backward in this way, he is confident that he cannot be hanged at all. But then Thursday dawns, and he is hanged, and he never anticipated it.

I was thinking about this, particularly in the light of this comment by Josh Marshall:

One thing we can say in Donald Trump’s favor, there was no bait and switch. They told us they would do all of this and more.

It’s true, and I’m not surprised. And yet… Trump did say he would ban Muslims. He would build a wall. He would ban abortion. He would revoke the Affordable Care Act. And yet, at the same time, he was saying over and over again, I’m going to be unpredictable. I won’t say what I’m really going to do. More than that, his whole demeanor suggested that you couldn’t believe the specifics of what he was saying. So, in the end, he does exactly what he said he would do, and it actually is somewhat surprising. (more…)

The view from one-in-a-billion

Nautilus just published my new article, a nontechnical introduction to large deviations: The theory of how unlikely rare events are, and which way a rare event chooses to occur, given that it occurs.

Being demographic

People have been saying for a long time that the Republican strategy of ethnic nationalism is running out of room, because of increasing proportions of ethnic minorities. I noted during the 2012 election how odd it was that some groups of people were considered to vote “demographically”, while others (white Protestant men) were assumed to vote on the basis of a broad array of concerns. According to the demographic fallacy, minority groups have special interests that are very important to them, but only of peripheral interest to the majority. Too much pandering can piss off the majority, but targeted appeals can motivate the minority, potentially to very high percentages, but there is no way to motivate the majority en bloc. After the 2012 election there were any number of comments of the sort “To win the presidency, Republicans need to make up their deficit among black and hispanic voters. They are losing them at such a level that (with changing povulation composition) a future Republican candidate would need to win the white vote at implausible levels to win a majority.” Now it appears that this argument is exactly wrong, for three reasons:

  1. As Trump correctly intuited, white people are also susceptible to ethnic appeals. And if you can motivate them as an ethnic group, they’re the biggest, baddest one of all. Meanwhile, the Democrats appeal to ethnic minorities was maxed out. The pervasive undercover racism of the Republican party gave Obama a huge edge among hispanics and blacks; naked racism, religious exclusion, and threats of deportation by Trump couldn’t move it any further, but could pull in vast numbers of white voters who share his racist world view and are relieved to hear it expressed openly. Those of us who move in educated circles should have taken more seriously the assertions early on that “Trump says what everyone really thinks”. Obviously, we didn’t know what people were thinking.
  2. Similarly for women. The model of what I called “demographic thinking” in politics is  I’m not the first to notice that women are not actually a minority. The power relations (yay intersectionality!) nonetheless seem to justify seeing the struggle for women’s rights as analogous to the struggle for rights of ethnic minorities.
    Feminists may have gotten suckered by a figure-ground second-sex fallacy with regard to women voters. If you think of males as the default, and women as the “minority”, then an openly misogynist candidate like Trump would seem to turn out the women to vote against him. But most of those women have been having to compromise with and make excuses for Trump-like figures in their lives — in their families — their whole lives. Some will recoil in horror, but most will continue to make excuses. And the women voters lost may be balanced by just as many men gained.
  3. It’s perfectly possible to maintain a semblance of democracy while entrenching the power of a minority to rule over the majority. Many countries have done this. With the single exception of 2004, the Republicans have not won a plurality in a presidential election since 1988. Democrats received a majority of the votes for representatives in 2012 and (probably) 2016. Nonetheless, the Republicans have attained unrestricted control over nearly the entire federal government, and very little stands in the way of further restricting voting rights to maintain their control and civil rights of minorities, expanding the political influence of the wealthy, to maintain their power indefinitely.

The electoral college was designed to leverage the 3/5 compromise to increase the power of southern slave-holding states in presidential election. Now, under very different circumstances, it is still serving this function.

Why were the polls so wrong?

While Tuesdays election result is a global disaster, it is most immediately distressing for three groups: American Latinos, American Muslims, and American pollsters.

First of all, let us dispel with the idea (that I have heard some propound) that they weren’t wrong. Huge numbers of polls done independently in multiple states gave results that were consistently at variance in the same direction with the actual election results. I can see three kinds of explanations:

  1. The pollsters shared a mistaken idea or methodology for correcting their tiny unrepresentative samples for differential turnout.
  2. Subjects lied about their voting intentions.
  3. Subjects changed their minds between the last poll and the election.

3 seems unlikely to account for a lot, as it seems implausible to suppose that many people changed their minds so rapidly. 2 is plausible, but hard to check and difficult  impossible to correct. 1 is a nice technical-sounding explanation, and certainly seems like there must be some truth to it. Except, probably not much. As evidence, I bring the failure of VoteCastr.

Slate magazine teamed up with the big-data firm VoteCastr to trial a system of estimating votes in real time. Ahead of time they did extensive polling to fit an extensive model to predict an individual’s vote (probabilistically) as a function of several publicly-available demographic variables. Then they track records of who actually voted, and update their totals for the number of votes for each candidate accordingly.

Sounds like a perfectly plausible scheme. And it bombed. For instance, their final projection for Florida was 4.9 million (actually, 4,225,249) for Clinton and 4.6 million for Trump, a lead of about 3% for Clinton. The real numbers were 4.5 million and 4.6 million, a lead of 1.3% for Trump. (The difference in the total seems to be mainly due to votes for other candidates, though the total number of Florida votes in VoteCastr is about 100,000 more than in the official tally, which I find suspicious.) They projected a big victory for Clinton in Wisconsin.

The thing is, this removes the uncertainty related to reason 1: They know exactly who came to vote, and they’re matched by age, sex, and party registration. Conclusion: Estimating turnout is not the main problem that undermined this year’s presidential election polls.

Rehabilitating the single-factor models

Lots of people — myself included — have mocked the penchant of a certain kind of political scientist who like to say that all the superficial busyness of election campaigns is just a distraction, it matters not at all, nor do the candidates really. Presidential elections are decided by the “fundamentals” — usually one or two economic variables. Except that the models work much better for the past than they do for the present or future, and so end up with lots of corrections: So much for an ongoing war, so much for incumbency, or for a party having been in office too long, and so on. They seem kind of ridiculous. Obviously people care who the candidates were. And, of course, these experts agreed that those things weren’t irrelevant, they just tended to cancel each other out, because both major parties choose reasonably competent candidates who run competent campaigns.

And last year they said the fundamentals mildly favoured the Republican to win a modest victory. But the Republicans chose a ridiculous candidate who ran a flagrantly incompetent campaign. So of course this couldn’t be a test of the “fundamentals” theory. But after all that, the Republican won a modest victory. Kind of makes you think…

Making sense of the predictions

I absolutely agree that Sam Wang and the Princeton Election Consortium have a good argument for there being a 99+% chance of Trump winning. Unfortunately, I think there’s only about a 50% chance of his argument being right. It could also be that Nate Silver is right, that there is a 65% chance. Putting that all together, I come down right about where the NY Times is, at about an 85% chance of escaping apocalypse.

I’ve written a bit more about how I think about the likelihoods here. But a fundamental problem with the PEC estimate is that it clearly puts very little weight on the possibility of model failure. A fundamental problem with the 538 estimates is that they are very clearly not martingales. That is, they are not consistent predictions of the future based on all available information. One way of saying this is to note that a few weeks ago Clinton had close to a 90% estimated victory probability. Now it’s 65%. That seems like a modest change, but if the first estimate was correct, the current estimate reflects an event that had less than a 1 in 3 chance of happening. So we’re more than halfway there. But does anyone really think that the events of the past month have been that unlikely?

Optimal fakes

I put together some ideas I’ve had about the self-limiting nature of lies and fakery into an article for the wonderful online popular science magazine Nautilus.

Some other articles I’ve published in Nautilus on prenatal sex ratio and compression of morbidity (people staying healthier at advanced ages).


Ars Technica reports on testimony by Mediacom, a large US cable company, explaining why they should not be required to stop capping data usage:

People thus shouldn’t complain when Internet providers impose data caps and charge more when customers go over them, he wrote. “Even though virtually every other industry prices its products and services in the same way, some people think that ISPs should be the exception and run their businesses like an all-you-can-eat buffet.”

“Virtually every other industry”… Yes, it’s pretty hard to think of any industry that offers all-you-can-eat buffets. Who could possibly afford to offer all-you-can-eat? It’s a fantasy.

Statistics and causal truth: Police edition

As usual, Andrew Sullivan — who has now returned temporarily to blogging, attracted like a moth to the Trump conflagration — manages to take a common, superficially convincing argument, and express it with moral fervour and personal conviction that makes the tenuous logic really conspicuous. In this case, it’s the argument based on the much-discussed study by Roland G. Fryer, Jr. of the rate of various violent outcomes of police stops, finding that black people are more likely than white to be physically abused by police, but not more likely to be shot.

(Here’s an excellent NY Times report, and  the original study.)

…the Black Lives Matter activists, whose core and central argument is that black men are disproportionately killed by cops. The best data shows this is false…  I find [the study] conclusive. Feelings do not, er, trump data in a deliberative democracy. A reader writes:

I understand that there has been the recent study suggesting that given an interaction with a police officer occurs, then the police officer is no more likely to use a gun with a black person than with a white person. However, given that many black men have a much higher rate of interaction with police (such as, anecdotally, Philando Castile, with 52 traffic stops), then is it not fair to say that black men are disproportionately killed by cops?
The point is that there is no evidence of individual racism in these police encounters, despite the impression from many chilling phone videos. The structural bias still exists as a whole, as I said, but the narrative about cops being more likely to kill a black member of the public when encountering him is false.

I have no criticism to make of the study — I have not analysed it in any depth, but it seems credibly and even impressively done — even if I find the premise absurd, that a single study of such a complex phenomenon could be “conclusive”. But they do not “trump” the data that black people make up 13% of the US population, but 31% of those killed during an arrest, and 42% of those killed during an arrest when unarmed. The point is, what these facts (and many others, including the others) mean jointly depends on what we think is the reason for black people being so much more likely to be arrested.


Tag Cloud