The model didn’t fail us, we failed the model

THE ALGORITHM. It’s all anyone can talk about, when they’re talking about universities these days. Illustrative of the unique ability of the current UK government to take a challenging societal problem in hand, and transform it into a flaming chaos that simultaneously exacerbates divisions and satisfies no one.

In this case, it’s about the assignment of marks in A-levels (18 year-olds) released last week, and GCSEs (16-year-olds) still ahead this week. Scotland had its own small version of the fiasco that played out earlier in the week for their own Scottish Higher exams, but the UK government, responsible for English A-levels, managed not only not to learn from the Scottish situation and change course early, it managed to parlay the political challenge into a systemic disaster for higher education that will now roll on for at least the next year or two.

Like any great governing disaster, this one has been years in the making. Pupils doing A levels used to have intermediate exams — AS levels — after the first year of their two-year course, as well as significant amounts of coursework that counted for a substantial portion of their final marks. AS levels were progressively eliminated and coursework reduced over the past decade in England (but not Wales), as the Conservatives seem to have believed that todays pupils were being inappropriately coddled by having too little stress and uncontrollable randomness in their lives, leaving several weeks of exams right at the end of their course as the only determinant of the marks that would decide high-stakes competition for university places. Then they cancelled the exams, in a panicked response to the first wave of Covid-19. Leaving them with nothing.

Weirdly, it’s not as though they don’t have frequent exams during their school (and university) time. But these exams are called “mock exams”, and don’t count for anything in the end.

Which brings us to THE ALGORITHM. How do you assign marks to students when you don’t have any exams? Teachers have quite a lot of information, even if it doesn’t formally count for anything in the regular process. (Weirdly, teachers are regularly expected to produce “predicted grades” based on mock exams, coursework, and general impressions, because the official marks arrive too late for university admissions.) But on average they tend to be overly optimistic — or, one might also say, either generous or strategic, since the university admission offer that results from an overpredicted A-level grade is not necessarily withdrawn when the exam result exceeds it, whereas the university place that is lost from an underprediction is almost impossible to make good.

If you were a mindless machine-learning bot trying to optimise the accuracy of prediction of missing marks in an overall minimum-mean-error way, you would take data about each student’s family income, ethnicity, sex, parents’ occupations, and region, all of which are likely to be correlated to exam scores. But that would seem outrageously biased: Why should the young person with wealthy parents get higher A-level marks than the one with poor parents, after they had the same mock exam grades? The machine-learning answer is, because that’s what’s happened with real grades in the past. The wealthy family is likely to provide more support, maybe tutors, a more stable environment for studying for the exams. The child of the poor family may have been working hard since year 12, but there’s a much higher chance that the family would have had a crisis — maybe a parent losing a job, illness, homelessness — that would have distracted from exam preparation and led to underperformance at the exam. And since that might have happened in reality, that needs to be reflected in our optimal prediction algorithm.

But that looks bad, so the Ofqual boffins used past school performance as a proxy. Effectively, they said that each school gets the same marks this year that they got last year. Teacher evaluations were used to rank the students in each subject, to decide which students get the school’s quota of A*’s, etc. If you made the bad life choice to go to a low-performing school where no one in living memory has scored better than B in chemistry, then B is the ceiling for your marks, no matter what scores you may personally have been achieving on your mockeries.

Averaged over the whole population of English students your misfortune is just a small blemish on an overall excellent prediction.

It’s a good illustration of the problems of ethical machine learning. People say, if you don’t want your algorithm to be biased based on gender, don’t include gender information in the dataset. But if you instead include height information, say, the algorithm will learn all the gender bias in the training set and assign it to the height variable.

Just to rub salt in the wounds, there was an extra fillip for students in small — heavily private — schools: Since average performance fluctuates more in small groups, courses with 15 or fewer students had their (generally higher) teacher predictions more heavily weighted in their final marks, and those with 5 or fewer received their teacher predictions unfiltered.

Now, this way of using past school performance seems… surprising, to those of us who have been involved in UK university admissions in the past, given the extent of government and public outrage every year when the elite universities once again draw their intake from a very small sliver of UK secondary schools, predominantly private schools. You might think that this outrage reflects a belief that the differences in average exam performance, that drive most of the differential in university admissions, are unfair, that they do not accurately represent student ability, performance, and potential. If you believed that, you might propose a very different way of using school performance to assign marks, namely: Every school gets the same proportion of A*, A, B, etc., to be allocated within each subject according to the teacher rankings. I’m not advocating this method, but it is no more extreme, in its own way, than the application of past school performance that was actually implemented.

To the extent that A-level marks are primarily a tool for sorting graduates for university admissions, this would function somewhat similarly to the practice of some US states, of guaranteeing admission to their state universities to a certain percentile of every high school in the state. This leverages housing and school segregation to benefit equality, as opposed to the opposite.

The fact that my algorithm seems obviously unfair to individuals, while the other algorithm was seen as not only credible but actually self-evident, reflects nothing but naked ideology about the nature of class.

Education minister (a position whose relationship to that of education secretary confuses me) Nick Gibb responded to the fiasco thus:

So the model itself was fair, it was very popular, it was widely consulted upon. The problem arose in the way in which the three phases of the application of that model – the historic data of the school, the prior attainment of the cohort of pupils at the school, and then the national standard correction – it’s that element of the application of the model that I think there is a concern.”

The minister went on: “The application of the model is a regulatory approach and it’s the development of that that emerged on the Thursday when the algorithm was published. And at that stage it became clear that there were some results that were being published on Thursday and Friday that were just not right and they were not what the model had intended.”

The poor misunderstood beast. It meant well…

Count no statue happy…

Count no man happy till he dies.

Sophocles, Oedipus the King (trans. Robert Fagles)

Must no one at all, then, be called happy while he lives; must we, as Solon says, see the end? Even if we are to lay down this doctrine, is it also the case that a man is happy when he is dead? […] for both evil and good are thought to exist for a dead man, as much as for one who is alive but not aware of them; e.g. honours and dishonours and the good or bad fortunes of children and in general of descendants.

Aristotle, Nichomachean Ethics, Book 1 (trans. W D Ross)

In all of the discussion of racist statues one fundamental point is rarely mentioned: Above all, public statues represent the unwillingness of “great men” to simply go away. Those who bestrode their narrow world like a Colossus are loath to let death remove them from the scene, so like the stuffed dodo in a diorama they have their effigies propped up in the public square.

While they lived they received the adulation of the crowds, and the opprobrium of their opponents. If the great one’s supporters need a public icon as a focus for their devotions, the icon will have to continue to participate in the hurly-burly of public life, including the scrutiny of their lives and deeds brought on by shifting ethical standards. If Winston Churchill were alive today he would rightly have paint and rotten tomatoes flung at him by those appalled at his racist ideas and actions. Reasonable can believe that his near-genocidal actions in Bengal, among others places inhabited by darker-skinned people, are more significant than a few well-crafted speeches that bucked up the spirits of the Island Race. Reasonable people did think so during his life. The place where one is beyond praise or blame is called the grave, and no one is suggesting disinterring WC’s bones — though an earlier generation of Tories did exactly that with Oliver Cromwell, after the tide of history turned against him.

His supporters are welcome to hide his statues away in private shrines, or public museums. If you put them up in public you have to accept that people are going to continue to engage with them. Sometimes angrily. Sometimes disorderly.

Plagues and statues

I’ve been reading Camus’ La Peste, hoping to obtain some insight into one of the great crises of the present, and finding him commenting on a completely different one. At the height of the epidemic of the novel, the narrator comments on the aspect of the silent, immobilised city, and expresses resentment toward the statues that are permanently in that condition.

La grande cité silencieuse n’était plus alors qu’un assemblage de cubes massifs et inertes, entre lesquels les effigies taciturnes de bienfaiteurs oubliés ou d’anciens grands hommes étouffés à jamais dans le bronze s’essayaient seules, avec leurs faux visages de pierre ou de fer, à évoquer une image dégradée de ce qui avait été l’homme. Ces idoles médiocres trônaient sous un ciel épais, dans les carrefours sans vie, brutes insensibles qui figuraient assez bien le règne immobile où nous étions entrés ou du moins son ordre ultime, celui d’une nécropole où la peste, la pierre et la nuit auraient fait taire enfin toute voix.

The huge, silent city had become nothing more than a collection of solid, inert cubes, where the taciturn effigies of forgotten benefactors or ancient great men were suffocated forever in bronze, evoking a solitary, degraded image of what man had once been. These mediocre idols, enthroned under a thick sky, in the lifeless crossroads, unfeeling beasts that symbolised well the immobilised realm we had entered, or at least its ultimate order, that of a necropolis where plague, stone, and night would have finally silences any voice.

I’ve commented before on how odd it is that, just because some of our ancestors chose to cast their images in heavy bronze or marble and plonk them down at significant sites in our cities, that we should feel obliged to keep them there. But I assumed that the current attacks on statues of racists was unrelated to the pandemic situation, mere coincidence of crises, except perhaps that the lockdown left people with lots of pent-up energy.

But maybe there’s something about coping with an epidemic that inspires iconoclasm?

Adrift on the Covid Sea

Political leaders in many countries — but particularly in the US and UK — are in thrall above all to the myth of progress. Catastrophes may happen, but then they get better. And to superficial characters like Johnson and Trump, the improvements seem automatic. It’s like a law of nature.

So, we find ourselves having temporarily stemmed the flood of Covid infections, with governments laying out fantastic plans for “reopening”. Even though nothing significant has changed. The only thing that could make this work — absent a vaccine — would be an efficient contact tracing system or a highly effective treatment for the disease. None of which we have. But we still have a timeline for opening up pubs and cinemas (though less important facilities like schools are still closed, at least for many year groups).

It’s like we had been adrift for days in a lifeboat on the open ocean, carefully conserving our supplies. And there’s still no rescue in sight, but Captain Johnson announces that since we’re all hungry from limiting our food rations, and the situation has now stabilised, we will now be transitioning toward full rations.

Rivers of statues

When protestors attack perfectly innocent statues of philanthropists who really demonstrated their love of humanity in the high prices they were willing to buy and sell humans for, and dump them in rivers, people ask, Where will it End? Have these activists considered the environmental consequences — both on water quality and on riparian habitats — of dumping every monument to a notorious British racist in the nearest river? Not to mention the potentially catastrophic flooding.

It’s a relief then to discover that at the same time other statues are being pulled out of rivers. In this case, a monument to unwed mothers that 14th-century cancel culture clearly found offensive. We may hope that the level of political statuary in our cities and in our rivers may reach a more or less stable equilibrium.

Rhodes in Context

This seems like an appropriate time to repost this comment from 2017. It seems very peculiar to me that one politician puts up a poster and the next one tears it down, buildings are torn down when they no longer meet the commercial or aesthetic needs of the current generation, but once a person has had himself poured in bronze or carved in stone.

Of course, it is something requiring debate and consideration — for statues as well as for buildings — and the massivity of these constructions is designed to thwart an overly hasty disposal. Surely no one can say that Rhodes has been hastily disposed of, or with insufficient consideration. At this point if Oriel College decides to retain its statue of Rhodes, it is taking an active decision that the complete record of Cecil Rhodes is such that that college wishes to commend and publicly honour.

Here is my comment from August 2017:

This story happened to a friend of a friend — FOF in urban legend technical parlance — when I was a student at Yale. Said FOF had applied for a Rhodes scholarship, and was invited for an interview. Reading the FOF’s application letter stating that he sought to “further the legacy of Cecil Rhodes”, one interviewer asked, “When you refer to the legacy of Cecil Rhodes, do you mean in particular his legacy as a white supremacist or as a pedophile?”

I’m not sure if it’s credible that a representative of the Rhodes Trust could speak so disparagingly of its founder — this may be an example of British establishment values refracted through the prism of 1980s American student sentiment — but the principle is solid: Many who advocate leaving monuments to dubious figures of the past in situ — whether Cecil Rhodes or Robert E. Lee — complain  suggest, instead of “rewriting history” that this statuary needs to be seen “in context”. But they rarely concern themselves with providing the full context.

Now that Charlottesville has deposed its racist monument and Oriel College has kept its own, I wondered if the Oxford City Council might propose a solution amenable to all. Accepting the right of Oriel and its not-at-all-racist historically-minded alumni who refused to donate to a Rhodes-free institution, there is still plenty of space in front of the facade for more context. As it stands, the college places Rhodes in the context of two 20th-century kings and four 15th-16th-century college provosts and bishops. The city (or enterprising protestors) could contribute more context by placing an exhibition out front of famous British racists — for example, Houston Stewart Chamberlain and Enoch Powell — with the Rhodes statue in the centre.

The conspiracy conspiracy

I was just listening to physicist Sean Carroll’s podcast interview of epistemologist Quassim Cassam. The interview mainly concerned Cassam’s book Vices of the Mind: From the Intellectual to the Political, but at the end it touched on his new book Conspiracy Theories. He remarked on the widespread belief that conspiracy theories are becoming increasingly widespread, and politically influential, and then made this Chomskyan comment

leaving aside the question of whether conspiracy theories are more or less prevalent, it’s true that conspiracy theories are popular and influential across the world and particularly in the US…. I’m actually much more interested in the idea that conspiracy theories or at least the sort of big ticket conspiracy theories are really forms of political propaganda, that what they really do is to advance a political agenda and that’s really their role.

So, you may think conspiracy theories are just arising naturally, but actually there are dark shadowy figures manipulating belief in conspiracies for their nefarious ends.

One small point: Carroll started the interview by remarking “I know a little bit about epistemology, not that much”. Which struck me as a deeply ironic remark, to which I yearned in vain for Cassam to reply “You may know more about it than you think you do.”

Capitalist omelettes

Communists are routinely stigmatised by association with the adage “You can’t make an omelette without breaking a few eggs”, which is taken to summarise their willingness to excuse present cruelty in the name of never-realised lofty future goals.. In the 1930s it was sufficiently widespread that the hero of Sinclair Lewis’s It Can’t Happen Here exclaimed

If I ever hear that ‘can’t make an omelet’ phrase again, I’ll start doing a little murder myself! It’s used to justify every atrocity under every despotism, Fascist or Nazi, or Communist or American labor war. Omelet! Eggs! By God, sir, men’s souls and blood are not eggshells for tyrants to break!

Sick of hearing this justification on a tour of the Soviet Union in the early days of Stalin’s rule, Panait Istrati famously retorted, “All right, I can see the broken eggs. Where’s this omelet of yours?”

Anyway, these days the true radical utopians are the capitalists, and so we have seen the right wing in the US — and elsewhere — following Donald Trump in obsessing over stock market declines in a pandemic. “Don’t let the cure be worse than the disease.” The deadly cure being a reduction in economic activity, and the disease being… an actual often fatal disease. The epitome of this tendency was the Lieutenant Governor of Texas proclaiming that the older generation — in which he includes himself, to be fair — needs to be ready to die in the current pandemic to avert destruction from the American economy: “keeping the America that all America loves for your children and grandchildren”. In other words, we need to break some elderly (and young immune-compromised) so our grandchildren can have the omelettes.

Except they’ll still have to eat their omelettes indoors, because the same people insist on boiling the planet. More eggs, more omelettes.

The bourgeoisie will not only sell the rope for their own hanging. On the scaffold they will try to underbid the hangman to take on the hanging themselves, and sell off part of the rope as really more than is strictly needed to carry out the task.

Capitalist utopianism, it might be mentioned, was beautifully summarised by Joe Hill in his song The Preacher and the Slave:

You will eat by and by
In that glorious land above the sky
Work and pray, live on hay
You’ll get pie in the sky when you die.

Covid-19 and climate change

I’m wondering whether the pandemic disaster might in some way mitigate our climate-change disaster. I don’t mean in the trivial short-term sense that the collapse of travel and general economic activity reduces CO2 output. I mean that the experience of the pandemic undermines the intellectual foundations of climate-change denialism. Again, not in a trivial, debating-point sort of way, but a visceral “I refute it thus“.

Climate-change denial rests, it seems to me, on two intellectual — or perhaps it might be better to call them “mental” — pillars. I refer here not to the occasionally valid but always irrelevant sand-in-the-eyes technical quibbles that are used to convey the impression of scientific disagreement. I mean the actual intellectual motivations for the position that drives the search for these quibbles, to the extent that the motivation is not simply pelf or partisanship. These are

  1. The world is too big for humans to change meaningfully. Often they say it is “arrogant” to imagine that insignificant humans could do something as grand as to change the Earth’s climate.
  2. Action against climate change is woolly feel-good sort of stuff. Maybe it would be a good thing in principle, but hard-head thinkers care about people’s jobs and the here-and-now.

In a sense these are opposites: The first says human activity is trivial compared to the whole planet. The second says human activity is autonomous, and far more significant than the whole planet.

The experience of the pandemic is likely, it seems to me, to make people much less receptive to these arguments. Seeing how small the planet is, that a virus originating in one market in Wuhan can infiltrate the whole world within a few months is liable to leave people feeling that the Earth is quite a fragile thing.

And then, the economic cost of this pandemic is likely to be far higher than even the upper end of estimates of the cost of achieving carbon neutrality. Faced with the pandemic, we are thrown back on the material reality of the economy: Not jobs but work, not production but material goods such as food and shelter. We pay the cost because the alternative is clearly more expensive, in lives and social disruption. And this time, people have been willing, mostly, to pay the cost in advance, believing that the disaster would be vastly greater if we waited. The timeline for the climate-change apocalypse is much longer, but it is not implausible to suppose that the same frame of mind might then allow people to see that global catastrophe is a real thing, and worth making some effort to avert.

(Of course, the opposite might be true. People might say, the apocalypse-averting cupboard is now bare. Come back to us in a decade.)

The unexpected epidemic: A political paradox

An epidemiologist says, “A new pandemic will definitely sweep the world some time this century. But you won’t know until the day it starts when it will be. So you’d better start preparing now.”

The president is downcast. He doesn’t like preparing, but he also doesn’t like when the stock-market falls and people on TV blame him for millions of deaths and blah blah blah. What can he do?

His son-in-law comes to him and says, “I read a book on this. This prediction of an unexpected epidemic can’t happen. Imagine it’s 2099 and there hasn’t been a pandemic yet. Then people would know it has to happen in 2099. So it has to happen earlier. But now, suppose we get to 2098 without a pandemic. We know it can’t happen in 2099, so we would know for sure it must be 2098, which would contradict what the so-called expert told us.” And so, step by step, he shows that the unexpected pandemic can never happen.

You know the rest: The president disbands the National Security Council pandemic preparedness team and writes a celebratory tweet. And then in 2020 a pandemic arrives, and the president announces that “this is something that you can never really think is going to happen.”

(For the original version see Quine’s “On a so-called paradox“. For an account of some of the many times experts warned that a pandemic was coming and would be disastrous, see here.)