It is now generally in vogue to agree that much of the polling in the US general election was ‘wrong’, but Conquest’s David Penn asks what we mean by wrong: the results or our ability to understand them?

Should we blame “the polls”? The media certainly thinks so. Here’s Freddy Gray in The Spectator:

“Can we just agree never to pay any attention to polls from now on?...The big pollsters all assured us that they had learned the lesson of 2016*…Some projections even gave Biden a 97% chance of winning. Everyone who isn’t delusional knew that was absurd.”

This got me thinking: what does “wrong” mean anyway? 

Some pollsters might say, “Well, Freddy, we said Biden’s victory was 97% certain and look what’s happened! “Journalists, on the other hand, would say pollsters got it wrong because his margin of victory is much less than their figures suggested. 

So, I wonder: Are polls fundamentally flawed or is it the interpretation of them that’s out of kilter?

Why believe what people tell you?

If I ask you, “Do you like oranges?”, you could give me a pretty accurate answer. If on the other hand, if I ask “Are you going to eat an orange next Thursday?”, the chances of an accurate answer diminish sharply. You may, of course, eat oranges every day, so it’s highly probable that you will eat one that day. If, like me, you’re an irregular consumer, it’s only possible that you will. 

So why, when we ask people who will they be voting for in an upcoming election, do we believe what they say? My company does commercial market research, not political polling, and, if we test a new baked beans concept, do we believe it when 50% say they will “definitely buy” it? Well, we don’t disbelieve it, but we don’t report to our client that they can expect almost half the population to rush out and buy their product. The point is: (a) figures like this are known to include overclaim; and (b) people’s willingness to buy will be affected by all sorts of extraneous factors – price, availability, product performance, etc. That’s why survey data forms only part of predictive models of future behaviour.

The essential problem with polling is that it assumes that people know what they’re going to do and, moreover, can tell you. It’s strange that pollsters are almost the only part of the market research industry to cling wholeheartedly to that belief, although it’s encouraging that one polling company has fairly recently developed a multiple regression model to predict voting intention from previous behavioural activity.

Polls are estimates not audits

When journalists say (as they often do) that the polls “got it wrong”, what they usually mean is that Biden (or whoever) polled 52%, or 54%, whereas he actually got 50%. In other words, they’re inclined to treat the data as if it’s an audit (with 99.999% accuracy) rather than an estimate with a significant margin of error attached. 

There’s over 300 million people in the US and, miraculously, it is possible to construct a poll based on a few thousand people that tells you with reasonable accuracy what percentage of people think X or Y – with reasonable accuracy. But even the largest polls are subject to margins of error of +/-2 percentage points; so, 52% for Biden could be 54% or 50%. And 48% for Trump could be 46% or 50%. In other words, they could be neck and neck, or 8 points apart.

A 48/52 split produces a range of possibilities each of which have a probability. A 46/54 split produces the same range of possible outcomes but with very different probabilities. Hence it was always possible (despite Biden’s consistent poll lead) that Trump could win but perhaps not very probable, and so it turned out. In the event, it looks like Biden won by c.3 percentage points and four million votes. 

When you’re comparing the voting intentions for two rival contenders, all you can ever be confident of is that A’s figure is higher than B’s. The bigger the gap, the more confident you can be that A will win. But that’s not the same thing as saying, “A will win by a landslide”. Unless the gap is more than 5 percentage points, we can usually only be c.95% confident. In other words, there’s usually around a 1 in 20 (or 5%) chance that A and B are not different. 

Now, 5% is a low probability, but it doesn’t mean it won’t happen. After all, Leicester City won the English Premier League after being given a 1 in 5000 chance by the bookmakers. Low probability events do occur. So, when I hear a journalist wail that the polls gave a failed candidate, say, a 90% chance of victory, I’m tempted to ask: Would you take a ride in a plane that had a 10% chance of crashing?

Polls are only as good as the people they speak to

If you’ve followed the argument so far, you’re probably thinking, “ Yeah, yeah, but Freddie is right, the US polls forecast a big win for Biden, and it’s much quite closer than that”. Fair point, and I think the lesson of this election is that there may be a systematic polling bias, either in favour of the Democrats or against Trump. It looks to me that Trump’s vote is consistently underestimated and that could be due to sample deficiencies. Perhaps a lot of Trump’s voters just don’t show up on online panels, or maybe – as US pollster Frank Luntz suggests – they deliberately boycott polls. It could be that polls are seen as part of the “fake news establishment” which Trump and his fans deride. 

Then there’s also the ‘social proof’ argument – the one that says some Trump voters ‘lie’ about their preference, because supporting him is not socially acceptable. I’ve always been skeptical about ‘shy’ punters, seeing it as a hangover from the days of telephone and face-to-face polling. Why, after all, would people lie online, when there’s no human interlocutor to deceive? 

I think the answer is that people don’t lie, but they can deceive themselves. For example, support in England for the government’s Covid-19 regulations appears to be running at over 60%, outscoring the naysayers by over 2:1. Yet, as I’ve written elsewhere, it doesn’t feel that way. However, saying yes is probably more socially acceptable than saying no – particularly when it’s mainly the ‘awkward squad’ who oppose the restrictions – so maybe there is a bias operating in favour of the status quo.

Pollsters tell the media what want to hear

Pollsters love to get on the media and the media love to get stories from pollsters. There’s at best a symbiotic relationship and at worst a kind of Faustian pact. My view is that pollsters take too much credit when they get it right but too little responsibility when they get it wrong. This is not new. Throughout the 1980s, Robert Worcester (founder of MORI) crowed that the polls in UK were consistently right, up until 1992 when they came a cropper at that year’s General Election. His response? Not our fault guv, the census was out of date. No one really believed him then and I don’t believe it now. 

The problem is that polling is by far the most visible part of the market research industry and frankly I am getting a bit fed up with them pouring a bucket of ordure over the rest of the industry at election times. I know people who hold market research in contempt and that, in large part, is due to opinion polling. So, for once, I (almost) agree with Donald Trump: I do blame the polls, or more specifically the pollsters and their chums in the media who consistently misreport what they see.

*Actually in 2016, most polls forecast a marginal victory for Hilary, which was correct in terms of the popular vote, but not enough to win the Electoral College.