FactCheck Q&A: Should we trust the polls?

The Conservative’s lead over Labour has tumbled in the opinion polls. Experts now believe the election will be much closer than they thought.

But there’s a huge amount of variation between these polls. For instance, on 29 May an ICM survey gave the Conservatives a 12 point lead. But two days later, YouGov said the party was leading by just three points.

Does the age divide affect things?

To understand the polls, we need to look back to the last election in 2015.

Before the election, lots of young people told pollsters they would vote Labour. But on the day, many did not turn out to vote.

Older people, on the other hand – who tend to support the Conservatives more – were more likely to stick to their intention of voting.

Because of this, the pollsters wrongly predicted the outcome of the election, by overestimating the number of Labour votes. (There were other factors at play too, but this was chief among them).

Since then, the pollsters have been trying to work out the best way to get around this problem. How do you factor in the likelihood of turnout?

Each company has taken a slightly different approach but – in one way or another – they’ve all tried to take into account what happened in 2015.

Take ICM, for instance. Back in 2015 they weighted each survey response by simply asking people how likely they were to vote, on a scale of one to ten. If they said five, their vote was given a weighting of 0.5, and so on.

But since this method proved to be unreliable, the models were updated. ICM’s results are now weighted “so that the anticipated level of turnout for those in different age groups and social grades matches the reported level of turnout for each combination of age and social grade in 2015”. In other words, their methods partly rests on an assumption that the intention/turnout ratios will be similar to 2015.

ComRes has taken a fairly similar approach. “Using all sorts of statistical analysis, we concluded a formula for taking account of those demographic factors which were likely to lead to people overstating their propensity to vote,” the company’s chairman, Andrew Hawkins, told FactCheck.

“The challenge is identifying the people who said they’re absolutely certain to vote, but then go on to not vote. Until 2010 there didn’t seem to be quite as much volatility among those groups – they were less likely to overstate their intention to vote.”

Have voting habits changed?

The concern for pollsters is whether there will be a significant change in turnout trends compared to 2015.

If there is, some polls might underestimate Labour support. But equally, it’s possible the pollsters still have not gone far enough to rectify the 2015 problem. If that’s the case, they could still be overestimating Labour support. We will only know for sure on election day itself.

One thing we do know is that this is a very volatile demographic, making it extremely difficult to predict what will happen.

This was highlighted by the Brexit vote last year. Pollsters tried using their new, improved methods to forecast the EU referendum, but got it even more wrong.

“The reason we were so far out was that we were down-weighting people who tended not to vote in a general election,” Hawkins told us. “But in a referendum, they did. A lot of those people had never voted before, and sometimes they didn’t even tell us they were going to vote in the first place.”

But was that just an anomaly for the referendum, or a longer term trend? Most pollsters think it was a one off – very different to a general election. So they are still basing their current models broadly around what happened in 2015, rather than 2016.

Hawkins said: “I guess, if I’ve got a concern this time around, it’s that some of those people who we didn’t model in in 2016, are they sensitised to voting now? Are they going to make it a habit?”

There’s also questions over how pollsters deal with other demographics. Political scientist Dr Chris Prosser told us: “If I were a pollster, I think the thing that would keep me up at night in this election is a different sort of non-response bias – namely whether switchers and loyalists are equally likely to respond to polls.

“Pollsters can make sure they have the right number of 2015 Labour voters in their sample by weighting, but if the 2015 Labour voters who respond to the polls are more or less likely to vote Labour in 2017 than the 2015 Labour voters who don’t answer the poll, this could lead to errors in the polls.”

So, could the polls be seriously wrong?

“Of course!” polling expert Professor John Curtice said. “But unless you can come up with an argument as to why the bias is worse now than it was four weeks ago … then the one thing that the polls have told you is that there has been a marked decline in the Tory lead. The one thing you can rely on in the polls is the trend.”

Curtice said it is possible that some of the polls are skewed because of the challenges of dealing with the age divide. “The age gap for turnout has tended to be rather strong at recent elections and maybe it will narrow. Who knows?

“This is the thing you can’t be sure about. Relatively small errors in the polls can make an enormous difference to the political story.”