Young Labour supporters at the leaders’ debate in Cambridge on Wednesday. Whether the surge of younger voters to Labour actually translates to ballots cast on Thursday lies behind the divergence in the polls.
Photograph: Dan Kitwood/Getty Images

The key issue behind this week’s divergence in the opinion polls is whether the younger voters inspired by Jeremy Corbyn’s populist politics, who make up the sudden surge in Labour support, will actually make it to the polling station next Thursday.

The latest YouGov/Times poll published on Thursday puts the gap at only 3% as Theresa May’s lacklustre campaign stalls and Labour’s share of the vote – at 39% – reaches its highest level for three years. It follows a Survation poll earlier this week also showing a single-figure lead for the Tories of only six points.

But at the same time a clutch of other polling companies, including ICM and ComRes, have published surveys showing the Tories maintaining double-digit leads of 10% to 12% – far lower than at the start of the campaign. These polls show Labour making progress especially among the under-35s, but the overall voting intention figures still point to healthy 80-plus seat majorities for May.

Guardian/ICM poll: Tories' 12-point lead offers Labour crumbs of hope

Read more

Some have seen these polls showing single-figure leads as some kind of secret Tory plot designed to convince Tory voters that there is a real danger that Corbyn is knocking on the door of Downing Street to maximise their turnout.

The criticism usually attaches to YouGov, which was founded in 2000 by two Conservative activists, Stephan Shakespeare, a onetime spokesman for Jeffrey Archer, and Nadhim Zahawi, who quit in 2010 to become a Conservative MP.

For a long time, however, the public face of YouGov was Peter Kellner, a highly respected pollster who is married to a Labour peer, Catherine Ashton. Kellner stepped down as YouGov president last year and was replaced by its chief scientist, Stanford University politics professor Doug Rivers, whose Silicon Valley company was bought up by YouGov.

Rivers, a data scientist, is behind the company’s innovative new election model which startled the Tory campaign on Tuesday by producing the first estimate in the campaign that pointed towards serious Conservative losses and a hung parliament.

This provoked Jim Messina, May’s US consultant, to tweet that he had spent the day “laughing at another stupid poll from YouGov” and offering a charity bet that they had got it wrong.

Salt was poured into the wound when it emerged on Thursday that YouGov’s new constituency estimates suggest that the home secretary, Amber Rudd, fresh from her performance as May’s TV debate stand-in, could actually lose her Hastings and Rye seat to Labour.

Martin Boon of ICM, whose Guardian poll on Monday which put the Tories on 45% and Labour on 33%, says that in fact the divergence in polling results has brought to a head the new methodological battleground in the polling industry.

This centres on their treatment of younger voters, especially 18-24s but also 25-34s – two groups in which Labour actually enjoys a lead over the Conservatives – and how they are reported in the final voting intentions.

Put simply the polls such as Survation and YouGov are doing a brilliant job in reaching more 18-to-24-year-olds than other polling companies. But in doing so they are in danger of repeating the mistake the polling companies made in the 2015 general election when they overstated the Labour vote and understated the Conservative vote.

This is because while younger voters may tell pollsters in large numbers they intend to vote Labour, they are far less likely to actually make it to the polling station. The difference is quite big. Only 44% of 18-24s in the ICM poll say they are 10/10 certain to vote next Thursday, compared with 66% of 35-64s and 80% of over-65s.

As a result of the inquiry into the 2015 UK polling debacle, companies such as ICM and Comres now weight their voting intention figures for younger and less affluent voters by their actual turnout rates at previous elections.

Other companies such as Survation and YouGov are not doing this and weighting them by current turnout intentions. For example Survation reported that 82% of 18-24s intend to vote at this election, compared with 43% in the 2015 general election. It is more complicated than this but that is the key to why YouGov and Survation polls are producing single-figure leads for the Conservatives.

Boon says if ICM were using similar methods, it too would be producing similar single-figure leads: “How pollsters address the turnout issue is now central to what a poll says.

“We at ICM turnout weight using a matrix that assumes younger people will be less likely to vote than older, and less affluent people will be less likely to vote than the wealthy. This has been the general pattern of general elections for an age, and whether you believe our poll findings or those of others will depend on whether or not you think Jeremy Corbyn can actually buck that trend,” said Boon.