Wednesday, March 15, 2017

The datasets for last night's downright weird YouGov poll are now out, and there is actually some good news to be found in them. (Most of this was probably already revealed in the Times report long before I went to bed, but I don't pay the Murdoch levy so I was none the wiser.) Most importantly, respondents were asked whether the UK government should "agree" to an independence referendum if Nicola Sturgeon asks for one. Inconveniently for Theresa May's "the Jocks don't even want a vote" narrative, the answer was a narrow "yes".

If Nicola Sturgeon calls for a second independence referendum, do you think the UK government should or should not agree to one taking place?

Should agree : 52%Should not agree : 48%

That's the first time I can recall seeing the question posed in that way, and it probably produces more meaningful results than the misleading "Should there be a referendum before Brexit negotiations are concluded?" questions we've seen in a few other polls (misleading because Nicola Sturgeon is not actually proposing such an early referendum). It's also worth bearing in mind that YouGov's sample is, for whatever reason, unusually "No-heavy" - so if it turns out they're understating the Yes vote, it's not unreasonable to suppose that the majority in favour of a referendum is also somewhat bigger than they're reporting. However, most of the fieldwork for this poll was conducted before Nicola Sturgeon actually went ahead and called for a Section 30 order to be granted, so it remains to be seen what effect that development will have on public opinion.

The other piece of good news looks like bad news at first glance - by a 60-40 margin, respondents say the Scottish Government should not campaign for independence over the next two years. That represents substantial progress since exactly the same question was asked a few months ago, when the margin was a much bigger 64-36. I suspect Nicola Sturgeon's announcement may have a further impact on those numbers as Yes supporters rally behind her decision, but time will tell.

Let's turn now to the really burning question : is there any explanation in the datasets for the oddity of the Yes vote in this poll being lower than in any poll conducted by any firm since the autumn of 2014, in spite of the fact that a telephone poll only last week put Yes in the lead? To a limited extent there is. To my surprise, and utterly incomprehensibly, the datasets confirm that YouGov have stopped interviewing 16 and 17 year olds, even though they must know that the minimum voting age at the next referendum will almost certainly be 16, just as it was in the last one. Now, it's true that 16 and 17 year olds make up only a relatively small percentage of the electorate, and that a poll of over-18s probably isn't going to produce radically different results from a poll of over-16s. But it might well make a small difference. If the correct electorate had been polled, it's quite possible that the Yes vote would have been 1% higher and the No vote would have been 1% lower. So it's entirely wrong for YouGov, the Times and the wider mainstream media to present Yes 43%, No 57% as being reliable and relatively precise voting intention numbers. Perhaps even more importantly, it's grossly misleading to suggest that the Yes vote has fallen slightly since the last YouGov poll, given that 16 and 17 year olds were interviewed last time around. If the two polls had used the same minimum age, they might well have produced identical results.

As I've said before, I don't think this is a conspiracy from YouGov - I just think it's a mixture of Anglocentricity and institutional laziness. Because they usually poll over-18s only, it seems they simply can't be bothered making special arrangements for Jock polls. If the voting age for Westminster elections had been reduced to 16 in 2014, there's no way on Earth they'd still be polling the wrong electorate a whole three years later.

There are no other obvious smoking guns in the datasets - the question asked looks fine, and is exactly the same one YouGov have been using for years. Very little weighting has had to be done by recalled 2014 referendum vote, although that in itself raises a few questions, because it must mean that YouGov are sending out invitations in a very careful and extraordinarily controlled way. So if there is something fishy going on, perhaps it's at the invitation stage. The notorious "Kellner Correction" made an artificial adjustment to the sample of SNP voters to make sure that they were the "right sort" of SNP voters - could something similar be happening now, but this time using the invitation stage as a filter? It's possible, but it's also completely impossible to tell, because that sort of information isn't publicly available.

By far the most suspicious feature of the datasets (although not proof of dodgy methodology) is that the No vote is much stronger among the least affluent respondents. That's entirely counterintuitive, and is the opposite of what we see in most polls. OK, sometimes you just get weird subsamples which cancel each other out to some extent (for example too many female Yes voters in a sample might be partly offset by the presence of an excess number of male No voters), but the problem is that less affluent voters are generally less likely to take part in polls, and therefore their responses usually need to be weighted up, while the responses of affluent voters are weighted down. That weighting usually works in favour of Yes, but in this particular case the opposite is true - the No-friendly raw sample of 397 less affluent respondents has been upweighted to count as 493 'virtual' respondents. So if there is something wrong with the raw sample (and it's hard to escape the conclusion that there must be), that problem will have been sharply magnified by the weighting.

To give an example of what I'm talking about - in the most recent Panelbase poll, the Yes vote among the less affluent part of the sample was 11% higher than among the more affluent part. A very similar pattern was seen in the Ipsos-Mori poll last week, and the BMG poll at the weekend. And yet in the YouGov poll the Yes vote is 12% lower among the less affluent than it is among the more affluent. It simply doesn't pass the 'smell test'.

48 comments:

The mood music is definitely moving in the swan song direction and the Tories know this.Their chances of winning another referendum are disappearing fast so they will have to come up with a way of neutralising it.Dirty tricks dept. at Whitehall will be working overtime on this.

Scottish Labour (in fact Labour generally) is in a huge mess. If you are the kind of person who would never vote Conservative, and there are plenty of those in Scotland from the Thatcher years, then you may vote SNP. Doesn't mean you are necessarily pro Indy.

I disagree about YouGov. I think they're very very far from being unbiased about anything regarding the "Union".

Cast your mind back James. Pretty much every time something seriously dodgy with sample/data/postprocessing of data occurs then the polling company is almost invariably YouGov.

Personally I reckon a polling company which pays* people to complete political polls is inherently flawed from the start.

YMMV of course.

*In theory. If you match the demographic. If you give the right answers. Who knows? Even staunch defenders of the union like The Guardian have had readers letters wondering if its a scam. I've never heard of anyone who got paid but I guess I'm not in the right "demographic" and nor is anyone else I know....

My best guess is that the C2DE's on YouGov's panel are more enthusiastic about Brexit than other companies for whatever reason. Also as James pointed out there had to be huge upweighting, so if that's the case then it could cause quite a lot of distortion.

The Nat sis want to herd the English into camps and dispose of them. Adolf Nicla is an English hating racist that has formed an axis with Jihadi's such as Abu Humza, OSalmond Bin Laden, Tasmina Islamic State and Jihadi John (Swinney).

May just has to apply the constitutional law and tell Knickerless tae fuck aff.Any poll that favours the Nat sis is legitimate any contrary is deemed wrong and a mistake.Brexit will go ahead and the Nat sis do not have to be consulted moreso as they would support the EU stance.The EU is on a road to collapse the Nat sis need tae smell their keech.

The ideological fanatics want to herd the English into chambers and dispose of them. With a country shorn of its security services and open borders without controls Scotland will become nothing more than a training ground for ISIS death squads to launch attacks on the Northern Atlantic. Please Scotland stop these Jihadi's now.

When a person has uncomfortable thoughts or feelings, they may project these onto other people, assigning the thoughts or feelings that they need to repress to a convenient alternative target.

Projection may also happen to obliterate attributes of other people with which we are uncomfortable. We assume that they are like us, and in doing so we allow ourselves to ignore those attributes they have with which we are uncomfortable.