Stuff

Today the polling inquiry under Pat Sturgis’ presented its initial findings on what caused the polling error. Pat himself, Jouni Kuha and Ben Lauderdale all went through their findings at a meeting at the Royal Statistical Society – the full presentation is up here. As we saw in the overnight press release the main finding was that unrepresentative samples were to blame, but today’s meeting put some meat on those bones. Just to be clear, when the team said unrepresentative samples they didn’t just mean the sampling part of the process, they meant the samples pollsters end up with as a result of their sampling AND their weighting: it’s all interconnected. With that out the way, here’s what they said.

Things that did NOT go wrong

The team started by quickly going through some areas that they have ruled out as significant contributors to the error. Any of these could, of course, have had some minor impact, but if they did it was only minor. The team investigated and dismissed postal votes, falling voter registration, overseas voters and question wording/ordering as causes of the error.

They also dismissed some issues that had been more seriously suggested – the first was differential turnout reporting (i.e, Labour people overestimating their likelihood to vote more than Conservative people), in vote validation studies the inquiry team did not found evidence to support this, suggesting if it was an issue it was too small to be important. The second was the mode effect – ultimately whether a survey was done online or by telephone made no difference to its final accuracy. This finding met with some surprise from the audience, given there were more phone polls showing Tory leads than online ones. Ben Lauderdale of the inquiry team suggested that was probably because phone polls had smaller sample sizes and hence more volatility, hence spat out more unusual results… but that the average lead in online polls and average lead in telephone polls were not that different, especially in the final polls.

On late swing the inquiry said the evidence was contradictory. Six companies had conducted re-contact survey, going back to people who had completed pre-election surveys to see how they actually voted. Some showed movement, some did not, but on average they showed a movement of only 0.6% to the Tories between the final polls and the result, so can only have made a minor contribution at most. People deliberately misreporting their voting intention to pollsters was also dismissed – as Pat Sturgis put it, if those people had told the truth after the election it would have shown up as late swing (but did not), if they had kept on lying it should have affected the exit poll, BES and BSA as well (it did not).

Unrepresentative Samples

With all those things ruled out as major contributors to the poll error the team were left with unrepresentative samples as the most viable explanation for the error. In terms of positive evidence for this they looked at the differences between the BES and BSA samples (done by probability sampling) and the pre-election polls (done by variations on quota sampling). This wasn’t a recommendation to use probability sampling (while they didn’t do recommendations, Pat did rule out any recommendation that polling switch to probability sampling wholesale, recognising that the cost and timing was wholly impractical, and that the BES & BSA had been wrong in their own way, rather than being perfect solutions).

The two probability based surveys were, however, useful as comparisons to pick up possible shortcomings in the sample – so, for example, the pre-election polls that provided precise age data for respondents all had age skews within age bands, specifically within the oldest age band there were too many people in their 60s, not enough in their 70s and 80s. The team agreed with the suggestions that samples were too politically engaged – in their investigation they looked at likelihood to vote, finding most polls had samples that were too likely to vote, and didn’t have the correct contrast between young and old turnout. They also found samples didn’t have the correct proportions of postal voters for young and old respondents. They didn’t suggest all of these errors were necessarily related to why the figures were wrong, but that they were illustrations of the samples not being properly representative – and that ultimately led to getting the election wrong.

Herding

Finally the team spent a long time going through the data on herding – that is, polls producing figures that were closer to each other than random variation suggests they should be. On the face of it the narrowing looks striking – the penultimate polls had a spread of about seven points between the poll with the biggest Tory lead and the poll with the biggest Labour lead. In the final polls the spread was just three points, from a one point Tory lead to a two point Labour lead.

Analysing the polls earlier in the campaign the spread between different was almost exactly what you would expect from a stratified sample (what the inquiry team considered the closest approximation to the politically weighted samples used by the polls). In the last fortnight the spread narrowed though, with the final polls all close together. The reason for this seems to be because of methodological change – several of the polling companies made adjustments to their methods during the campaign or for their final polls (something that has been typical at past elections, companies often add extra adjustments to their final polls). Without those changes them the polls would have been more variable….and less accurate. In other words, some pollsters did make changes in their methodology at the end of the campaign which meant the figures were clustered together, but they were open about the methods they were using and it made the figures LESS Labour, not more Labour. Pollsters may or may not, consciously or subconsciously, have been influenced in the methodological decisions they made by what other polls were showing. However, from the inquiry’s analysis we can be confident that any herding did not contribute to the polling error, quite the opposite – all those pollsters who changed methodology during the campaign were more accurate using their new methods.

For completeness, the inquiry also took everyone’s final data and weighted it using the same methods – they found a normal level of variation. They also took everyone’s raw data and applied the weighting and filtering the pollsters said they had used to see if they could recreate the same figures – the figures came out the same, suggesting there was no sharp practice going on.

So what next?

Today’s report wasn’t a huge surprise – as I wrote at the weekend, most of the analysis so far has pointed to unrepresentative samples as the root cause, and the official verdict is in line with that. In terms of the information released today there were no recommendations, it was just about the diagnosis – the inquiry will be submitting their full written report in March. It will have some recommendations on methodology – but no silver bullet – but with the diagnosis confirmed the pollsters can start working on their own solutions. Many of the companies released statements today welcoming the findings and agreeing with the cause of the error, we shall see what different ways they come up with to solve it.

@Neil A – I read a fascinating bit of research a couple of years ago about a software programme you could download to track your online activity. It’s purpose was to identify how you used internet and social media, and by classifying your browsing activities, measure how you were interacting with and being influenced by the online world.

The fascinating bit was that the experiment gave striking evidence that the notion of the online world giving increased access to a vast array of potential viewpoints and ideas is largely illusory, with the findings that most people merely migrate towards like minded groups and sources.

In practical terms, we can see the effect of this with certain online communities forming ‘groupthink’ approaches to each and every issue that comes along, with a dialogue of the deaf seemingly the most common outcome, and I’m sure self selection works in forums like this, when people of one view retreat when the opposing view seems to have taken a dominant role.

Moving political discourse increasingly to social media troubles me, as it seems that our world view horizons and exposure to new ideas are in fact shrinking, not expanding, while at the same time our ability to anonymise our interactions removes the behavioural checks needed for a civilised debate.

I don’t know if many on here have read the recent newspaper article by Sam Delaney, detailing how Crosby masterminded the Tory election victory in May last year. There were several interesting references to the canny use of social media in targeting potential supporters. Here’s what Tom Edwards, a communication and PR guru who worked with Crosby on the Tory campaign, had to say:

“The first two years was a case of building our database of Tory voters – and potential Tory voters,” he says. “We grew our mailing list from 300,000 to 1.5 million. That doesn’t happen overnight, and it took money. We did some stuff on change.org and other petition sites to find people who were interested in causes that chimed with Conservative values. We sought out like-minded people on YouTube and Facebook. Once you have these people on your mailing list, you can start communicating with them on a regular basis.”

As the election drew closer, Edmonds and his 10-strong team – under the stewardship of Crosby – began producing bespoke content for specific voting groups around the country. They ran a carefully planned digital campaign which produced a selection of messages that were sent out to different types of potential Tory voter. For example, one tweet was designed specifically with female Ukip waverers in mind and there were specific tweets aimed at Lib Dem waverers too.

Edwards went on to say:

“In terms of how we communicated with the public, it was a complete turnaround from 2010. We could put together ideas quickly in-house, get a quick approval from Lynton, who would be sat beside us in the office, then send it out immediately. We could also track its impact. If it seemed to be working, we would press on. If people weren’t engaging, we tried something else.”

The amount of online activity by the Conservatives was immense: in April, the press reported that the party was spending £100,000 per month on Facebook advertising alone (a figure which Edmonds confirms was “roughly accurate”).

The Delaney Guardian article was in fact an extract from a book he has written called “Mad Men & Bad Men: When British Politics Met Advertising”.

A must read, I would think, for any aspiring electoral strategist thinking already of 2020!

In most things ethical/moral I tend to believe that technology is pretty much neutral in terms of benefits. We gain as much as we lose. We solve some problems and create others.

The point being that you can’t stop people inventing stuff, and you can’t uninvent it, so we’re stuck with whatever comes along and just have to make the best of it.

The real boon for me of technology is that, if you live outside China and North Korea, quality information is available at your fingertips virtually for free. I consider myself capable of sifting it from the vast amounts of misinformation and so for me that is a Good Thing. For society at large the balance of information vs misinformation probably means that yet again its neutral overall.

Social media largely recycles MSM (and not so mainstream), but with a different timeframe and sensitivity. It is not quite new. FT’s influence (whatever great paper it is) is reliant on the pieces picked up by BBC really.

I do think social media’s public perception is overdone by both MSM and here. There are some highly reliable academic research on it, and a lot of awful ones. I was involved in two such projects recently. One on a sport issue showed how time consuming it is to do it properly (the other one was abandoned and used a different methodology). I also know a number of software companies that have attempted to use social media for business. None of them got to the third round of funding.

So, I’m really agnostic about it – as long as there is no evidence. The Tory campaign sounds solid, but also an industry story, so I have doubt.

Social Media is where GE 2020 will be fought and looking at the Tory social media spend possibly where 2015 was won.

In political terms it doesn’t matter if people are getting confirmation of their already held views most importantly they are getting arguments and counter-arguments to take to their family, friends & workplace.

Many people on social media are leaders and influencers in their social group so now have the ‘message’ to spread to others.

I am really surprised that Labour who have a strong online presence didn’t make more use of SM seems to be a big mistake.

The MSM is largely discredited and seen as a vehicle of their rich owners or as a state broadcaster. So social media it is.

I agree with you about political debate & social media.It depends on the site of course, but on so many you can see groupthink & aggressive rejection of alternative ideas in action. Its like the evolution of Cults.

But we musn’t fall into the trap of equating Social Media with the Internet’s massive store of knowledge & information.

I’m not much persuaded by Orwellian notions of groupthink, to be honest, and it may well be just a derogatory term to describe a body of opinion that grates with ones own. Most people like their prejudices pandered to and their opinions endorsed and flattered, so I don’t hold much store to people’s over-sensitivities when they have the shock of confronting opinions that differ from their own.

Of course people tend to migrate to groups that agree with them and you’d have to be a confirmed contrarian to always want to be in a minority, but I’ve not come across some of the Orwellian nightmares and cults that some appear to have done. I suspect a lot of it is an overblown sensitivity and irritation at having to confront opposing views on life. Insulting these views by labelling them as groupthink is a convenient way of devaluing them and comforting yourself that you possess more rarefied critical faculties. I detect a bit of intellectual snobbery at play.

I think there is an issue about the cloak of anonymity on social media and in the blogosphere, however, and how this allows a lot of bullying and moronic abuse. It’s amazing how brave people can be when on a keyboard and in front of a screen and how social niceties and inhibitions can quickly disappear. Maybe this is the price we have to pay for the explosion in electronic and faceless interaction.

My remedy to all the fear and doubts though isn’t to twitch the curtains and take to the keyboard but it is to basically get out more and meet some real people, many of whom will agree with my view on the world and many who won’t.

One does notice some aggressive rejection of alternatives by peeps who bring stuff up then when you indulge them say they don’t wanna talk about it. Then they do the same again about summat else. Then wierdly they may bring it up again moments later talking to someone else. It’s a bit surreal but one tries to go with it…

I dunno, an important part of posting online is putting your views up to have them challenged to root out any flaws one might have missed.

With some things that happens less, because one has already had them tested numerous times elsewhere and already modified them to be more robust. With newer topics, like dredging, one’s views might change a whole lot more.

To some extent, whether peeps change their mind politically, depends on whether parties change, or situations.

I mean, there’s been quite the change in Scotland. Look too at the rise of UKIP, and the collapse of LDs.

The most interesting thing about Crosby’s social media campaign was that Labour were completely oblivious it was happening.

That is, the Conservatives were targeting some 1.5 million people on Facebook and Labour didn’t realise it because they weren’t part of those groups.

That’s essentially the difference between Facebook and Twitter. On Twitter everything is public so the other side can see what’s going on (especially as Twitter publicises trending tweets). On Facebook it’s private – so unless you are part of the target group, you won’t have seen the ads or the material being shared.

I think it goes a bit deeper than that. The really crucial part was Crosby’s method of identifying potential activists from looking at online petitions where people put their names down for conservative-ideology proposals. Then using Facebooks targeting systems to multiply that out by targeting those people, and their friends.

The problem for the Labour establishment, is that they seem to despise the people who fill out online petitions for progressive-ideology. It’s not that they can’t do the same thing as Crosby, it’s that they refuse to do so. And they do so in the sake of “maintaining the middle ground”, in spite of all clear evidence that this tactic isn’t working.

Corbyn however, clearly did, and can, tap into this if he wants to. If he can get the Labour Establishment on the same side, and start buying their own Facebook targeting, then they can pull the same tricks as Crosby. But first they need to stop accusing their own voters of being bullies for disagreeing with them.

It’s always useful to see where on-line comments have come from. It reveals attitude and bias (which doesn’t of course decide whether the facts are correct or not). For anonymous posters the only guide is what they have previously said.

@Carfrew. Do you mind telling me why you refer to people as peeps (which are the sounds car horns make)?

Incidentally, yes, Facebook are entirely willing to let you target “promoted posts” to a specific list of people and their friends… and for a bit more money, *those who’s Facebook activities are similar to them*. That second part is the big multiplier that Crosby tapped into.

My understanding is that Google will target your search results to match things you have previously shown an interest in. I think it is called a filter bubble. The issue when it comes to Politics is that it tends to mean you end up with results that fit your own biases when you do searches and so are perhaps do not get as much access to different viewpoints/ideas.

@colin – I don’t, and it may well be one of those apocryphal tales, but one that’s funny enough to be aired again.

@Candy/Jayblanc – I noted the method that Crosby used to gather target voters, and thought it was very imaginative. Tories actually set up petitions to gather contacts. One of Labour’s problems was that the thing that ultimately gave the election to cameron was what happened in Lib dem/Tory seats, and Crosby was effectively targeting wavering Lib Dem voters in 23 seats, telling tham that a hung parliament would lead to Salmond bullying a weak Milliband, so only a Tory MP in their seat would do. It worked brilliantly, and also feeds back into the theme of this thread.

had the pollsters not got things so wrong, the boot would have been on the other foot. If a Tory majority was on the cards, I suspect a fair few of those 23 seats could well have stayed Lib Dem, and perhaps other labour waverers might have turned out to vote elsewhere.

@”the thing that ultimately gave the election to cameron was what happened in Lib dem/Tory seats, ”

Why choose these as the key factor in Labour’s failure? -LD’s lost 11 to Labour..

Labour had other disasters-some anticipated like the undoubtedly critical losses to SNP ( nothing at all to do with Crosby, and the Polling for which merely underlined Crosby’s s warnings about SNP power at Westminster)-and some not so anticipated like the 9 losses to Cons.

Using petitions and causes isn’t a new thing per se. Obama did it. But that was a case of your party doing some local activism and using that, whereas here Tories were tapping into stuff already happening? And then of course using Facebook to leverage and find similar peeps (or North American waders).

It also shows an issue with polling. Pollsters, concerned only with the overall percentage, may go “but there was only 0.6% down to late swing over SNP hegemony fears”, which doesn’t sound like much, but if that 0.6% of car horns are concentrated in marginals and targeted by advertising etc., then the percentage vote may not change much, but the seat outcome is significant.

“Labour had other disasters-some anticipated like the undoubtedly critical losses to SNP ( nothing at all to do with Crosby, and the Polling for which merely underlined Crosby’s s warnings about SNP power at Westminster)-and some not so anticipated like the 9 losses to Cons.”

I wasn’t commenting on campaigning at all. I was just suggesting that-as is indicated in the MORI Poll I referenced, whilst a large majority agree that Social Media encourages political debate, 50% ish think it makes political debate more “divisive” & ” superficial”.

I suppose the emergence of Party campaigning techniques which identify & target Facebook users with “friendly” views merely emphasises the point.

I know you weren’t commenting on campaigning. I mention it because it might challenge the your idea of waders who get stuck into politics online as being unrepresentative. Tories seem instead to have gone looking for these peeps.

It also puts me in mind of MPs who took letter to them seriously, because for every one car horn who puts pen to paper, there might be another thousand who are also exercised on the matter if not moved quite enough to write…

It’s not really a great innovation by Crosby, this became an established marketing technique before the 2010 election.

The differential is that the Blairites *refuse to engage with the party’s ground level activists*. Mainly because the party’s activists became rather antagonistic towards the directions the Blairites wanted to take the party in, and the Blairites becoming intractable about it in the name of “electability”. The result being, that they became less electable.

I expected Labour to win last time, because they do have a better ground level activist pool. There’s more entusiastic engaged young Labour activists abour than Conservative ones, in part that’s what threw off polling because they’re louder.

But that pool of activists are useless if that pool is not activated to campaign, which it hasn’t been. Instead, many of them ended up being alienated by the PLP. I strongly underestimated how huge the distance between the Blairites and the Activists had become, and even after the election it wasn’t made explicitly clear till the leadership contest.

Sadly, I do think the party will be unelectable as long as there is such a great disconnect between its activists and the party establishment. And it does look like establishment Blairites are reluctant to be the ones to change.

The Conservatives did not go looking for people who didn’t agree with them. They looked for existing polls that agreed with Conservative sentiment, and possibly created some of their owns. Then made notes of the names of the people who *were agreeable*. This wasn’t a plan of convincing people, it was a plan of finding the easy to convince and turning them into loyal party activists.

Again, this is a known marketing method. You find the people who are already buying into your product, that you can turn into evangelists for your product. Then they’ll be doing a whole bunch of social media marketing for you, and you don’t even have to pay them.

Along those lines, I wonder if we’ll eventually see ‘prize bounties’ for getting your friends to sign up as party members. Or “Tweet about using the hashtag #TobinTax, to enter this prize draw…”

They had (found) opinion formers who would feed their own networks. These set up closed (secret – in FB terminology) groups with a purpose, and used these to push messages through. There were also specifically targeted feeds to identified FB users (people who liked what you liked also liked this). This is not the usual FB advertisement campaign that you see.

That is, the Conservatives didn’t target populations through ads (they did to some degree but it wasn’t the main one), but used (and funded some of the) groups. The groups discussed “issues” and these “spilled over”. Members of such groups can invite others. In addition, these allow targeted ads (your “friends X and Y liked this). It also used news feed not normally referenced.

So, the point was finding these opinion formers (not that easy – the reliable software is expensive, but they probably used a bespoke one).

A little bit clandestine, but perfectly legit. The point was creating a united message against something (rather than for something), which eventually turned to for something.

The Labour Party is nowhere close to this (still).

Also I can’t judge how many votes it meant. There is an industry story, but it is far from verified.

Eh? I know. Maybe I wrote it poorly, but what I meant to say was saying is what you are saying.

I was saying in the past, they might get stuck into some local activism and find people engaged in something that might make them presdisposed towards your message that way. Now they can go online and find peeps already involved in summat, and then via them find others using Facebook’s ability to find matches with others similar.

Can you point out what you thought I said that was opposite to this so I know where at wasn’t being clear?

The secret groups thing is interesting. Because it’s exactly the sort of thing Colin is concerned about. On here, if he doesn’t like what someone says about Lidl, he and others can challenge it. This is the opposite of what happens in a cult where peeps are isolated and dissent is quashed. And in these secret online groups of the like-minded, it seems rather less open to challenge. Peeps could be trashing Lidl, and it would be under the radar…

Well, *Corbyn* is already there. Since this is practically how his leadership campaign worked.

The problem is that the PLP are entirely antagonistic to the kind of people needed to participate in the social media groups. To the point of labelling them “cyber-bullies”.

Cameron is all too willing to butter up the activists on the right of his party when he needs to. Meanwhile, parts of the Labour Establishment are openly talking of a split if the left of their party doesn’t stop being so uppity. There’s simply no way for the Labour party to get an active ‘ground game’ of activists, let alone mobilise on social media, if they hold those activists in contempt.

@Carfrew

In that it’s opposed to finding new people who didn’t think the same as you. It’s finding more of the same people who do.

In my constituency the young and slightly rebellious Tory candidate apparently eschewed CCHQ assistance, but he was a towering figure on Facebook. I’m not sure if Crosby had any hand in this but his campaign account kept popping up with little videos and pledges and get to know you pieces.

It may have helped that he’s a fit former Army officer who apparently has something of a fan base amongst the city’s young ladies.

It worked though. Detaching a safe Labour seat that noone even had on their target list.

Yes, there’s a development in Labour’s social media strategy, but it’s largely national (elections aren’t…). Some of their closed groups are pretty good and use a variety of feeds. Quite interesting and yes, largely lefty, but still rather tolarant). So they don’t combine social media with intimacy (what works there is creating FB friends through these networks). Still these are early days.

However, Labour in general is much more efficient on the ground around here than in the last 20 years. Collecting donations with declared purpose(s) – essentially financing campaigns in wards where there will be elections in May (but they collect donations in every ward).

Alec – “One of Labour’s problems was that the thing that ultimately gave the election to cameron was what happened in Lib dem/Tory seats, and Crosby was effectively targeting wavering Lib Dem voters in 23 seats, telling tham that a hung parliament would lead to Salmond bullying a weak Milliband, so only a Tory MP in their seat would do.”

It’s important to remember that it wasn’t just the polling that made those seats vulnerable.

Labour took the decision to target Nick Clegg personally – which drew thousands of LibDem activists to Sheffield to defend him when normally they’d have been defending seats in their south-west heartland. So the Conservatives had a shot at an undefended goal.

Miliband’s team just didn’t think strategically. It never occurred to them to wonder what the consequences of their actions would be on their opponents.

@Jayblanc – am amused that you are calling the Miliband people “Blairites”. I thought he was well to the left to Blair. Are “Blairites” everyone who is not a Corbynite now?

“In that it’s opposed to finding new people who didn’t think the same as you. It’s finding more of the same people who do.”

——–

I asked if you would help by showing me where I indicated that, because I don’t think I did. My point was that previously campaigners found the likeminded via local activism, now they can use Facebook. And having found some, Facebook lets them find still more.

HOWEVER… later on, having found the like-minded, they might craft messages to peeps to vote DIFFERENTLY, e.g. Lib Dem waverers. Maybe by hyping the SNP thing.

Thus, it’s a bit more complicated than you may think. Some campaigning is not to change peeps minds. Tories don’t necessarily want Tories to change their mind and vote LD. They prolly want them to stay voting Tory.

Unless… it might for example stymie Labour by keeping an LD in the seat. Then they might want them to change their vote from Tory to LD. At other times, they’ll be campaigning for waverers to change say from LD to Tory if Tory stands a chance of winning.

So one rejects the idea they don’t want peeps to change their mind. Often, they do. Parties like to get peeps to vote the way they want. It’s just that one core method is that they try and persuade peeps to change their mind by gaining trust by agreeing on things involving core values.

It’s straight forward influencing, marketing. Get trust by agreeing with them on values, then get them to behave how you want.

“Miliband’s team just didn’t think strategically. It never occurred to them to wonder what the consequences of their actions would be on their opponents.”

——–

Well, you have a point, in hindsight, now peeps know how the polling was wrong.

Strategically, it wasn’t automatically a bad idea to get the Lib Dems to squander resources defending Cleggo, even if it meant Tories picking up some seats, because in the process it neutralised a party that had been a thorn in Labour’s side, splitting their vote for years.

Unfortunately for Labourists, in the interim, two new vote-splitters had arisen: UKIP and SNP. However, polling still suggested Labour might be the largest party. Turns out to have been wrong, and those LD seats suddenly became very important in giving Tories a majority.

That was the crux of the Tory approach: to fashion new vote splitters for Labour. By hyping immigration, by campaigning with Lab. over Independence, by hyping SNP hegemony etc.

And then with that being still insufficient, just getting over the line by micro-targeting waverers.

“Crosby probably even has someone working for him on ukpollingreport.
I wonder who?
:)”

I’ve had suspicions about Chris Lane 1945 for some time. An anagram of his name is Linch Arse, Linch being Aussie strine for lick. The 1945, when added together, equals 19, the exact number of key marginal seats targeted by Crosby in May last year.