Wednesday, May 6, 2015

Redemption for the pollsters, revolution for Albertans

It was an historic night in Alberta, as the Progressive Conservative dynasty was cut short at 43 years and Albertans elected an NDP government for the first time. And they did so in dramatic fashion.

Others will parse what this means for Alberta and the New Democrats more generally. But let's talk about how the polls did.

The sigh of relief you might have heard was from pollsters from coast-to-coast. Whether or not they were involved in either the 2012 Alberta or 2013 B.C. elections, the credibility of every pollster was chipped away by those misses. With the success last night, the ghosts of the 2012 result have finally been put to rest.

It was all the more a relief because it was Alberta, and because what the polls were saying could happen was so outlandish that everyone bent over backwards to try to explain why they were wrong. There were a lot of people ready to gloat last night before the votes were counted. It was a terrifying prospect.

But in the end, as I pointed out in my final analysis, the polls could argue for nothing but an NDP landslide. They were right. Individual pollsters had good nights. They were even better in the aggregate - though what they missed on, as marginal as it was, is revealing.

I was pleased with how ThreeHundredEight.com's projection performed, and I'll get into that more below. But one thing I was disappointed with was the assumption I made about the support of the Liberals and the Alberta Party. I penalized them for not running full slates, and distributed those lost votes to the other parties proportionately. This had the effect of boosting, above all, the NDP, and overall made the projection less accurate than it could have been.

As you can see, an adjusted projection would have performed better than any pollster except Léger. With the adjustment, ThreeHundredEight.com finished fourth.

The winner of the night was Léger, with a total error of just 1.6 points per party (for the purposes of this calculation, I've lumped AP together with the others). A bit of luck played into this, though, as Léger's final poll was out of the field seven days before the vote. It was the only pollster that was not active over the final weekend to not finish behind those who were.

Only its estimates for the Liberals and other parties was outside of the (theoretical) margin of error, and Léger was one of only two pollsters to put the Progressive Conservatives in second place.

This is an important point, as I think it is very revealing of what happened last night. All but Léger under-estimated PC support, by margins of four to eight points. It is unlikely to be coincidence. The Tories either benefited from better turnout (relatively speaking), had harder-to-reach supporters, or experienced a bit of a ballot box boost that might have been worth some four to five points. I think a combination of all three is likely.

Insights West and EKOS Research also had good nights, though both had the PCs outside the margin of error and EKOS had the NDP a little high.

Forum Research and Mainstreet Technologies performed well, but both had the NDP high and the PCs low.

Ipsos Reid had the Liberals too high and the PCs too low, while ThinkHQ only had the NDP within the margin of error. Return on Insight had way too much support for the Liberals and other parties, which had an effect on their overall estimation.

But Ipsos Reid, ThinkHQ, and Return on Insight were out of the field a week before the vote, so they may have simply stopped polling too early to catch a collapse in Liberal support that benefited the Tories (though Léger did not have the same problem). They get a bit of an asterisk.

Nevertheless, for the worst performance to be an average error of just 3.8 points per party is absolutely stellar. And no pollster presented a narrative that was wrong. They all suggested the NDP was headed towards victory, and probably a majority one, while Wildrose and the PCs would fight it out for second place. In the end, they under-estimated the Tories in the popular vote, but their view that Wildrose would fill the Official Opposition role proved correct. Albertans were not misled by the polls whatsoever.

A few other revealing insights from these polls: all but Insights West and Forum Research over-estimated the Liberals, to the tune of one to six points. So perhaps my assumption that this is what would happen was not wrong (it was just too penalizing).

The polls in the field before April 29 all under-estimated NDP support (by two to four points), while all the polls in the field after April 29 over-estimated that support (by one to four points). There was a bit of bandwagon effect in the final days, perhaps, but also a bit of a deflation in the final moments.

Wildrose was estimated just about correctly by everyone, and certainly in the aggregate. But why was Wildrose able to win more seats than the PCs, despite winning fewer votes?

Wildrose benefited from a very efficient vote outside of the two main cities.

The New Democrats did, in the end, win Calgary. But the margin was a lot tighter than expected, with 33.1% of the vote instead of the 35% to 41% the projection model awarded them. The PCs were the largest beneficiary, taking 31.5% of the vote (instead of 25% to 29%). Wildrose under-performed, at 23.8% instead of 26% to 29%.

But despite this performance by the PCs in Calgary, they just didn't win many seats out of these numbers. With only a 1.6-point advantage over the PCs, the NDP managed to win 14 or 15 seats in Calgary, with the PCs winning just seven or eight. The Liberals won one and the AP won another of the 'Calgary'-named ridings.

Why did I say 14 or 15 seats? Oh, nothing. Just that the riding of Calgary-Glenmore had a freaking tie at 7,015 votes apiece for the PC and NDP candidates!

The New Democrats dominated Edmonton, taking 57.6% of the vote. That was just about as expected. The PCs over-achieved a little at 22.7%, while Wildrose under-achieved at 13.6%. Probably not too much of a surprise - in both Calgary and Edmonton there seems to have been a small coalescing of the conservative vote in the PCs' favour.

In the rest of Alberta, though, there was no such coalescing. And that was the death-knell for the Tories. Wildrose won the rest of the province with 35.2%, just about as expected. While the PCs did take a little more of the vote than projected, with 29.3% in this region, they still finished in third. The NDP dropped from a range of 34%-40% to 30.6%, but it was enough. The party won a few rural ridings, as well as sweeping Red Deer and Lethbridge. The PCs just came up short everywhere. Their uniform support was disastrous when the NDP had the advantage in the cities and Wildrose everywhere else.

Overall, I was very happy with the performance of the projection. All parties but the Liberals (who nevertheless had a maximum of one seat) fell within the likely ranges, and very comfortably so.

The NDP and Wildrose fell just short of the low range in the vote projection (but within the minimum range), and the Liberals outside of the maximum range, but this was in large part due to my penalizing of the Liberals. The PCs fell within the high-to-maximum range. No party but the Liberals ended up outside the 95% confidence interval anywhere, which is a very happy result.

The model did very well at the riding level. The winner was called correctly in 74 to 75 ridings (depending on how Calgary-Glenmore, projected to go NDP, turns out), for an accuracy rating of 85% to 86%. This is a good result considering the number of incumbents who were defeated.

More importantly, the ranges identified the potential winners in 80 to 81 ridings, for an accuracy rating of 92% to 93%. The average confidence in the ridings that were missed was just 63% to 64% (and four of them were 50/50 toss-ups), and the margin in these incorrect ridings averaged just 1,033 votes.

This election was an usual example, though, of the model actually doing slightly worse with the real results plugged into it. This, too, is revealing. It shows just how inefficient and unlucky the PCs were.

Had the polls been exactly right, the projection would have awarded 26 to 52 seats to the NDP, 15 to 25 to Wildrose, 12 to 43 to the PCs, zero to two to the Liberals, and zero to one for the other parties.

All results would have been within the 95% confidence interval, and only the PC and NDP results would have fallen outside of the likely vote ranges. But we would have headed into election night far more uncertain about the NDP's potential victory.

Many of the errors would have occurred in Calgary, giving NDP seats to the PCs. Again, this demonstrates how inefficient the PC vote was in the city. Nevertheless, the correct winners would have been identified in the likely ranges in 79 of 87 ridings, for an accuracy rating of 91%. That is a perfectly acceptable result.

This is where I think post-mortems using seat projections come in handy. It tells us a lot about what happened. The PCs did a lot worse than they should have in many ridings, while the NDP did a lot better. Wildrose performed up to expectations.

Kudos go to the pollsters, who called this election boldly and correctly. Lesser men and women would have balked at re-entering the field in the final days in such an unusual campaign. But because they did, we knew that the NDP was on track to win a majority government. People just needed to see it to believe it.

48 comments:

> But one thing I was disappointed with was the assumption I made about the support of the Liberals and the Alberta Party. I penalized them for not running full slates, and distributed those lost votes to the other parties proportionately.

I think your penalty might be built on an inconsistent assumption.

You reduce the party's vote-share by the percentage of missing candidates, but in so doing you assume that a person who tells a pollster that they intend to vote for Party X is equally likely to live in any riding.

Applying the same idea at the national level would irrationally penalize the Bloc Québécois for not running candidates in Saskatchewan.

For a model-consistent penalty, I think that an ideal solution would be to run the model first without penalty and assuming a full-slate of candidates, then apply the vote-culling on a per-riding basis.

If that is not possible due to the structure of your model (I'm not sure how your riding projections and aggregation intersect), then a reasonable "top-line" penalty would be to reduce a party's polling share by the percentage of previous party-voters who would be unable to cast their ballot for the party again. A party that was missing candidates in its stronghold regions would face a larger penalty than one that gave up on marginal areas.

Speaking as someone who's been supporting parties with single-digit support, and talking to other people who do the same, for a couple of decades now, I think the seeming correlation between marking a party with less than a full slate down, and minor parties generally doing worse in elections than they do in polls, is a happy coincidence, not something that's actually a product of causation.

If you've ever been to a nomination meeting of a party that's at or around, say, 1% of the polls, the sort of party that can't nominate a candidate in every riding... they don't nominate candidates in random ridings! They nominate candidates where they can find someone (they may be pickier now, but back when I was active "someone" meant "anyone") who will volunteer, and that generally means someone living in the riding. So the ridings where they're not running candidates have a strong tendency to be ridings where no-one would vote for them any way.

I think that the reason these parties underperform their polls is simpler: potential voters for these parties know that we vote in a first-past-the-post system. So if they're on the phone to a pollster anyway, they might as well name the party they actually support: it doesn't take any extra effort. But I think those voters are more likely to perceive voting on election day as a waste of time. Why waste an hour or so voting in a system that's rigged so your vote doesn't matter? (Not that I'm bitter.) Whereas voters for major parties feel that their party has a chance of winning or losing, so voting seems more meaningful.

The Alberta Liberals differed from most parties with support in the single digits in that their vote was highly concentrated, focused on several incumbents. If you're thinking of voting for an incumbent, it doesn't seem futile the way voting for a party that's at 2% overall and is probably at 2% in your riding does.

After misses in BC, Alta 2012 and partially last time in Ontario (i.e. predicting a slim Lib minority vs. the actual majority), the Alberta and PEI and last Quebec elections seem to have the pollsters back on track.

There are two things that I wonder about in the aftermath of last night. The first is whether or not the high precision of the polls will prove to be a harbinger of accurate federal polling later this year. If they nailed the Tories' percentage in Alberta, will this be the year they finally nail it federally?

The second isn't so much about polling as much as what the results mean for the federal election. If the honeymoon period lasts long enough, it makes me wonder if the party might improve their standing in Alberta in the Federal election. Then again, if they get off to a rocky start, I wonder if they might even lose the gains they seem to have made.

I'm not Canadian, and thus don't see all the daily news updates, so I may be way off-base with all of my thoughts, but it seems to me that last night makes October look even more interesting.

Eric, after reading this, I went back through your archives to the lowest of the low, the BC projection debacle.http://www.threehundredeight.com/2013/05/polling-industry-dealt-major-blow-in-bc.htmlYou even questioned the value of your work on this blog, which I and many others consistently read with interest.A heartfelt congratulations should be given to you and the polling industry, who have clearly made efforts to improve their data, by including very late polls to catch any swings at the last minute. Looking forward to insight in the federal election in the fall.

Bryan at tooclosetocall points out that the polls called the NDP/Conservative margin wrong by ~7-8%. That's actually a pretty big miss. Had this been a closer election it'd have been BC all over again.

The problem with that sort of thinking is that it assumes the polls should be 100% right, all the time. It forgets the fact that polls are snapshots, and that voters can do unexpected things.

Who's to say the polls were not right on Monday, but that there was a 4-point swing to the Tories on Tuesday in the ballot box and because of turnout? That seems to be a more plausible answer than the idea that virtually every pollster had a bias.

I disagree. We don't live in a vacuum, polls are not being conducted under perfect testing conditions. Yes, they should be within the ballpark. But a win for me is whether or not polls informed voters accurately. If they can do that, they are a useful tool. If they mislead voters, as they did in 2012 and 2013, that is a failure. Whether they get it within an increasingly theoretical-only margin of error is of academic importance.

I don't think the polls in 2012 or 2013 did mislead voters. In 2012 in particular, voters did intend to vote WR. They wanted WR to win.

And then, as the moment approached, they changed their mind.

BC 2013 is a more interesting case. I think the polls there showed what BCers wanted them to say. BCers were angry at the Liberals, and wanted the Liberals to know it. They very publicly declared that that Liberals' behaviour was not acceptable.

But they also wanted not to have an NDP government (few provinces have been as badly served by an NDP government as BC was). So they told the pollsters what they wanted the government to hear. And then they voted for the government.

Polls like that can only mislead the public if the public misinterprets them, and the blame for that failure lies with the public, not the pollsters.

I came across the similar question of what to do with the Liberal and Alberta Party numbers when I was doing up my projection model, Eric. I tried the way you did it and played around with it a bit, but came ultimately to the conclusion that a) there was no way to really know how these orphan voters would go, and making assumptions about them would skew an already strange trend, and b) their numbers were so low to begin with that even if 100% of the orphaned voters moved to another party, it would likely be captured within the ranges of the model and the margin of error in polls. It was a good decision it seems, as I got lucky and was able to pinpoint the vote totals fairly accurately for each party, with the exception of flipping the PCs and Wildrosers around.

However, I think your idea was sound if there were major party candidates missing, like say there was no PC candidate in a riding where they usually got upwards of 20% or so. That is a serious chunk of voters who have no home, and also could not be captured by the usual ranges set up by models. Maybe when you have that sort of situation, proportionally parsing out the vote would work much better, or at least give some indication of how things can go. For smaller parties missing candidates, it won't even matter.

Following up on a comment I made on the previous article, too many people are jumping the gun on October. If the NDP do win at the federal level, it will be because campaigns matter, and they ran a good one, and not because of what happened here in Alberta. As with the comparisons between 2012 and 2015, the comparisons between May and October are superficial. Why?

1. The vote split on the right.

It would be a disservice to Notley's brilliant campaign to call it the major factor in their victory, but the fact remains that the split between PC and Wildrose did help them out. The CPC, in contrast, is as united as ever.

2. The vote split on the left.

Another factor in the NDP victory was the complete collapse of the Alberta Liberals. Lefties in the province had been pining for a united left for years, and you know, I don't think anyone is quite sure how it happened. So while it's entirely possible that Trudeau and co. will be just as defeated come election night—once again, campaigns matter—you can hardly expect them to be dead before it even starts. But in Alberta, the Liberals were. Their single seat is one of the few things that we saw coming.

3. The voter disgust

There have been scandals, and boy has Mulcair screamed. Robocalls. Fighter jets. Senate expenses. But there's just no comparison to the levels of voter disappointment and yes, even disgust, that Prentice faced in Alberta. Whereas the people who have been truly upset with the feds are those who were against Harper from the start, Prentice's usurping of the Wildrose, and then "look in the mirror", and then the bad-news budget, and the attitude behind it all, angered traditional supporters and detractors alike.

—

The third point, of course, is subjective. Many have said that they are, in fact, disgusted with the Harper government, and though I doubt that any of those voted for him to begin with, I don't have the actual numbers. But one and two are solid and objective. No one has yet to form a centre-right Conservative competitor. The Liberals are already over their soul-searching and interim leader. So.

Prentice deserved to lose. The man disrespected the electorate when he resigned his seat just hours after being re-elected. This shows that he does not care for the people. Has that ever happened in Canadian politics before?

Prentice effectively guaranteed an NDP victory with his tactics. Voters wanted change - they'd wanted change for quite a while. In 2012, WR managed to look like crazy people, so voters were scared back into Redford's arms. But in 2015, with no boogeyman, and all of the other parties either infantile (AP), leaderless (Libs), or effectively leaderless (WR), the only option available was the NDP.

All the NDP had to do to win was not be scary. And by scary, I mean like 2012 WR levels of being scary, where they were basically claiming that climate change was part of a communist plot to kill Jesus (or something).

Be less scary than that, and they win.

So they won.

But that split on the right absolutely needs to be taken into account. There were a ton of close three-way races (especially in Calgary). Unite the right, throw away 20% of the support (because merging parties never keeps all of the combined support), and the NDP don't win any seats in Calgary (and they lose several rural seats).

Smith was less keen on silencing the more disruptive elements of her party. She was unwilling to risk alienating people who, frankly, had nowhere else to go. If she had told the creationists that they weren't allowed to deny evolution in public, they weren't going to turn around and vote for someone else. But she still wouldn't do it, and that unwillingness to stand up to the fringes of her party made most Albertans worry that maybe those weren't just the fringes.

The fear is that there are actually lots of Albertans who believe those far-right ideas, and if that's the case then everyone else needs to unite behind a more sensible alternative in order to stop them.

What the post-Smith WR managed to do was look less crazy. Also, it benefitted enormously from the socially progressive vote being split between the PCs and NDP. It won several rural ridings this time which it would have lost had the NDP been weaker (and not siphoning votes from the Tories).

I don't think Smith would have done any better than Jean, particularly if she campaigned the same way she did last time.

Ira - she tried to move the party democratically, which I think is laudable, and politically astute. Her party disagreed, and is in opposition because of it. That being said, defecting to Prentice has go to go down as one of the dumbest political moves in history.

So with the 308 aggregate, my model would have been right in 68 of the 87 ridings, for an efficiency of 78,2%. That's not really good. Of those ridings, 8 were considered coin-flips, 2 were maybes, 2 probably, 3 likely and 4 sure-bets. That's not great either, but since I haven't really tested those margins, it's not too problematic. If I change it to increments of 3% instead of 2%, I'd have missed 9 toss-ups, 3 maybes, 2 probably, 3 likely and 1 sure-bet. That is more acceptable, so I'll use 3% increments from now on for my ranges. The only sure-bet I would miss would be Grande-Prairie-Wapiti which I projected NDP but went PC.

Using the actual numbers though, my model would have projected:

49 NDP21 WR12 PC4 ALP1 AP

That's almost bang-on what happened, if we change some super-incumbents for the ALP to the NDP. But that's on the surface. Because while this number is closer to the results, the ridings themselves fare much worse, with only 63 of them being correctly predicted, for an efficiency of 72,4%. That's, by more than 15%, my worst result (compared to QC 2014 and ON 2014). Looking at the missed ridings brings a bit of comfort though, as I would have missed (using 3% increments as margins of error) 8 toss-ups, 10 maybes, 4 probably, 2 likely and no sure-bets. So again, somehow, more mistakes, but maybe more precise, as I miss ridings that are closer compared to using the aggregate.

I'm a bit confused as to what to do now. Correctly predicting 75% of the ridings isn't acceptable, especially with the actual numbers. My model was almost perfectly projecting numbers in Calgary (compared to Edmonton and the Rest of Alberta), but that is were it made the most mistakes (11 out of the 24). It is also where they were the closest though. It's all very confusing that more precise means more mistakes, so I don't know what to think at the moment.

I'll consider what to do and see what changes I bring to the model. Obviously, once that's done, I'll let everyone here know.

Have you considered a neighbourhood effect, basically a given riding is more likely to vote the same as an adjacent riding, so areas like Edmonton are very strong because all borders are orange, while Calgary is very weak because there is so much diversity of opinion.

As has probably already been mentioned here, and by various pundits elsewhere, the electorate, or a significant portion, is becoming more fluid and likely to change allegiance from one election to the next, and even during the campaign itself. Less of "our family always voted ....so that's what I'll vote". And that is a good thing!

Election campaigns do matter, even more so now, and polling should be done right to the very legal end to try to get it as right as possible.

The UK is much harder to model accurately, at least in part due to multiple regional parties and 12 parties won seats. UKIP had 12.7% of the popular vote, but won 1 seat, while the SNP had 4.8% of the popular vote, but won 56 seats.

Must say Ryan that I'm a bit blown away by the UK results. Polling indicated nothing like this would occur. And now we have both Milliband and Farage resigning as heads of their respective parties !! We should be so lucky here !!

He's built his whole public persona on truthfulness, so he needed to resign because he said he would if he didn't win Thanet South. And given UKIP's tremendous growth in support, and Farage's excellent rhetoric in the European Parliament (where he is still an MEP), I imagine they'll keep him.

I think you are confusing polling and models. This result was in line with the polling data, but the models didn't provide enough probability of a Cameron majority. It did exist, but the reporting tends to focus on the median seat numbers, not the range and probability of various outcomes.

Eric, I'd be curious to know what you think of the performance of 1abvote. I haven't done a lose comparison, but as the campaign went on, it's numbers seemed to track everyone else's, albeit with a slight bias to the NDP and Liberals until near the end.

1) That if you vote Wildrose / PC in this election that you would not vote CPC in the fall.

2) That there were any NDP/Liberal voters that stayed at home,

This really calls into question the federal polls that have the CPC at 45 %. The extra 7% that this poll of 1,481,477 very very likely voters gives to the CPC would provide (I guess) another 5-6 seats in 308's seat projection pushing the most likely total to 150+.and the high end total to 170 or so.

There's no necessary connection between federal and provincial voting intentions - you might be trying to achieve different things at each level. I once knew a guy who voted NDP in BC but Reform federally. He wanted a socialist paradise in BC, but he didn't expect the rest of the country to want that, so he supported Reform hoping to see devolution of power to the provinces.

But even if people do vote for the same thing at all levels, I can see some Alberta PC voters voting for Trudeau's Liberals. I don't undersstand why, but I wouldn't be surprised to see it.

However, I do expect the CPC to get at least 80% of the combined PC and WR vote in Alberta, so I don't think they're in any real danger.

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.

Details on the methodology of the poll aggregation and seat projections are available here and here. Methodology for the forecasting model used during election campaigns is available here.

Projections on this site are subject to the margins of error of the opinion polls included in the model, as well as the unpredictable nature of politics at the riding level. The degree of uncertainty in the projections is also reflected by the projections' high and low ranges, when noted.

ThreeHundredEight.com is a non-partisan site and is committed to reporting on polls responsibly.