Thursday, October 22, 2015

How the polls and the projections did

The 2015 federal election was a success for the polling industry, with the results of the election within the margins of error for most of the pollsters. The results were certainly better than in 2011, when the polls were not pointing to a Conservative majority government. The polls in this campaign did not seem to be pointing towards a Liberal majority government either, but that was because the seat projections were bamboozled by a growing and changing Liberal voting base.

There was a late surge for the Liberals that pushed them over the top, but some of the polls that had older data (even only a few days old) were not able to capture the wave that pushed them from between 35% and 38% to the 39.5% the party actually got. For that reason, the vote projection was also lagging a little bit. Nevertheless, it still painted an accurate picture of what was going on, particularly when considering the likely ranges.

The seat projection model did not perform up to my expectations. When the results were plugged into it, it still projected a Liberal minority - though the chances of a majority government were higher.

The Liberal result was far more comfortably within the high-to-maximum range, but was still out of the likely range of outcomes. The Conservative result falls further out of the likely range than it did initially, though the NDP's projection is improved and the Bloc's projection hardly changes.

The regional projections edge closer to the mark, but you can still see the problems the model had.

In British Columbia, the Conservatives were greatly over-estimated, with both the Liberals and NDP under-estimated as a result. Alberta would have been accurate, but the NDP would have been under-estimated in the Prairies to the benefit of the Conservatives.

In Ontario, the Conservatives would have been over-estimated at the expense of the Liberals. In Quebec, the Bloc and Conservatives would have been accurately pegged, but the New Democrats greatly over-estimated, at the expense of the Liberals again. And in Atlantic Canada, the Liberal sweep is still unimaginable - even at the maximum ranges. It is the only region of the country where the maximum ranges would have still not been enough, demonstrating how unusual it was that the Conservatives and New Democrats lost the ridings that they did.

I have not yet gone through the riding-by-riding results, but will in the coming days.

The performance of the polls

The polls did a very good job, with Nanos Research and Forum Research taking the cake both at the national and regional levels.

The list below includes only those polls conducted and publicly released in the week before Election Day. Abacus Data and Innovative Research did not release a final poll in this period.

Nanos and Forum finished at the top of the table, the biggest mark against them being the under-estimation of the Conservative vote. Nevertheless, for all of the parties neither missed the mark by more than 1.9 points.

(This is looking at Nanos's one-day result, though the three-day result was also relatively close to the mark.)

Ipsos Reid, Mainstreet Research, and Léger were all quite close as well, putting the Liberals at 38%, the Conservatives between 30% and 33%, and the New Democrats between 21% and 22%. That slight over-estimation of the NDP vote was behind the under-estimation of the Liberals.

EKOS had less total error than Léger, but Léger told a story that was in my view closer to the reality in that it gave the Liberals an eight-point lead over the Conservatives, suggesting a majority was possible (the Liberals won by 7.6 points). EKOS's poll, which put that gap at just 3.9 points, was the biggest source of doubt in the Liberals' ability to get close to that majority mark the night before the election. The Angus Reid Institute also contributed to that doubt, and was at the bottom of the table.

Note that, as its post-election release mentions, Mainstreet Research did indeed send me their topline numbers before the votes were counted (and, as the report says, the data was also sent to Postmedia). That result would put them in the top three with Nanos and Forum. But the numbers were not published before the election results were known and I imagine they would not have been published had they been off the mark (Quito Maggi of Mainstreet says in a comment below that a commitment had been made to Postmedia to publish the results of the poll regardless of the election result). It would not have been fair of me to judge Mainstreet based on its unpublished numbers had they been wrong, so it isn't fair of the other pollsters to judge Mainstreet by its unpublished numbers because they were right.

But it should be noted that Mainstreet's final polls of the campaign (published before the date or not) were quite close to the mark.

Also note that, as Nanos did not publish any regional numbers for its one-day sample, I have graded them below according to their three-day sample. And as they combined their Alberta and Prairie results, I have left them out of the rankings for those regions.

The polls were quite good in British Columbia, though the ranges were wide. But there was no systemic over- or under-estimation of any of the parties, and in fact nearly every pollster was almost exactly on the mark for the Conservatives and NDP. Insights West had the best result by a wide margin.

There was a systemic under-estimation of the Conservatives in Alberta, which seemed to benefit the NDP more than it did the Liberals, who were well-gauged. Angus Reid, Ipsos Reid, Mainstreet, and Forum did well here.

Considering the small sample sizes, the polls did very well in the Prairies, with the results falling well within the ranges of the polls done in the region. Léger had the best results.

Ontario was very well polled. All the pollsters had the Liberals between 43% and 46%, when the party took just under 45%. They under-estimated the Conservatives across the board, however, and in most cases it was the NDP that got the extra points. Nanos and Forum did well here.

Quebec was the miss of the election. Only Forum had the Liberals anywhere near the 35.7% the party took, with all the other pollsters under-estimating the Liberals by a significant mount. The NDP was better gauged by most pollsters, though some over-estimated them. The Conservatives were mostly over-estimated, while the Bloc was generally well polled. Nanos and Forum did the best here.

And in Atlantic Canada, the polls either got close to the Liberal result or under-estimated them. With few exceptions, the polls over-estimated the NDP and Conservatives across the board.

On the whole, though, the polls did a very good job. The only real error was in Quebec, and that was enough to make the difference between the 38% the Liberals were pegged at nationally in half the polls published in the last week and the 39.5% the party actually took on election night. That is a performance the pollsters can be proud of.

I'll have more on how the seat projections performed at the riding level soon. That should be illuminating in why the projections missed the mark.

51 comments:

The Signal by the star seems to have done much better with their seat projection giving the liberals 160 seats with just 36.7%. If they punched in the weekends polls only they probably would have hit the mark. What did they do differently?

Eric, the release mentions that we sent you the numbers, but it also says that we had 100% committed to our media partners to release the final report. You can confirm with David Akin that not only did we provide a topline number to Postmedia, but a regional breakdown and seat projection at 177 for the Liberals, 113 for the Conservatives and 45 for the NDP. The report would have been published regardless, it is unfair to say that it would not have been. Our majority prediction was already based on the numbers released on Saturday which showed increased leads in Ontario and a Liberal lead in Quebec. We were the only pollster to predict the majority and it was based on our data as explained on numerous media interviews and podcasts that are part of the public sphere. The prediction was clear, the basis of it was clear, the numbers were part of the public beyond the email to just yourself.

Eric, many "strategic" voting resources cited your riding projections as the basis for their recommendations throughout the campaign. I suspect many voters wrongly misinterpreted your projections this election, as they have in previous elections, as being based on actual, on-the-ground polling being done in each riding, despite the occasional disclaimers to the contrary.

Ancedotally, I found myself repeatedly having to remind friends your riding projections, and the strategic voting guides using your projections, were not based on actual riding-specific data. I suspect many more voters similarly cast their ballots with your projections, one way or another, in mind.

Furthermore, I think this misconception particularly benefited the Liberals in otherwise strong NDP urban ridings, where your algorithm may have given the Liberals a boost. I'm thinking of star MPs like Andrew Cash, Megan Leslie, Paul Dewar, etc., all of whom were otherwise considered safe seats when factoring in their incumbency and well-resourced campaigns, but when projecting based only national and regional polling data, were behind the Liberals.

A great example of this happened in University-Rosedale. Where actual in-riding polling during the campaign showed Jennifer Hollett with the lead, your riding projections over the same time period had Chrystia Freeland ahead. I know for certain the Liberal campaign used your numbers as evidence when talking to voters about who had the best chance of winning (and beating the Conservatives, who realistically didn't stand a chance in that riding anyway), even though the real, on the ground polling data at the time refuted this. I think this talking point was an extremely important factor in the minds of voters, who were perhaps otherwise uninformed about the election (and the necessity of strategically voting to stop the Conservatives from winning).

So considering that "strategic" voting against the Conservatives very well may have been an important factor in helping them surge to a majority at the last minute, do you feel responsibility for the impact your riding projections - and the strategic voting guides that used them - might have had on how voters cast their ballots? I would contend that your projections may have directly impacted the election's result in a significant way.

More broadly on that point, wouldn't you say the horse-race nature of the news media's coverage of this election, in particular CBC's, may have also been a factor in the Liberal majority win?

Personally, I would really like to see a post-mortem poll on this election that really examines the impact of "strategic" voting choices. Sadly, in the eurphoria of the Liberal's win, I think this has been greatly overlooked in the rush to praise the Liberal campaign's "efficiency". I'm always disappointed to see this kind of empty conjecture used by pundits in the place of actual data after an election.

I don't think the Liberals are being described as efficient. I think they're vote is being described as efficient, which it demonstrably was. It turned votes into seats much more efficiently than in recent elections.

Not how much more efficient the Green vote got when they stopped campaigning nationally, and instead focused on a handful of winnable ridings. Suddenly they elected someone.

But claims of efficiency aren't praise - they're just descriptions. The CPC has historically been quite efficient because of their targeted campaign strategy (which became a targeted governance strategy), and I don't know anyone who thinks that was a good thing.

I think you're vastly overevaluating how much attention was paid to actual, projection-based strategic voting. For most people, strategic voting meant voting for the liberals since the NDP was so far behind.

And people really did like the Liberals campaign. The NDP campaign felt empty to me and didn't appeal to me at all. The fact that they released their platform so late in the campaign only played against them.

Furthermore, based on the 308 projections, Dewar was winning until the last few days. Same with Nycole Turmel and Francoise Boivin, if we're talking National Capital Region. The projections greatly favored the incumbent, but people preferred to vote Liberals.

Well when polling was showing a nearly 2:1 ratio of Liberals to NDP, the tactical vote option to ditch Conservatives was clear. It seems that Liberals really routed the NDP on their left flank by saying Mulcair signed up to balanced budgets, so that would disproportionately affect a demographic of NDP strength, "star candidates" or no. Kind of like Kathleen Wynne's path to victory spanning the centre and left. Also, local polls are really not reliable (see the UK general election).

Well, I think it is easy to think it had impact but in reality I am not entirely convinced.

My own riding showed the liberals at about 15% chance, with the ndp in the higher 30% as the closest party to beat the CPC, who led the projections.

In the end that split the vote - those who understood strategic voting at a riding level (plus the actual ndp people) voted 30% ndp - and another 30% voted for the liberals - letting the conservative get the seat with 35% of the actual vote.

On the basis of that it is tempting to suggest that the effect of strategic voting and these projections had an effect, where the ndp was over estimated and the liberal really underestimated...but....

But in other strong ndp or cpc ridings that I was watching, the liberals did win - BUT that surge was actually reflected in the projections. Why not here, right? The answer has to be that some other factor had to have been in play.

It is because of that I am really hesitant to say that using the projections as a basis for a strategic vote helped the liberals. IF true, it hurt them here - but the truth is more likely to be that there is just a strong NDP base here, which was accurately reflected, and those people were never going to vote LPC no matter how strategic.

I am curious about which polls were the worst outliers to the polling "conventional wisdom" along the way. I noted that Mainstreet tried to shape their sampling to better conform to the recent voter turnout profile (much heavier in seniors than the general population) which would have served them well had the profile held. Forum seemed to flag the NDP or Liberals strongly depending on the campaign stage compared to the other polls. Leger noted how much they had to upweight their polling outside Québec to get the numbers proportional, while substantively reducing the weight of their Québec sample.

I noticed on my landline phone a few calls from pollsters, but I work evenings. I have taken part in surveys, but I fear that the available response group is just getting to small to not be able to guarantee a representative population from which to choose. My landline seems to only get calls from telemarketers and pollsters, with the occasional politician, leaving me wondering why I pay for it for them to contact me. Anyone else in favour of making the telemarketers and the like pay for everyone's phones if they are going to call us?

Éric, Are you giving more weight to polls that are more recent? It seems that the main disparity between your projections and the actual results are simply because the projections are based on older polls.

But you can also read above that he plugged in the actual voting intentions and still the seat projection methodology model suggested a Liberal minority.

When you build a projection model, you do your best to minimize the possible error range. This leads to conservative (small c!) results that mean your outlier results often get "smoothed away".

Take Peter Stoffer's riding: an incumbent popular MP, won his riding with more than 54% of the votes and nearly twice the number of votes as his runner up in 2011. Further, if you believe people voted ABC strategically, then he was the obvious vote (the Liberal candidate came in 3rd with only 11% of the vote in 2011). And yet...

You can see the difficulty in taking what by all accounts should be a "safe" NDP riding and crafting a model that will predict it to flip Liberal. When you are crafting such a model, the math will bias away such a result as it maximizes the chance for error.

So perhaps the conclusion is that it may very well be a losing game to do riding level projections with the current available polls. We'll see what sort of electoral reform we get from the Trudeau government, but my guess is that it's not going to make the task of projecting seats any easier.

I take your point with Peter Stoffer and a conservative projection model, but, a straight forward swing model would have predicted Stoffer would lose. That is the (almost) inevitable result of the Liberals at or near 50% in Atlantic Canada and the Tories and Dippers far behind.

The only prediction I got right last Monday was the Liberal sweep of the Atlantic, based on the fact that the Liberals won 31 of 32 seats in 1993 with a smaller lead in the polls. 1993 was a major realignment and the ideal seat projector would consider the swings that have happened back and forth since then to predict voter behaviour.

I suspect if you used a logic regression model your seat totals would have performed better, particularly in Atlantic Canada. It would reflect the fact that it's harder to move from 80 to 81% of the vote than it is to move from 40 to 41%.

One thing that I found interesting was that this election was all about voter turnout. If you look at the real numbers, and not the percentages, it is easy to assume that if the left-leaning people stayed home, the Conservatives could have won again. Conservatives retained 96% of their 2011 real vote numbers, the Bloc 92%, the NDP 77%. Even if you assume all the votes the Conservatives and Bloc lost went *all* Liberals, or *all* NDP, the Conservatives would have won. Even on top of that adding in 61.4% (2011 turnout) of the nearly 1.5 million new eligible voters all going the same way, I only see a close call, perhaps a stalemate with the popular vote, between the Conservatives and whichever left-leaning party got all the extra votes. While the Greens got almost 30,000 more votes, the Liberals essentially got all the votes the Conservatives, the Bloc, and the NDP lost, plus the 2.7 million additional votes cast in this election! Would some more of the other parties' supporters stayed home if the overall turnout was lower? That seems unlikely. The Conservatives are very good at getting out their vote, looks like they have hit near their rock-bottom, and they are still they official opposition.

The CPC path to victory was always low voter turnout. It worked brillaintly in 2008 and 2011, and they tried it again in 2015.

As you pointed out, they retained almost all of their vote. They were defeated by a influx of new voters.

What the CPC needs to do now is grow. They're in a good position to do that, and I expect the 2019 election to be competitive unless Trudeau turns out to be a genuinely good PM (he looks pretty good so far, I must admit). I look forward to he showdown between Justin Trudeau and Maxime Bernier (my choice for CPC leader).

Liking at your analysis, the low standards Canadian pollsters have compared to other nations is very evident. Famously, Gallup announced they won't he polling this election because they overestimated Romney in 2012 by a fee points and underestimated the Republicans in the Congress by a few more. Such errors would be considered a huge win by Canadian pollsters!

Even the UK, which, like Canada, is harder than the US to poll due to the multi-party system, lack of party voter registration, and shallower partisan loyalties, the 2015 debacle, where a Tory majority was not seen and the Conservatives were underestimated and Labour overestimated the British polling Council was compelled to conduct an investigation into their errors. Their mistakes were no worse than Canadian pollsters this election, yet ous are patting themselves on the back for simply not doing as hilariously bad as they did in Alberta in 2012, BC in 2013, and Ontario in 2014.

Given how much pollsters seem to be influencing some elections, maybe they owe it to our democracy to at least be somewhat accurate?

I think fault should reside with those who wrongly interpreted the numbers. IMO the numbers or at least the polls before becoming aggregated, did predict a Liberal majority, the fault lies with the punditry and those who blindly follow them, for not interpreting the data correctly.

Canadian elections are much more dynamic than American elections. In the US, there are six shifts possible (Dem to Rep, Dem to Other/None, Rep to Dem, Rep to Other/None, Other/None to Dem, Other to Rep), while in Canada there are 720 combinations. That means much more uncertainty in polling trends.

The UK Conservative majority was also predictable by the polls, It was the punditry who misinterpreted the results and in particular failed to model Scotland correctly, making it appear Labour had a chance to form government when in reality the chance was slight.

To be clear, the British Polling Council initiated an external inquiry, they were not compelled. The largest miss in this election was 1.9%, whereas in the UK the smallest Conservative miss was 1.8%. The UK had an average error of 1.83%. SNP were placed under "Other".

We'll have to wait until March (January for preliminary results) to see if the polls were appropriate to their task. Scotland is the Quebec of the UK. It explains a lower seat count for Labour, but it doesn't explain the under counting of the Conservatives.

If it were that simple, then it wouldn't be that big of an issue. However, if you look at the England & Wales Only numbers, Labour was still over estimated. Looking at ComRes specifically (best of the May 5/6 released polls), their average error for all UK was 1.32% and for England/Wales it was 1.32%.

It's not that big an issue except among those who voted for the Losers. The losers like to throw tantrums, it explains the election of Jeremy Corbyn as Labour leader, a man who by his own admission has accepted funds from known supporters of terrorism and who is himself a supporter of terrorism (though not terrorist acts specifically) through his close connexion with the IRA among other sociological malcontents.

What you fail to understand Mapleson, is it is not about the margin of error-politics is about people and momentum more than margin of error and polling councils. For months before the election Labour was unable to break and sustain the 35% threshold, the UK like Canada, America, New Zealand are small "c" conservative countiries so the failure of Labour to break 35% says all anyone needs to know about where momentum was in that campaign!

The issue is in the accuracy and reliability of polling methodology. Just like Canadian polling firms did a self check-up after the provincial misses in the last few years, which resulted in better methodology (specifically, not presenting a "likely voter" model as the polling numbers, as the "unadjusted" model was actually closer. The UK is currently completely their own examination.

Jeremy Corbyn has nothing to do with the polling results, you're just trying to obsure the issue. What you fail to understand Ghost, the margin of error is about people and momentum and understanding that polls are trying to capture an accurate representation of where they are and where they are going. The UK is a politically divided nation, like Canada, and unlike the US. There is a political spectrum that runs beyond left-right, and the collapse of the UKIP support and it's general inefficiency are as large a factor from a few months before as any positive gains possible by Labour.

Actually, my compelling argument is that the law was made as 12 when there were only 285 seats. Official Party Status should have been adjusted with the new seats and now be 17. Alternatively, as it is very hard for small parties to win seats, instead of 5% of seats won, it could be 5% national popular vote.

Reform ran 72 (BC, AB, MB) seats in 1988, received 2.1% of the national vote, and won 0 seats. A better example is the CCF in 1958 who won 8 seats of 265, but had 9.5% of the popular vote or the PC in 1993 who won 2 seats of 295, but had 16% of the popular vote.

I would use the national numbers as an expanded definition of an Official Party because whatever seats they have represents a significant proportion of Canadians.

Ira - you can have have 100 local elections and get fairer results without having people vote for parties.

Popular vote is a useful metric though still. Look at it this way: how many voters are represented in the government caucus? When they have the confidence of the House, out of those 338 local elections, how many voters gave their support to the candidates offering confidence and supply to the Prime Minister?

In both this election and the 2011 one, the government has a "majority" mandate from just 30% of voters. Another 10% voted for candidates that support the government but didn't win. Another 60% of voters picked candidates who do not support the government. I agree with a candidate centric view of our democracy, but your argument doesn't address the very real issues with the way our electoral system works.

Yes, new parties do arise and win seats, and we've seen more and more parties do that over time. Our electoral system still artificially suppresses that, restricting voter choice, and ultimately allowing our existing parties to be complacent due to a lack of competition.

The people can vote for whatever they like. They can vote for candidates or they can vote for parties. But what's important is that each individual voters gets to decide that.

The electorate is not a hive-mind.

It doesn't matter how many voters are represented in caucus. Every single voter is represented in the votes in the House. A majority of voters are represented by the candidate they actually voted for.

I want new parties suppressed. I want new parties to have to displace old parties in order to win seats. Because I think majority governments are good for the country.

I'm a fairly die-hard Conservative. I was even employed by the Reform Party, both part-time as a student and full-time as an adult. And I'm an Albertan who won't even consider voting Liberal until they apologize for the NEP. But I prefer this Trudeau majority to any possible minority government.

If the interior of British Columbia holds a unique set of opinions, and they elect representatives who advance that set of opinions, that keeps that perspective in the national debate while creating a targetable voting block for parties seeking greater national support.

If you're too weird (rural Alberta, the north, francophone Québec), then the national parties don't pursue you, as that would be detrimental to their national goals.

This feedback loop helps tone down really radical ideas, while still giving them a chance to make a persuasive argument in the public theatre. We saw this very thing play out in the 1990s during the Chrétien government. Québec argued its nationalist case (by electing the Bloc), and was largely ignored. The west argued its fiscally conservative case (by electing Reformers), and moved the public debate.

And if one party gains support from enough parts of the country to win a majority, then I want that party to have free reign to implement bold policy, without having to worry about losing support all the time.

I want to ensure that the MPs have enough time to do their jobs before they have to start worrying about keeping their jobs. Because those two goals are largely incompatible.

I think we have the same goals, but the opposite approaches to the issue.

While the Bloc's primary goal is nationalism, they have many other good criticisms. Likewise, while the Green Party's primary goal is environmentalism, they have many other sound policies.

In 2004, with a Liberal minority, Harper talked about a CPC-BQ-NDP coalition (CPC-BQ were 2 seats short of a majority). In 2006 and 2008, Harper had to listen to other positions to continue his mandate.

In 2011, we started seeing the more radical parts of the CPC emerge. C-24 and C-51 were primarily the children on unmitigated policy. Specifically, C-24 was an attack on the fundamental right to be Canadian, and the fundamental equality of all Canadian citizens.

If we can break our majority addiction, then we can have more functional minority governments, including possible coalitions.

Minority governments can fall. The risk of falling changes how they govern.

Ideally, I'd like to see a dramatic reduction in the size of the Canadian government so that the provinces could then differentiate themselves more fully. If there's an intractable cultural divide between Alberta and Nova Scotia, I would like to minimize the degree to which they share a government.

New Zealand has had two minority governments though 1996 to 2015 including 7 elections (on a three-year cycle. In Canada, our minorities have been unstable because the opposition parties believe they can gain majority control for themselves. If either that belief were to change or a hung parliament enforced until the next statutory election, we'd have more stability and more willingness to advance the agenda on shared platform issues.

The size and role of the collective government is always an issue. I’ve always supported a hybrid concept of overarching standards and minimums to be supplemented by local contributions where desired. There are a few programs that should be run at a national level or higher: defense/security, immigration, environmental protection, and justice. Beyond the obvious departments, government should stop trying to meddle where it isn’t effective.

New Zealand has not had a majority government since MMP was introduced for the 1996 election-Every election since then has resulted in a minority government. it is not really correct to say there have been only two minority governments, if anything there have been three because Jenny Shipley served as PM for nearly a full term between 1997-1999.

Sorry, there's a misunderstanding of my use of 'government', that being the term of a party in power. Canada had a 10-year Conservative government split half and half majority and minority, but still the same government.

You are correct about Shipley. My thesis of the ability to have stable minorities remains as Shipley was an extension of the previous Jim Bolger National government.

"The Government" is a term for the party in power. You wrote: "two minority governments" which if anything is a poor pseudonym for "The Ministry(ies)" of which there have been eight since the introduction of MMP.

I will point out that your idea to force a hung Parliament to fulfill its term is undemocratic and borders on tyrannical in our system of Government. The Ministry is responsible to the House. If the Ministry can not pass supply it must resign. Your proposals would allow the unelected executive (The Crown) and the defeated Ministry to continue to govern after they have lost the confidence of the House, making patronage appointments, issuing Orders-in-Councils, implementing regulation for potentially years after they have lost the confidence of the House. It is one of the more idiotic ideas I have come across.

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.

Details on the methodology of the poll aggregation and seat projections are available here and here. Methodology for the forecasting model used during election campaigns is available here.

Projections on this site are subject to the margins of error of the opinion polls included in the model, as well as the unpredictable nature of politics at the riding level. The degree of uncertainty in the projections is also reflected by the projections' high and low ranges, when noted.

ThreeHundredEight.com is a non-partisan site and is committed to reporting on polls responsibly.