Tuesday, October 27, 2015

How the riding polls and riding projections did

One last go around on the 2015 federal election polls and projections before we can put this long campaign to rest.

At the riding level, the projection model was certainly not as close as I would have liked it to have been. Overall, the model called 269 ridings correctly and identified the potential winner (as defined by the parties considered capable of winning the riding by the high and low projections) in 291 ridings. That adds up to an overall accuracy of 79.6% on the calls, and 86.1% for identifying the potential winners.

Where did the model do better? It identified the potential winners in 94% of ridings in Alberta, 91% of ridings in Ontario, and 89% in the Prairies. It performed worse in Atlantic Canada (84% of winners identified), British Columbia (83%), and Quebec (76%).

Not surprisingly, the biggest misses were in terms of the seats that the Liberals ended up winning. The largest group of misses were ridings in which the New Democrats were projected to win, only for the Liberals to pick them up. There were 24 of these ridings, located primarily in Quebec, urban and northern Ontario, and in Atlantic Canada. This is where Liberals unexpectedly defeated New Democrats.

The next largest group were the 19 ridings in which the Conservatives were favoured but the Liberals actually won. These were largely in the Greater Toronto Area, in Vancouver and the Lower Mainland, and in New Brunswick. This is where Liberals unexpectedly defeated Conservatives.

There were seven ridings projected to go NDP that actually went to the Bloc Québécois (mostly north of Montreal), and seven ridings projected to go Conservative that actually went NDP (in the Prairies and the B.C. Interior).

On average, the misses were called with just 65% confidence and the average margin of actual victory in these ridings was about 6.7 points. So, they were modestly close races.

One aspect of the seat projection model worked very well. The assigned probabilities of victory turned out about as expected, though at the lower levels of confidence they were somewhat more confident than they should have been. This is likely due to many instances of three-way races, when the model is designed for two-way contests.

As you can see, the calls were generally as correct as they were expected to be.

At the 50% to 64% level, where the calls performed significantly below expected levels of confidence, Liberal victories were missed in 58% of them, or in 22 ridings. That alone gives an indication of how the Liberals were winning close ridings they were not expected to win. Add those 22 ridings to the final projection of 146 for the Liberals, and you have them knocking on the door of a majority government, rather than apparently coming up well short.

With the actual results plugged into the model, the accuracy of the riding level projection increases to 81.4% (or 275 out of 338 ridings), and to 87.3% (295 ridings) for identifying the potential winners.

As discussed in my analyses of the projection model's performance, we're looking primarily at the Liberals picking up new voters in unexpected places, with strategic voting apparently helping the New Democrats out-perform expectations in Western Canada. The Liberals' vote efficiency in Ontario and elsewhere was also above expectations, though well in line with what the Conservatives were capable of with a similar amount of support in 2011.

Riding polls
Now that we've dissected my performance at the riding level, how about the pollsters?

In the charts below, I've included only the polls done within the last two weeks of the election campaign, and compared the riding-level polling only for the parties that finished in the top three slots on election night. The actual results are in the gray areas, and the date refers to the last day the poll was in the field. Let's start in B.C.

Across the board, you can see that in every riding the riding-level polling under-estimated where the Liberals ended up. In two cases, it turned a third-place showing into a win: Burnaby North–Seymour and Coquitlam–Port Coquitlam. Depending on the riding, either the New Democrats or Conservatives found themselves over-estimated as a result.

The results in Vancouver Granville were particularly interesting, as Mainstreet's final poll came very close, whereas Environics' poll for LeadNow did not. There had been a lot of controversy in the riding due to LeadNow's endorsement of the NDP, despite the edge given to the Liberals in their final poll.

A few riding polls came quite close to the mark, considering the margins of error. For example, in Courtenay–Alberni, Nanaimo–Ladysmith, and South Okanagan–West Kootenay. Not coincidentally, these were NDP-Conservative races in which strategic voting might have kept the Liberal surge at bay.

Now to Alberta.

Again, in Alberta we see the riding polls under-estimate the Liberals significantly in every riding. Unlike in B.C., however, the polls were quite good at gauging Conservative support. Instead, it seems that NDP support collapsed in the final days and went to the Liberals. This appears to be another indication of the role strategic voting played in these races. That especially appears to have been the case in Calgary Centre and Edmonton Centre.

Ontario was slightly different.

In Ontario, the Liberals were under-estimated in most riding polls, but not all of them. In the ridings of Brampton North, Bruce–Grey–Owen Sound, Hamilton West–Ancaster–Dundas, Kanata–Carleton, Kitchener Centre, Perth–Wellington, and Peterborough–Kawartha, the results for the Liberals were within the margin of error of the final riding polls.

This should not come as a surprise. Unlike Alberta, in which the NDP did worse than expected in the popular vote, the NDP's support had largely already collapsed in Ontario well before election day. There was no surge that sunk the NDP's chances at the last moment in Ontario (as there might have been in Quebec). The Liberals were already riding high in the province in the week before the vote.

And in the ones the polls did miss, it wasn't always the same party that took the hit at the expense of the Liberals. In Timmins–James Bay and Nickel Belt it was the NDP, but in Flamborough–Glanbrook, Nepean, Kenora, and Sault Ste. Marie it was the Conservatives.

There were fewer riding polls done in Quebec in the final days, but they did moderately well.

But here again we're looking at the Liberals being under-estimated, and significantly so in Chicoutimi–Le Fjord, one of the most surprising Liberal wins of the night. That vote came primarily from the Conservatives, but also the NDP. In Jonquière and Lac-Saint-Jean, the polls did quite well.

There were only a few polls done in Atlantic Canada as well, but they were generally poor. The Liberals out-performed these polls by nine to 15 points, with the NDP taking the hit where they were most competitive and the Tories taking the hit where they were competitive.

More lessons to be drawn from the discrepancies, then.

Who did best? In terms of average error per party (only the top three) of those riding-level pollsters in the field in the last two weeks, Segma Recherche did the best with an average error of 3.4 points per party. Next was Environics at 4.6 points per party, followed by Insights West and Mainstreet at 5.5 points per party each. MQO Research had an average error of 7.0 points per party, while Oraclepoll had an average error of 7.8 points per party. Of note is the performance of ThinkHQ in Edmonton Centre, off by just 1.7 points per party among the top three.

68 comments:

If I understand your model correctly, it basically takes the previous election results in any given riding as a baseline and then applies the national/provincial polling information change from the previous election to that riding (with various adjustments for in-riding polling, incumbent advantage, etc.). The 2011 election can be viewed as an aberration (it could have been transitional, but the NDP failed to complete that transition; ergo, it is an aberration). Given that, what happens if you apply your model, more or less as is, but use the 2008 election as the baseline rather than 2011? Does it produce more accurate riding results?

I think Ignatieff takes too much of the blame for that. Yes, he was devoid of charisma, but so is Stephen Harper and he did pretty well.

Plot the Liberal results starting with Chrétien's last campaign. Liberal support falls consistently election after election. Given Martin's fall from Chrétien, Dion's performance was predictable. Given Dion's fall from Martin, Ignatieff's performance was predictable.

I think Trudeau succeeded this time for the same reason Layton succeeded in 2011. He's likeable, and all of the other options were bad. If Trudeau can maintain this level of support for the next election, then I think we can credit him. But not yet.

My theory on efficiency is that the higher your share of the popular vote, the more efficient your seat winning becomes. This is because your support goes up more in close races than in ones with forgone conclusions Therefore, 40% of the popular votes equates to the same percentage of seats no matter if it's the NDP, Liberals, or Conservatives getting that support.

In terms of the pre-election projection, the biggest surprise was St. John's East, which was projected with 99% confidence. Sackville-Preston-Chezzetcook was projected at 90% confidence.

With the actual results plugged in, St. John's East drops to 98% confidence. The only other ones at 90% or higher were Kildonan-St. Paul in Manitoba and Edmonton Mill Woods in Alberta. I wouldn't consider either a 'surprise', though, as the Liberals had a good candidate in Kildonan and riding polls were suggesting EMW was competitive.

Imagine the fractured parliament we'd have now if there had been proportional representation. This majority is a good thing. If it turns out to be awful (looks good so far, I must admit), we can replace it then.

There is no need for the government to answer to our concerns between elections. They cannot govern effectively if they do that.

Trudeau has promised something, so rather than a unilateral change to something else, I hope we see something like the process BC undertook. They crafted an alternative to FPTP though a series of citizens' assemblies, and then put that alternative to the people in a binding referendum.

In other words better the system you know than what is proposed. I can only hope that Justin does something sensible although the program he put out calls for change before the next election. I sure don't like any of the options I've seen so far.

57.7% of BC chose the preferred alternative in 2005, and the other 42.3% was split between other types of reform and FPTP, so you can't really say they chose to keep the FPTP system, just that the government set an unrealistic threshold for reform. The 2009 referendum was mostly about poor voter awareness and understanding of the proposal.

I'm for Parliamentary Reform as IRV or STV, and a Senate appointed proportional to provincial party results. However, senate reform is a harder problem to address, needing to get Quebec approval of the Constitution.

Changing the constitution or the way we elect M.P.s or appoint/ elect/ parachute senators for a second time without Quebec's consent would be tantamount to raising a giant middle finger to that Province. Support for separation would gain 10 points overnight.

I was in BC at the time and I agree wholeheartedly with Ira. The referendum was not between BC-STV and FPTP. This idea that the "No" vote encompassed other electoral reform proposals is simply not true and laughable. The BC-STV Campaign implicitly and explicitly campaigned on this fact and their literature was written as a comparison between the two systems!

The logic why such momentous change requires a super-majority I think is pretty straight forward: Such a dramatic constitutional change deserves and should have the support of all regions and most communities of the Province. A 60% threshold helps ensure such general agreement and post-referendum harmony.

As the 2009 referendum rightly demonstrated support for BC-STV was far below the 57.6% of the vote they achieved in 2005.

While I'm not claiming to have been in BC at the time, several of my classmates were. The ballot was FPTP or BC-STV, so anyone who didn't like the idea of larger electoral districts was forced to stay home or vote FPTP.

BC required 60% to change how they vote, but Quebec wants 50%+1 to secede from the confederacy. I agree that the campaigns would have been different, but there was definitely a strong desire.

I don't oppose Québec separation. I like the idea of nations breaking into smaller pieces to give smaller groups of people the government they want, rather than forcing larger groups to share one they don't all like.

This is why I supported the Scottish referendum. This is why I support UKIP. This is why I was in favour of Grexit. This is why I applaud the Faroe Islands, the only jurisdiction to have successfully left the EU.

That said, support for independence in Québec is pretty low right now. If there's a time when a 10 point gain doesn't make much difference, that time is now.

I don't oppose Québec separation, but I don't think it's in their best interests or that of the nation as a whole. I strongly support devolution of powers to lower levels of government so that local accomidations can be made to a national standard. Québec, Scotland, Greece, and the Faroe Islands all benefit financially from drawing more services than they pay for locally. The Faroe Islands were excluded from the EU from the start (1985 accession of Denmark).

How small a group do you support breaking into their own nations? The Faroe Islands have under 50K people, so the West-end of Montreal island should be allowed to be it's own nation as well?

India/Pakistan/Bangladesh is my prime example of separation resulting in problems being magnified by the division, and not being resolved. The main issue is respect and tolerance.

The Faroe Islands were retroactively excluded from the start. They have, officially, never been in the EU. But there was a brief time when they were, before they pointed out that the Kingdom of Denmark lacked the authority to make them join.

"Overall, the model called 269 ridings correctly and identified the potential winner (as defined by the parties considered capable of winning the riding by the high and low projections) in 291 ridings."

In how many ridings did the results fall outside the high/low range for non-winning parties? I'm thinking places like Desnethé—Missinippi—Churchill River.

Liberal support was generally underestimated in this election, but the Liberal numbers were clearly on a steady rise during the last days of the campaign. If we accept that trend as real *and continuing*, we can lay a ruler along the middle of the dots and predict where the numbers should be at least a few days out from any given poll.

If you're looking for Fun With Figures, try projecting support using a linear extrapolation from recent data. The most relevant test is obviously the election itself, but every poll is a test of this approach when compared to the projected numbers based on recent polls. If this approach has any validity (and I'm most emphatically just speculating, not championing), then a key indicator is the change in the derivatives, not just the values.

This may have limited applicability to the post-FPTP next federal election, but there will be many provincial elections to bolster or discount models based on previous polling. The Daily Nanos gives some single-pollster data to start building that model.

And to note the obvious points,(1) This is nonsensical over long time periods (where "long" is likely to be pretty short).(2) Significant events (bozo eruptions and the like) change everything.(3) House effect adjustments are paramount if polls from different sources are used in a short-term prediction.(4) Higher-order curve fitting could also be experimented with. The results will probably be no worse than rolling dice.

There's a lot of talk about strategic voting, but I'm left scratching my head at my riding (North Okanagan - Shuswap), and the one where I used to live in (Cariboo-Prince George). Polls were done in both showing the NDP in either 1st or 2nd with the Liberals behind, and many of my friends were posting all over social media about how we all need to vote NDP to defeat the Conservatives, and yet the Liberals beat the NDP on election night.

Could someone explain this? Does this show that most people aren't strategic voters?

I think the liberal surge in the BC interior was . mostly from Conservatives who would never vote NDP. These were not your typical ABC voters, they were disenfranchised conservatives looking for an alternative.

People are strategic voters at the national level IMO. last election the liberals collapsed due to that. This election, the NDP collapsed due to that.

I got that from water cooler conversation with my colleagues who are not politically inclined. My work is in Markham Ontario, and most my team were past Liberal voters, and in 2011 , almost all of them voted conservative, although their ridings are between conservatives and liberals, where NDP has no chance. This time around I was on a long business trip and I couldn't know how they voted.

Most people aren't strategic voters at the riding level, but may be at the federal level. So with the PLC leading in the vote intentions at the end of the campaign, the strategic voter went for them, notwithstanding what was going on in their own riding.

An alternative hypothesis is that many people *were* strategic voters, but the only message they actually heard was that the Liberals were way up in the polls. So regardless of what was happening locally, they voted Liberal.

I personally like that hypothesis. It's consistent with observed behaviour.

If correct, it suggests that LeadNow may have had much less influence than they like to think.

We saw the Conservatives hold almost all of their votes from 2011 (their total vote declined by about 200K). I blame this on attrition.

We saw the NDP lose a large portion of their votes (their vote declined by over a million). Given their dramatic climb last time, I don't see evidence of strategic voting here. This is more a case of the votes that were parked with them last time finding a proper home.

But the Liberal vote skyrocketed, with them winning more than 4 million more votes than the last election.

What we saw here wasn't votes moving to the Liberals strategically. We saw the Liberals turning a mountain of non-voters into Liberal voters.

Yes Ira I think you have hit the nail. Now the question of why the Libs gained so many "new" voters is the big, interesting one and I think in large part the "Anybody But Harper" vote is the answer. That allied to the Libs new face as leader and a really good, came across, personality.

I don't think that's it. I think Trudeau was really appealing to people who didn't otherwise have a dog in the race. In Trudeau's absence, I don't think those voters find a different ABH candidate. I think they just don't vote.

And then we'd have seen the Tories win 40% of the vote again, with the Libs and NDP getting about 20% each.

I didn't give Trudeau nearly enough credit during the campaign (and before it).

What has been promised is a review of the voting system. At our conventions and most liberal MPS when asked have said they support a move to a preferential ballot, or immediate runoff. That ensures that all votes count, including your 2nd and perhaps 3rd choices, until the candidate in your district receives 50% + 1. Still means direct election of your mp and limits the power of the political party. The problem with perportional rep is that it increases the power of the party. IMHO.

It has it's own weaknesses, but that is party of why I like STV. It doesn't entrench political parties into the electoral system. It just has local elections in superridings that elect more then one person, so that in theory the results are more proportional. (Not as proportional as in other systems, but that isn't the goal of STV. The goal of STV is to ensure that most people have a local representative they actually voted for)

I think that is the biggest problem with STV-IRV: how can a second , third or fourth choice vote equal a first preference? and If one degrade votes if a second third or fourth choice how then are all votes equal? Secondly, if a second choice etc...is counted does that not mean in effect the person's ballot is counted twice? Yes, only the second preference maybe tabulated, but, the process to determine what vote will be tabuilated in effect makes the ballot counted twice since, the first preference must be tabulated then crossed off in order for the second preference to count. STV-IRV is undemocratic in my opinion since, some peoples ballots count more than others!

How can all ballots be "counted equally" when in the same sentence you write: "until part (of the ballot I assume) is accounted for"? Are you not writing about portions?! and then, "Under IRV the second tally has the first place candidates votes counted a second time". So, some ballots are counted more than others! _That is not fair nor democratic! Do you even understand what you advocate?

Éric, you seem to be concerned about your model's performance in the coin-toss ridings, suggesting that, "At the 50% to 64% level ... the calls performed significantly below expected levels of confidence". I think you're being too hard on yourself / your model. I'd say your model correctly identified which ridings were in coin-flip territory and which were weird. The main thing it didn't (couldn't) do is predict that all the coin-flips *and* weird misses would go the same way. And that is exactly why you show your ranges, which also performed very well. I know (based on your previous comments) that when you say "significantly" there, you aren't actually invoking a statistical model, but - factoring in the expected degrees of uncertainty - I would say that your calls performed *exactly at* the stated levels of confidence. Bravo!

Because of the two-party system and the fact that only a small fraction of those 50 states are actually swing states, I'm pretty sure Silver would agree that he has an easier task. He's actually said so in the context of the UK elections.

The devil is in the details. National proportional representation isn't really fitting for Canada. Provincial proportional representation possibly is. For example, using the 2015 results for a provincial based distribution of seats:

The main difference as I see it is a government's investment in pleasing voters in every province. The NEP basically amounted to the LPC writing off Alberta and the Prairies in their electoral foundation.

I wasn't putting forward provincial PR forward as a preferred alternative. I was just saying that it would allow for greater regional representation and to avoid "alienate idiosyncratic regions". They were easily available numbers in order to run a quick comparison.

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.

Details on the methodology of the poll aggregation and seat projections are available here and here. Methodology for the forecasting model used during election campaigns is available here.

Projections on this site are subject to the margins of error of the opinion polls included in the model, as well as the unpredictable nature of politics at the riding level. The degree of uncertainty in the projections is also reflected by the projections' high and low ranges, when noted.

ThreeHundredEight.com is a non-partisan site and is committed to reporting on polls responsibly.