Thursday, May 16, 2013

B.C. post-mortem, polling methodologies, and where to go from here

Note that this post was written before the final count that was concluded on May 29, 2013. The final count changed the results slightly, with the Liberals winning 49 seats and 44.1% of the vote and the New Democrats taking 34 seats and 39.7% of the vote. The riding which flipped over to the New Democrats was Coquitlam-Maillardville, which the projection had originally forecast for the NDP. That increased the accuracy to 82.4%, or 70 out of 85 ridings. The Electoral Track Record has been updated to reflect the final counts, but the post below has not been.

Now that the dust has settled a little and those in the polling industry (along with myself) have had some time to reflect on Tuesday's results in British Columbia, it is time to take a look at how the projection model performed. But I'd also like to discuss the methodological debate in Canadian polling, how this site has approached it, and the future of this site within the context of a plummeting faith in polling.

The model did about as well as it could considering how different the election's results were to the final polls of the campaign. The model is not capable of second-guessing the polls to the extent that it could have predicted an eight-point NDP lead turning into a five-point Liberal win.

The forecast ranges were included to try to estimate how badly the polls could do if another Alberta-like scenario played out, and aside from the NDP falling two points below the forecasted low they were able to capture all of the vote and seat results at the provincial level. They were not, however, able to capture the performance of the Liberals and New Democrats in metropolitan Vancouver and in the Interior/North, demonstrating just how unbelievably well the Liberals did in these two parts of the province. Their vote came out in huge numbers here (and/or the NDP's stayed home), and the Liberals won the election.

Of course, the forecast ranges are somewhat absurdly wide. But that is more of a reflection of how unpredictable elections have become in Canada. They are absurdly wide, and yet still needed.

The parties did about as well as expected on Vancouver Island, however. If turnout was one of the factors in explaining why the polls missed the call, the Liberal ground game did its work in the rest of the province, while Vancouver Island was left to the NDP.

In all, the seat projection made the right call in 69 of 85 ridings for an accuracy rating of 81.2%, while the potential winner was correctly identified (by way of the projection ranges) in 73 of 85 ridings, for a rating of 85.9%. This shows how the election was really won in just 12 ridings, as the projection ranges (which did not consider a Liberal victory likely) only missed those 12.

Metropolitan Vancouver was where the election was primarily won. The projection gave the NDP between 45% and 51% of the vote and the Liberals between 36% and 41%. Instead, the Liberals took 46.1% of the vote in the region (as this site defines it) to only 40.4% to the NDP. The Liberals won 24 of the 40 ridings, instead of the 14-16 they were expected to win.

The Interior/North was also a major factor in the Liberals' victory. They were expected to win the region with between 38% to 45% of the vote, narrowly beating the NDP out at between 37% and 45%. This gave the Liberals between 12 and 22 seats and the NDP between 9 and 16. Instead, the Liberals won 24 seats with 48.2% of the vote, while the NDP won only 7 seats with 35.4% of the vote.

On Vancouver Island, the NDP won 11 seats, the Liberals two, and the Greens one. The projection did not give the Greens any seats, but expected 11 to 14 for the NDP and 0-3 for the Liberals. The NDP was expected to take between 44% and 53% of the vote, the Liberals between 27% and 35%, and the Greens between 10% and 17%. The NDP actually took 43.9% to 34.2% for the Liberals and 17.2% for the Greens. It would seem that some of the Conservative vote (they took 4%) went to the Liberals and some of the NDP vote went to the Greens, but overall the island played out mostly as forecast.

As usual, the seat projection model was not at fault. If the polls had been accurate, the model would have projected 49 seats for the B.C. Liberals and 36 for the B.C. New Democrats, mirroring the result closely. The ranges would have been 37 to 57 seats for the Liberals and 27 to 46 for the NDP, while up to one Green would have been projected and two independents.

The right call would have been made in 76 of 85 ridings, for an accuracy rating of 89.4%, while the potential winner would have been correctly identified in 81 of 85 ridings, for a rating of 95.3%. The challenge remains getting the vote totals closer to the mark. Frustratingly, that is the one thing I have the least control over.

How the projection model would have been wrong in a few individual ridings is interesting, and reflects just how important local campaigning can be. Three of the incorrect nine ridings (with the actual regional vote results) included Delta South, Vancouver-Point Grey, and Oak Bay-Gordon Head. The model would never have been able to boost Andrew Weaver's support enough to give him the win without some improper fiddling with it on my end. In Delta South, Vicki Huntingdon's support was stronger than would have been expected. And most significantly, Christy Clark's rejection in her own riding is all the more starkly shown. She did not lose it because the overall race was close - the overall swing should have kept the riding in her hands.

Polling methodology and what went wrong
All eyes have turned to how the pollsters are doing their work. Some of the pollsters are looking at their methods and trying to figure out what went wrong and what can be done to avoid these issues in the future. Others are crowing that this or that poll they did a week before the election turned out to be prescient, and it appears that some lessons will not be learned.

A hypothesis does seem to be forming as to what happened. I'd identify a few factors:

Turnout - Turnout was only about 52% in this election, and that can throw off a pollster's numbers to a large degree. However, turnout was also very low in the 2009 election and the polls did a decent job that time. Turnout is not a silver bullet, then, but the effect turnout had in 2009 may not have been the same as in 2013.

Motivation - According to Ipsos-Reid's exit poll (which I will return to in the future), very few British Columbians thought the Liberals would win a majority government (only about one-in-ten), while one-half thought the New Democrats would win. This might have depressed turnout even more, with some New Democrats not bothering to vote since they felt they would win, and some Liberals turning out in greater numbers to ensure their local MLA would get re-elected, even if the party itself would be booted out of government. Conceivably, though, Liberals not bothering to vote for a lost cause should have cancelled things out. And in most cases, people tend to vote in greater numbers for a perceived winner.

Election Day Shift - Yes, it is unbelievable that the polls were right all along and a dramatic change of heart occurred in the final hours. But Ipsos-Reid's poll showed that 9% of Liberal voters made-up their minds in the voting booth. If all of those voters had instead voted for a different party, the Liberals would have been reduced to about 40%. That would have been closer to most polls, but still much higher than even the margin of error would have considered possible. And, of course, some of those 9% might have just been wavering Liberals who did not make up their mind until the last minute, but had told pollsters they were still intending to vote Liberal. While certainly part of the equation, it cannot be all of it.

Bad polling - This is probably the main reason why the polls missed out on the call. The other three factors may have been worth a few points each, but there does seem to have been a problem in building a representative sample. Pollsters will need to figure out why that is.

One of the problems that has been identified most often (especially by those pollsters who use other methods) is that most of the polls used online panels. These have had success in the past, including the 2009 B.C. election, but perhaps online panels are less able to consistent give good results - particularly in provincial campaigns where the panel may be smaller. But this cannot be the only reason, as Angus-Reid's online polling in Manitoba - a province with a quarter of the population of British Columbia - was stellar in its 2011 provincial election.

Nevertheless, the track record of online firms has taken a hit. Telephone polls using live-callers still seem to have the most success. Reaching people by telephone - including mobile phones - probably remains the best way to do quality polling. It is also a good way to do expensive polling.

Is the extra accuracy worth the extra cost? That might not be the case when it comes to market research. Whether it is 36% or 44% of people who say they have trust in your company's brand is not vital information, as long as it is in the ballpark. Even at their worst, online polls have been in the ballpark (the Liberals and NDP were not polled to win the election in Alberta, and nor were the Greens or Conservatives ever pegged to have more than marginal support). But in an election, the quality of a poll, and not the cost, should be the deciding factor in whether or not to report it.

The chart below reveals some information that I have up to now kept to myself. Pollsters are rated in my weighting system by their track record. That track record extends back over 10 years, with more recent elections being more heavily weighted. The difference between one firm and the next is usually not very large, and some of the difference is due to the elections in which these firms have decided to take part. Those that stayed out of Alberta and B.C. are inevitably going to have better ratings than those who didn't. I have considered overhauling the rating system to take into account these sorts of considerations, but I have not yet done so. Because I haven't, I am reluctant to actually rank the polling firms publicly by name.

But I am willing to rank them by methodology. These are the 10 firms in Canada I consider to be major firms, and the method they have used in their most recent election campaign. They are polling firms that release national, regional, or provincial polls on a regular basis. The chart shows each firm's average error per party in any election in which they were active, going back ten years.

As you can immediately see, the polling firms that conduct their surveys using live-callers occupy the top three ranks. The online and IVR polling firms have had less success. The difference is not huge, however - on average, the third best firm misses the call by fewer than 0.5 percentage points per party than the seventh best.

However, it is clear that polls conducted over the telephone with live-callers have had a better track record. That does not mean that they will always have a better result: in the 2011 federal election, Angus-Reid's online panel had the lowest per-party error. But it does suggest that the online panels still have some work to do.

Where to go from here
There were moments yesterday when I contemplated the end of ThreeHundredEight. Why run a site about polling when polling in Canada is so horrid?

But the polling is not always horrid, and even when it seems to be on the bad side there are some indications of something else at play. Alberta is an obvious example, but maybe British Columbia's errors have some mitigating factors as well.

Even if that is not the case - and I am not convinced that it is - polls are not going away and I still believe that they are a useful tool. The electorate deserves to know what the collective wisdom of the people is on various issues, including on the question of who should govern them. But the electorate deserves good, reliable information. Bad information is much worse than none at all, but polls are not going to disappear.

Though I could never claim to be impartial on the question of whether polls should be paid any attention at all (if they are ignored, I would need to find a new line of work), I can continue to be an impartial observer, analyst, and (when need be) critic of the industry. In its own tiny little way, ThreeHundredEight can be part of the solution.

That means more of a focus on methodological and transparency issues, sweeping trends, uncertainties in the polling data, and wider questions about what the numbers mean, if anything at all. It means less focus on the horserace, more caution in reporting numbers, a forecasting model that emphasizes what we don't know, and more reserve in giving attention to questionable polls. And when a poll is questionable, drawing attention to the reasons why.

It might mean a drop in traffic and it will certainly require more work and effort on my end. And like all junkies, I might relapse. But I think it will be a worthwhile endeavour. I welcome your thoughts in the comments section.

45 comments:

Polls have been failing to accurately predict election outcomes of late, but at the very least I think they reliably reveal trends, which is definitely useful information. The BC polls at least correctly detected the narrowing gap between the Liberals and NDP. They correctly detected the rise of the CAQ during last year's Québec election. They correctly detected the "orange crush" in 2011. I don't doubt they correctly documented the slow descent of the NDP after Layton's death and their temporary resurgence after Mulcair took over, nor do I doubt that there's really a surge in support for the federal Liberals right now while CPC support is near its lowest levels since they came to power. Would the current numbers be validated if an election were held tomorrow, or a month from now following a vigorous campaign? Who knows? But the trends revealed by the polls are undoubtedly real. That's why I think your monthly and even rolling averages constitute valuable information. I say keep at it, Éric!

I think what you and me should do (not together) is get closer to polling firms. Maybe request the detailed data systematically so that we can maybe correct the polls. As you said, frms who do polling don't only do political polls. We might be able to extract more info from their data than they do.

I hate to say it, but is there any question about the accuracy of the vote itself? Is there any reason to think not all votes were counted? Was electronic voting used at all in BC (I don't know as I don't live there) as that carries a big risk imo vs paper ballots. It just seems very odd to see such a break from the statistical norms. There is normally a cause and one of the potential causes is very, very ugly.

Now, other more likely causes could be who votes for each party and how dedicated they would be to getting out and voting. Polls suggested the NDP would win in a walk and normally left wing voters are less dedicated to voting than right wing based on the federal experience. Could it be that NDP voters just felt there was no need to vote thus didn't bother? NDP also does well with students and it is May thus most university/college students are done school and hunting for work thus might not see a need to vote.

Another shift is the 3rd parties - the Greens were dedicated to doing well and getting someone in, following the federal example of getting Ms. May elected. The Conservatives, on the other hand, seemed to give up as the campaign wore on. Generally Greens 'steal' votes from the left while the Conservatives (in BC) 'steal' from the right. It still doesn't cover everything but at least gives another factor strength.

Plus I've seen how American polling firms do both 'likely voters' and 'overall' - it might be past time for Canadian pollsters to do the same, or to adjust their methods if they are doing it. A simple 'did you vote last election' question could help there, mixed with 'do you plan to vote this time' - a yes to both = likely voter, while others are used to help political parties (who would buy the info) see potential trends should an issue come up at the last moment.

EKOS has lately begun consistently showing results for "all voters" as well as "likely voters", which I think they should be commended for. But bafflingly, in their final B.C. poll their "likely voters" results actually gave the NDP a *bigger* lead than their overall results. I think it's probably quite challenging to come up with a reliable voter turnout adjustment, but you're right that U.S. pollsters seem to be way ahead of us on this.

Eric - when you plug in the actual numbers for the election into your model, what results do you have for Delta North? What ranges? I was involved a lot on the BC Liberal campaign there, and I'd like to know how our ground game compares to the "generic" BC Liberal, which I think your model would give us a really good idea of. Thanks.

It would have been tough for you to know all the local variables. We had a very strong candidate, and the NDP candidate doesn't live in the riding. It's gratifying to hear that we punched above our weight though.

On the Delta South result, one issue you couldn't really account for is that Vicky Huntington tends to vote with the BC Liberals on confidence motions anyways. So she's already capturing a lot of BC Liberal support, and wouldn't necessarily have been hurt that much by an increase in BC Liberal support.

In my personal opinion, none of what happens falls on you, Eric. Given the proper data, the model did well. Maybe we all have a tendency to put too much faith in polls at times, but it's interesting nonetheless to look at the underlying dynamics of them, especially after the fact to me.

The solution--which you seem to be hinting at in the closing paragraphs--would seem to be more aggressive use of polling metadata in your own analysis. By incorporating firm track record and polling collection methods in your model, perhaps the results would be less vulnerable to bad polling.

After all, if pollsters are giving you (and everyone else in Canada) nothing but online samples to work with for a given election, it is actually a valuable service to spit out and publicize a result that basically says "these polling numbers are potentially deeply flawed. We really can't trust them."

Hi EricOne thing I have noticed in the last federal, Alberta and BC elections is that voters have opted for the status quo, whatever that is, often at the last minute. I think because of the lingering recession, people are scared of change. I would bet that if Ontario went to the polls now the Libs would get re-elected. It would be interesting to fing out if thats why people swung to the libs in BC.EricRW

Very early polls are well known not to be accurate, as most people have not properly given thought who to vote for nor have they paid close attention to the respective campaigns.

A key difference between Eric in Canada and Nate Silver in the US is that campaigns there are year long affairs so a good month before the election a substantial number of people have made up their minds.

In contrast in Canada the entire campaign lasts a month, so people often make up their minds in the last few days, including 9% who made up their mind in the voting booth, as Eric points out.

It follows then that the model in Canada needs to weigh much more heavily whatever trend is seen in the last few days, combined with more emphasis on "have you followed the campaign" and "have you made up your mind" questions.

You might want to have a look at the 1992 UK election when John Majors Conservatives beat Neil Kinnocks Labour party. The election result and polling had a lot of similarities. Then the polls were out by about 8%. After an inquiry by the polling companies a certain amount of the error was put down to "Shy Tory Syndrome", i.e. conservatives by their nature were less likely to answer polling companies. Another possible cause was thought to be some people were too embarrassed to admit that they were going to vote conservative because it was seen as the selfish party.

Another election that is similar is the Irish election of 2007. When a scandal plagued Bertie Ahern won election because the opposition couldn't give enough good reasons for people to vote for them, the power of encumbency (at seat level & national level) and lingering memories of the (supposedly bad) economic record of the opposition. The polls didn't get this election right either, (except possibly one done very close to the end which was could have been just good luck).

In these two elections plus Alberta and B.C. the encumbent party won re-election. In at least two out of the four the voters were worried about the economy. In all four the distrust level for the opposition was high and the electorate chose the devil they knew rather than the devil they didn't.

I wonder if you altered the polling data to add an encumbency / "Shy Status-Quo voter" bonus (3% maybe) and a bonus to the governing party that is proportional to the percent of voters that pick the economy as their top priority if it would improve the accuracy of the polling in the recent Canadian elections?

Kevin

(Interestingly in the UK and Ireland in the next election the governing parties got savaged!)

There's a belief that in most democracies long lasting governments tend to win one election too many for whatever reason, and at the following election the electorate turn with a vengeance. The surprise of the final re-election can in itself play a factor as voters find themselves unprepared for another circa four years.

What level of detail is released about the polls? Can an indepedent third party observer see regional breakdowns, demographic breakdowns, weighting by likeliehood to vote, how the "don't knows" and "won't says" have been allocated and so forth? Or does one only get tables as tarted up for a quick glance in the media?

Eric, I'm a faithful reader of your blog, although I don't comment often. The only part you got wrong was the only part you can't control, so you shouldn't beat yourself up too bad. In the majority of elections, things have been normal. As an Albertan, I know what happened here last year, and you were one of the only ones who caught what was going on, partly because you chose to report that one late poll that caught the beginning of a shift that obviously began on the Sunday and continued all day on election day. I was surprised at the result that night, but also knew that I had seen the trend begin on the graph you posted, and (as a PC supporter) had hoped that it would continue.

What happened this week was a total shock. I remember reading years ago that B.C. is notoriously difficult to poll because of the areas in metro Vancouver where you have extreme right and left people living in the same areas and it can depend on who answers the phone. (I realize that could be less relevant with online polls, and when all the pollsters are agreeing.) I suspect what happened is a combination of the factors you suggest. Perhaps a Shy Tory effect with an old government that had been unpopular for a while combined with confidence on the left of a win, that may have encouraged soft Green supporters to feel comfortable voting Green and may at the same time have encouraged soft Conservative voters to vote Liberal. It is interesting that you point out that the election was won in 12 ridings, as I counted 11 where the combined Green/NDP vote outnumbered the winning Liberal vote (I realize it's more complicated than simply adding the two numbers together, but suspect my point is valid in some of these 11). I wonder how many of these ridings in the 11 and the 12 are the same ridings.

Eric, on behalf of your many fans, especially those like me who usually remain silent, I say not to be too discouraged, and to keep up the great work. As others have pointed out, the polls in the past have shown the larger trends (including in Alberta), so although I acknowledge that there will be much reflection on where polling goes from here, this week's results may be simply the perfect storm of weird anomalies. (Maybe the Liberals were praying harder!)

One of the things that I really like about your blog is how up-front you are about both what we know and what we don't know. I would be very disappointed to see the blog falter or fold as a result of missing or misinterpreted data. If anything, we need a stronger voice making the limits of polling more widely known while working to mitigate those same limits.

I agree completely. I've been reading your blog regularly for a quite a while now, and have found your analyses of the data available to you most insightful. And that's the key issue: how robust is the data? One of the reasons Nate Silver was able to forecast the US election results so well was because he had reams of data to work with, across the entire country. It seems to be a lot more difficult with the kind of occasional polling that occurs with provincial elections in Canada.

The analyses you provide are valuable because they give insight into the _meaning_ of the polls and the trends. I can understand how you might feel a bit despondent about your forecasting model at the moment, but it's just suffering a GIGO effect. Anyway, more important than getting the forecast right is helping educate those of us who are not pollsters but only consumers of their data, and on that front you do an admirable job. Please don't stop.

Have you considered building a different sort of model? You know, one that takes say demographic variables into account like Silver has done (I remember the 'WalMart:Starbucks ratio" as one of his variables back in 08). I am not sure this would be even possible here, as there is less polling, and as our system relies on votes in ridings rather than provinces (states) it may be impossible.

One factor I just thought of--I was involved in a campaign a while back. The party head office had polling and could actually identify on our sheets when we went door knocking how the residents of a house had answered telephone polls. There was a guy in our riding who was a personal friend of our candidate. He came to our campaign office just to hang out. Yet, I knew from looking at the software, that in a telephone poll, he said he would vote for the NDP- a rival party.I don't believe he actually voted against his good friend, so it could very well be that asked about a party he would say NDP to show he disliked our guy's party, but he would vote for our guy regardless of his party.So it is possible that the incumbents received votes because of personal popularity.Could that account for 5%? I don't know, I suspect the problem is larger than that. But perhaps the polls didn't dig deep enough beneath the surface.

Finally, I suspect that machine politics, sophisticated get out the vote strategies, were crucial in both Alberta and BC. Certainly Alberta with the one really well established party, and the other party with a groundswell of support but a lack of sophisticated staff etc..

Since voters on not as loyal to political paties on the provincial and state levels, elections tend to be more difficult to call on the state and provincial level than on the federal level in both Canada and the United States.

I think what this election (in particular) shows is how polls at best can indicate (i.e. point to) an instant in politics, but can't provide an explanation of politics. Most frequently, poltiical forces in action during a given election are going to be comparatively simple and polling will capture that. But this was one of those situations where the political forces were simply too complex for polling to describe. It's importatn to remember that polling and politics may correlate, but there is no deep connection... We have a situation nationally and in BC where party politics are in flux, we have several polling methodologies in use at a time, when people are using more means for communication than ever before, where there'S a multi-cultural and multi-lingual population, where the mainstream media excercises a strong influence in support of the status quo, and where people various political issues (the environment, employment, etc.) are seen as being at odds with each other. It'S a moment with too much political movement for narrow statistical operations to gauge. It won't always be this way...

This was an unusually intense and complex situation - though some elements are increasingly 'the norm', it's true. I don't believe pollsters can ever fully adapt, simply because their mission is only (at best) to understand an extremely narrow part of the situation.

As you say, if there is movement a month of polling is not long enough to capture the steady state. The model needs to amplify swings, specially if they are as large and persistent as that shown for the BC liberals.

in our riding,which was a classic 3 way battle to the very end, we were peppered with pollers throughout the campaign. since we do not answer the phone if a 1-800 number is displayed, we were not surveyed. many friends voiced similar comments. why was forum research not quoted more? they picked up the swing, at least partially, and early. no media outlet in b.c. quoted them, to my knowledge,although the poll was done for the national post. imo, online polls cannot be trusted as there isn't a true randomness to the process. activists are more likely to participate than average folk. ALL of the major pollsters in b.c. were using online polling and were the furthest out. here is a topic i'd love to see addressed here. the parties on the left of the spectrum are better equipped to address the environmental changes we are facing and the negative consequences for coming generations if we don't do something about them now. quite rightly, from my point of view. why then, are these same parties so unable to size up the negative consequences of financing current social programs from future revenue? mr. dix was prepared to do away with balanced budget legislation and run deficits and the green party's fiscal platform was incomprehensively ridiculous( move away from a resource based economy, no new taxes but for an increase in gasoline taxes, guaranteed balance budget and a guaranteed annual income at a "comfortable" level for all). really?? and i failed to hear anyone question this nonsense, eric, this blog is very readable on a number of topics. i hope you continue with it. it is most useful on federal topics but i go to it for provincial issues as well. thank you. cheers. paul.

As a supporter (but not a member) of the left wing parties, I'll try to answer the questions in your second last paragraph.

I think Mr. Dix's NDP budget plan was an reasonable assessment of what the reality will be under any party. The BC government is, in practice, running a deficit now and no party, realistically, is going to hike taxes/cut services enough to balance it. There will be deficits under the Liberals, too, (I think bigger deficits but there's no way of knowing), but the media give them a free ride on deficits. The Liberal "plan" for paying off the debt was IMO complete fantasy. (What are you going to do 15 years from now when the provincial debt's bigger than it is now? Vote Clark out of office? She won't be leader then, and chances are her successor won't be either.)

I don't think legislated balanced budgets are a good idea for a couple of reasons, one of which is that it's really, really easy to cook a budget. Remember, a budget is a plan, not an outcome. Balanced budget legislation doesn't stop a government from tabling a balanced budget at the start of every year and having a deficit in the audited financial papers at the end of every year.

Ms. Sterk was questioned about the Green "budget" during the leaders' debate and her response was that the Greens had not attempted to craft a realistic budget because they did not expect to win government.

I love the site and follow it regularly, but like many of us - in shock at how badly the pollsters performed this time (less than 2 years after the Alberta debacle). Here here! to the caveats and more caution about predictions. Yesterday on CBC the Current there was a debrief on the polling problems and one commentator raised an important issue regarding the wider implications of bad numbers. You alluded to it as well, NDP supporters stayed home (thinking it was in the bag) and Liberals were defending their local MLA and showed up to vote, even thinkin that they may lose at the prov. level. So my question to you is...what repsonsibility do pollsters have for generating these distortions in actual voting behaviour? I know people decide based on many information points, but certainly poll data (skewed or otherwise) is a key element in this mix.

I was just polled by Angus Reid. Lot's of questions regarding whether I voted, when I decided, when I voted, was I satisfied with the results. It seems they too are trying to get to the bottom of things.

Also, I know that this may be contrary to everyone's knee jerk reaction of blaming the polls. I am curious as to what the polls will say post election. One of the things that occurred with all of the other elections that had unexpected results is that the subsequent polls validated the last minute shift in support (Federal election, Alberta, Quebec). If we still get the same polling results as before v-day then maybe that begs a new question.

What if the polls were right?

Could turnout be so problematic for some parties (in this case the NDP) that polls simply can not predict an electoral outcome? The swing was not uniform at all, as the NDP scooped new seats and lost seats that were otherwise deemed "safe". The newly won NDP seats were very strong local campaigns - perhaps resulting in higher turnout. Obviously, it is too soon to come to conclusions, but maybe turnout had a bigger role to play than we think (ie. 60% Liberal supporters voted, 40% NDP voted resulting in net ~50% turnout).

The Liberals won the election, but did they really win the support of British Columbians? My own political feelings aside, I really do hope so, but am unsure.

Regardless of the accuracy of the predictions, I find the commentary is educational and thought-provoking. I would be loathe to see the site close down or even become excessively discouraged and overly cautious. Keep up the excellent work.

From a brief glance at the data--here and in Alberta--it seems like there was a problem screening actual voters from the sample as a whole. Asking people if they're likely to vote is a terrible way to do this. Nobody wants to say they're not going to vote. Prior behavior and demographics are probably better. But it's hard to say for sure without seeing the methodology and the questions.

Look at the Ipsos samples (last poll vs. post-election poll). In the last pre-election poll only 39% of their weighted sample was 55-and-over; 40% was 35-54; and 21% was 34 and under. In the representative post-election poll: 49%, 35%, 16%. That suggests some problems identifying what the electorate's actually going to look like. In Alberta, the last Leger poll had 64% saying they were extremely likely to vote and 24% very likely. Turnout was 57%...

If you're including lots of people who aren't going to vote in the sample, you'll overrepresent, among others, (a) certain aggregates--e.g. younger people and less politically interested people--who are more volatile and probably more likely to opt for less established or opposition parties, and (b) voters for parties that aren't going to win (BC Greens, Conservatives; Alberta Libs, NDP).

A separate phenomenon is trying to identify vote-parking with smaller parties (Greens and Cons, in the BC case) in respondents who actually are likely to vote and will ultimately vote for a potential winner. Maybe polling companies should--in situations like BC--ask a two-party preference question. I don't think this is as much of an issue in US general elections since third parties are essentially considered irrelevant from the start.

I'm skeptical about the motivation theory. There's no inherent reason why it should affect one party more than the other--and if you look at the pre-election BC polls, those who supported the NDP were much more enthusiastic than those supporting the Libs.

Éric, thanks for your hard work, and keep going! As an American following Canadian politics this site is one of my favorite destinations.

I think you're too hard on polling in Canada. It's not all that bad... but I think the pollsters in Canada are too Eastern-based. Polling in Ontario and Québec tends to be more accurate. In 2011, polls in Ontario did relatively well. In Québec too, even if Liberals had been a bit underestimated, the results were still very close.

But in BC and Alberta, polling is just plain awful. Not just during provincial elections, I remember following federal campaigns in recent years and seeing numbers being all over the place in BC. Numbers are also pretty inaccurate in Alberta, but since the Conservatives have such a massive lead, it doesn't matter whether they get 65 or 50% of the vote. Polling companies just seem not to know what is going on in the West. I think their models are way off.

Eric, you mention that the model couldn't have predicted Andrew Weaver's win. I actually estimated the Green results pretty well (nothing else except the obvious, but I estimated the Green results). These were my clues, in the off-chance that you can use them:

1. The recent Federal by-election in Victoria. Map that onto the provincial Victoria ridings, and it's no surprise at all that the Greens won in Oak Bay, and not in the other Victoria ridings.

2. "Star" candidates. Consider the Green candidate in Saanich North, Adam Olsen. He's a municipal councillor. For a mainstream provincial party, that is a totally normal background for a candidate who's not an incumbent. But for a minor party like the Greens, he's a star. Not just that he's a professional who's won election before, not just that he brings his own supporters to the table, but the very fact that someone like that thinks it's worth his while to run for a minor party in that riding tells you that stuff is happening on the ground in that riding. I thought he had an outside chance of winning, and indeed he get over 30% of the vote in a tight three-way race.

Compare the West-Van-Sea-to-Sky riding, where the Greens have gotten a lot of support in previous elections, but where AFAIK they didn't have anyone nominated when the writ was dropped this time around. It wasn't hard to predict that Adam Olsen would do better in Saanich North than whoever eventually got roped into carrying the flag in Sea to Sky, but your model just saw the results from the last provincial election.

3. Weakness in the incumbent. For the Greens to win a seat, they have to not only take votes from the parties that lost last time, but from the party that won last time as well. When Elizabeth May finally won, it was against a weak incumbent, although I suppose you'd have had to have talked to local people to know that. (I have family in the riding.) The guy was a cabinet minister, so he looked like a strong candidate on paper, but he'd done a poor job of representing the riding. This time around, the Liberal candidate in Oak Bay wasn't weak, but the Liberal party in Victoria was. So I knew that the Green candidates in the two former Liberal ridings in Victoria had chances. By contrast, the NDP was not weak in Victoria at all. So I knew that, for instance, Jane Sterk wasn't going to get very far against the NDP incumbent she was running against.

Not sure if you saw this yet, but the BC Liberals were apparently "certain" they were going to win a full FOUR DAYS before the end of the campaign, based on internal polling numbers - http://news.nationalpost.com/2013/05/18/why-the-liberals-were-able-to-predict-their-victory-in-b-c-while-public-election-polls-missed-the-mark/.

On a different note, I used to really like the Canadian map graphic you had at the top of your page. It was a nice, inviting visual, instead of just seeing a wall of numbers. Maybe you will consider bringing it back! :)

Thanks for all your work on this page. People will still be interested in poll numbers, if for no other reason that the fact that there's nothing better to go on (and we need SOMETHING to go on), and the polls are USUALLY correct. I don't think you have anything to worry about.

It has more details too on the age weighting. Since your methodology (and everybody else's apparently) doesn't weight based on demographics as a proxy for voting likelihood (can't because pollsters don't release that info?), then that may explain a lot of the error (and likely the compensation of the Green vote that you apply).

Unsurprisingly, the mock high school vote was closer to the polls than the actual vote which illustrates a lot of where that skew happens. That's probably also why the telephone polls were more accurate than the online polls on average.

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.

Details on the methodology of the poll aggregation and seat projections are available here and here. Methodology for the forecasting model used during election campaigns is available here.

Projections on this site are subject to the margins of error of the opinion polls included in the model, as well as the unpredictable nature of politics at the riding level. The degree of uncertainty in the projections is also reflected by the projections' high and low ranges, when noted.

ThreeHundredEight.com is a non-partisan site and is committed to reporting on polls responsibly.