Monday, June 4, 2012

Polls and projections in context

Canadians are three years away from the next federal election, but nevertheless polls on national voting intentions are released at a rate of about one per week. A provincial election in Quebec is around the corner, Ontario has a minority government that could potentially fall at any time between now and 2015, and British Columbians will be voting in May of next year. A discussion of polls and projections seems warranted.

With the next federal election so far away, there is good reason to question why we pay attention to polls. Meaningless! say critics. As I wrote about in my The Hill Times column last week, and as I will be speaking about at next week's MRIA Ottawa Chapter panel discussion, that could not be further from the truth.

Looking at federal polls today through the rubric of 2015 is to miss the point. Of course things will probably change by then, but what Canadians think right now is important and worth knowing. Arguing that polls are meaningless today is to say that the opinions of Canadians are meaningless. It is disappointing that these sorts of comments often come from partisans who see their party losing support. It is no less disappointing that these same partisans will happily gloat about the latest numbers showing their party on the rise.

The voting intentions of Canadians this far out from an election have nothing to do with the vote in three years. It has everything to do with what they think of their leaders right now. A mid-term report card, these public opinion polls give Canadians an indication of what the country thinks of their leaders. If polls are mere snapshots in time, elections are as well. Democracy is more than just voting once every four years, polls are a means (limited as they are) of keeping our political leaders in check. Would Stephen Harper or Thomas Mulcair be acting differently if the polls put them 20 points ahead?

Imagine a scenario where a government is at 10% support. What legitimacy would they have to make controversial decisions? Legally the legitimacy would still exist - but morally? And what if those same public opinion polls demonstrated that Canadians were hugely opposed to a particular government policy? Perhaps more to the point, what if this were the case but public opinion polls weren't available to tell Canadians what their fellow citizens thought?

Perhaps some people think that Canadians can be ignored outside of election campaigns. Undoubtedly, there is little to stop MPs from ignoring public opinion between elections and they can often get away with it. Whether that makes it right to ignore public opinion is another thing entirely.

At the very least, Canadians deserve to have some of the same information as political parties do. They can pay for proper polling, and likely have a much better picture of what Canadians think than can be derived from the polls paid by or provided to the media. Nevertheless, the information that media polls do give us at least levels the playing field somewhat.

Of course, critics of polls will always bring up the cases in which they have failed. The most recent whipping boy is Alberta. That the Alberta election campaign was exceptionally unusual from a public opinion polling perspective (not to mention historically and politically) is, to them, besides the point. That polls did magnificently in the recent Manitoba and Saskatchewan provincial elections, and quite well in Ontario, doesn't matter.

But Alberta might serve as the perfect example of why polls can be useful. Support for Wildrose during the campaign may have been more about being anti-PC than it was about being pro-Wildrose. If polls had not been around to show that Wildrose was making incredible gains, Albertans may not have given as much thought to what a Wildrose government would mean, and the media might not have put more effort into looking into the party's platform and candidates. Democracy is not, in the end, the sum of individual opinions. It is a decision made by society as a whole - that is why we discuss politics with our friends and families and read opinion pieces in the newspaper. If, in this way, we indirectly "poll" those around us and those with insights we trust, why wouldn't we want to know what everyone else thinks as well?

Along with the anecdotal evidence, the observations of people working for the parties in Alberta, and post-election polls, a look at the difference between results in the advanced polls and on election day suggest that Albertans were turning away from Wildrose in the final days of the campaign. This second look at the party, which might not have taken place without public opinion polls, appears to have had an effect.

Now, people who vote in advanced polls aren't necessarily the same sort of people who vote on election day and are not necessarily a fair representation of the voting public. Enthusiasm, commitment, and GOTV operations can play a big role.

Nevertheless, between the advanced polls of Apr. 19-21 and election day (Apr. 23), Wildrose support fell sharply. They took 37.4% of the vote in the advanced polls by my calculations, putting them ahead in 25 seats. A few days later, their share fell to 33.8% and they ended up winning only 17 seats. This is not the dramatic swing that would have had to take place to corroborate the final days of polling, suggesting that the polls may have been over-estimating Wildrose's lead throughout the campaign (or, perhaps, that fence sitters and weak Wildrose supporters were still making up their minds), but it does suggest that there was a swing taking place in the final days.

But as the polls did not call the result, a site like ThreeHundredEight could not call the result. I have often seen comments that since 308 was "way off" in Alberta, how could it be trusted? This seems to be a misunderstanding of what this site does. This site translates polls into seats. Any adjustments the model makes are still based on the raw data provided by the polls. People looking for sites making predictions and guesses have plenty of options - ThreeHundredEight uses the real data that is available. I didn't forecast a Wildrose victory because I thought Wildrose would win (though, reading my final projection, I challenge anyone to argue that I was projecting a Wildrose win without any reservations), I forecast a Wildrose win because the polls suggested they would win.

That should not bring into question the accuracy of seat projections, but it does anyway. Seat projections have been shown, time after time, to be able to come to reasonably accurate reflections of seat totals when the real voting results are plugged in. This is not due to the benefits of hindsight, this is due to the fact that seat projection models work. Though they do not get every seat call right, the errors generally cancel each other out. The overall result is what is important. Focusing on missed riding calls is to miss the forest for the trees.

Seat projections translate polls into generally accurate seat totals. This is a worthwhile exercise in a country like Canada, where the FPTP system obscures what national (and even regional) voting intentions truly mean.

But that brings up the criticism that seat projections are an attempt to forecast the 2015 election. When I projected that the NDP could win 133 seats on Friday, I was not projecting that the New Democrats would win 133 seats in 2015. As with every projection that is posted on this site, the seat totals reflect the likely result if an election were held on the days where the polls in question were conducted. That may seem like meaningless trivia, but if public opinion polls are a reflection of what Canadians think of their leaders (and if this information is, intrinsically, of value), seat projections put these numbers in the context of what the implications of these voting intentions would be in an election.

Despite the next general election being three years away, federal polls will still be released every month. They will give us an indication of how the parties are doing, and what Canadians think of what they are doing. Provincial polls will be released as we approach the elections in Quebec and British Columbia, and seat projections will provide further context as to what those voting intentions could mean. An understanding of all of this information gives us a more global view of Canadian politics and the decisions of our leaders. More information is good - if it is correctly interpreted, placed in context, and not ignored.

24 comments:

I feel polls are always worthwhile. As you have stated they give the citizens of this country a feeling for how other citizens are responding to the platforms and policies of the parties. One area that polls could be improved on, is if they were to be able to predict whether the people they are polling are the ones who will actually go out and vote. In your recent post about the Alberta election you talk about the polled supporters of the Wild Rose, who were probably just anti-government responses. I feel that there is a lot of anti-government poll response to any party in power among those who never bother to vote. If asked, they will say they do not like the party in power, and will vote for the party most likely to defeat them. However, they did not vote last time and will probaly not vote this time. If this element could be factored into your polling, it could help to improve the accuracy of your predictions.

Hi Eric. I'm a big supporter of what you do here. Despite this, I do need to disagree with this post. If the argument is whether polling for voting intentions 3 years from an election is useful I would have to argue the answer is probably not.

This is not to say that it is uninteresting or not a useful proxy for popular sentiment. However, if popular sentiment was the objective, why not survey it rather than using voting intentions as a proxy. There is no need for quantifying something so subject to fluctuation so far out.

1) Will you switch over to modelling with the expanded seat numbers when the commission releases its first report?

2) There are factors impacting polls in-between elections (increased number of people not paying attention to politics) - is there a way to quantify these variances? Have there been any snap election calls, or unexpected elections, that we could look at that would bring greater verification to the polls?

2) That is a good question. The one recent example that comes to mind is the Quebec election in 2008. Weeks before it was called, Charest was saying he would not call an election. So, it came as somewhat unexpected.

If we look at the polls between August and October 2008, we see the PLQ at 41%, the PQ at 33%, and the ADQ at 16%. The results in December were 42%, 35%, 16%, respectively.

You totally gloss over the accuracy an integrity of the published polls.

The polls are being used as campaign devices.

You even indicate taht the polls that showed the WRA leading in the election was one of the main reasons that the WRA lost the election.

So that means if there were more pollsters in the election and they showed a solid PC majority the WRA would have been elected?

Why wouldn't the WRA spend 5% of their campaign to have polls published showing them not to be a threat??

The pollsters that are off by more than the margin of error the day before an election should have to publish a warning (like on cigarette packages) that their polls are scientifically inaccurate and should only be considered for entertainment or as part of biased political campaign.

The polls right now are just for fun as there is no way to calibrate their accuracy.

You would not believe a poll I design and commission that show a Liberal majority if a vote were held today and teh NDP return to 4th party status.... but there is no possible way that you could disprove it.

Would that remove the moral authority for the NDP to act as offical opposition?

The Green Party got the included in the major party debates by polls that showed them with far more support than they ever got in an election.... That was an abuse of the polls.

In response to your comment, I'm copying and pasting a post I recently made to a G&M article in response to another commenter who was similarly suggesting that pollsters have somehow become shady partisan "campaign devices":

"While it's possible that certain pollsters may have manipulated/fabricated results on occasion, it's truly not in their best interest to do so. This is because the community of pollsters is engaged in an ongoing competition to see who can come closest to actual election results. It's considered a blow to a pollster's reputation if they totally miss the mark. Therefore, falsifying results, especially during an election campaign, is very risky. To suggest that half a dozen pollsters were involved in some sort of conspiracy to "engineer" the results of the Alberta election is a real stretch."

As Éric has argued, there is good evidence that the Alberta election was anomalous in that there truly was a remarkable shift in support at the last minute which most pollsters simply failed to pick up on because they conducted their final polls too early, possibly combined with an unusually large proportion of undecideds who waited till the 11th hour before massively converging on the PCs. I think the single strongest piece of evidence in support of this is the final two Forum Research polls conducted on April 21 (WR 41, PC 32) and April 22 (WR 38, PC 36). No other pollster kept polling that late, thus no one else captured that hint of the rapidly tightening race.

In all likelihood, all those polls showing a large Wildrose lead throughout the campaign were probably more or less accurate and may indeed have caused many Wildrose-wary, non-PC voters to panic at the last minute and strategically vote PC as well as caused many Wildrose "leaners" to seriously ponder their choice. Even if it would have been an option for Wildrose to commission a few "shady" polls to make it look like their lead wasn't so commanding, how could they have been sure this would be a wise strategy? After all, you could conversely argue that polls showing a particular party doing well could in fact cause more voters to jump on the bandwagon. This has more than likely happened before; I wouldn't be surprised, for instance, if this phenonemon helped boost the NDP last federal election.

So on the one hand there seem to conspiracy theorists suggesting that polls throughout the Alberta campaign were falsely showing a strong Wilrose lead in an effort to get voters to jump on a "fabricated" bandwagon, while others such as yourself seem to suggest that Wildrose somehow could have commissioned false polls to downplay their level of support and avoid scaring voters about the prospect of a Wildrose government. All in all, in my opinion, the theories just don't add up.

"The pollsters that are off by more than the margin of error the day before an election should have to publish a warning (like on cigarette packages) that their polls are scientifically inaccurate and should only be considered for entertainment or as part of biased political campaign."

This statement indicates that you do not understand the statistics behind polling information. The polls of 1000 random people are always reported as accurate to +/-4% 19 times out of 20. That means that 1/20 times you do that poll, you will get a result that is outside of the 4% error.

For example, in theory, I could correctly poll 1000 completely random people throughout the country and they may all say they are voting for the Green party. The probability of getting that result on a poll is extremely low, but, as long as there are 1000 Green party supporters in the total population, it is possible. Still the result of my completely correct poll would be Green party 100%, every other party 0% +/-4% 19 times out of 20.

While this was an important point, and it needed to be made (and I mostly agree with it), there was one statement within it to which I must object:

"Democracy is not, in the end, the sum of individual opinions. It is a decision made by society as a whole..."

This is arrant nonsense. Even ignoring the suspect nature of the concept of society in the first place, democracy is demonstrably the sum of indivudal opinions. The casting of a vote is a solitary activity. This isn't like a presidential caucus in the US where there's pressure from the group to vote one way or the other; these decisions aren't reached by consensus. As long as we employ secret ballots, democracy absolutely is the sum of individual decisions.

On the votes, I'm a strict nominalist. I deny the existence of groups. As such, there is no such thing as a community or a society. There are only indlviduals.

Moreover, I dispute your claim that there exists such a thing as moral authority in a system where the legal authority is so clearly estalished. Once we have formal rules governing the system, unwritten rules cease to be at all relevant.

Ira, I suppose you're very brave to go against several thousand years of discourse and thought that has attempted to describe, understand, and theorise the nature of society, a phenomenon that isn't even unique to humans, although the human variety is obviously more complex than those of ants or baboons.

Perhaps more accurately, I insist that positing the existence of groups is unscientific.

There is nothing about group behaviour that can't be describes as individual behaviour in the aggregate. As such, positing an extra layer of complexity (the group) over and above the individual violates Occam's Razor.

I agree with this post, and the importance of polling. Of course polling can never be fully accurate, and there should be debate about the accuracy of each conducted poll.

Polls indiciate the mood of the voters. Right now in the federal level the government has seen a slight dip in support, while the NDP had a notable increase in support throughout the country after Mulcair became leader. Polling also shows the numerous government scandals and mismanagement haven't took a toll in the Conservative core support, as the party is always polling at or above 30%.

Polls show that the premiers in B.C, Ontario and Quebec are not well received. The premier of B.C. would almost certainly lose the next election, while the other two premiers are facing opposition who aren't in a strong position to form government.

This information is very useful for parties for organization and strategy. What message would reason would the electorate and what would not.

I do not question the accuracy of polls or seat projections but, something must be said for the "on the ground factor". The accuracy of a poll is not able to capture the "feeling" on the ground-are people paying attention or interested, what people think of the candidates on a personal level, idiosyncratic ridings and the like.

As you mention a shift took place in Alberta in the last days of the campaign. Undoubtedly some nominal Liberal and NDP supporters moved to the PCs but, my impression speaking to people was the PC vote was underplayed by pollsters and media (particularly the Sun) whereas Wildrose vote was overplayed by the same groups. In this way a distorted picture emerged.

I spoke with one Wildrose campaigner who was concerned WR was not doing as well in rural areas as the polls led us to believe. In particular the Tory ability to GOTV, popularity of individual members and habit. In my own riding of Calgary-Mountainview your high projection was 13 pts lower than the winning candidate received.

Obviously, it is impossible for you to visit every riding in Alberta or the 308 across Canada. I understand your methodology precludes you from "picking and choosing" ridings, as you say generally the errors even themselves out. However, I don't think you should totally ignore your good judgement. To predict for example a WR victory in Mountainview (perhaps the most liberal riding in the province)is really going against the grain especially when late polls indicated "progressives" coallescing around the PCs. If anything this riding would have followed the trend-a PC victory.

I would suggest in future you create two projections; one purely based on the numbers and the second factoring your own judgment calls for individual ridings. Such judgment calls may not be possible for some areas of the country where you have scant knowledge but, in others I am confident they would predict a more accurate result.

Including judgement calls at all though undermines the purpose of the site: to turn polls into projections without personal judgment intruding. The point is "what do the polls say" not "what does Eric think is going to be the outcome."

I am unsure how you are able to speak for Eric on the purpose of this site: "to turn polls into projections without personal judgment intruding".

If we look at the seat projection methodology we see a number of subjective criteria included; star candidate factor, new cabinet minister bonus, incumbency, floor crossers and individual oddities. Eric attempts to quantify the "bonus factor" through statistical analysis but, as Eric states when explaining the "star factor": "My method of choosing a star candidate was based on my own personal judgment" and so we are left with a methodology that employs both qualitative and quantitative data.

In the case of "floor crossers" Eric gives a bonus to the party that received the MP and penalises the party that lost the MP. I do not know Eric's reasoning for doing so as he does not go into detail but, clearly the result is a subjective choice to penalise one party and reward the other. Without data a strong case could be made that the opposite would be correct. The floor crosser would be penalised in the subsequent election. Certainly this was the belief of David Emerson.

Clearly the vote projection model and seat projection model left much to be desired in the Alberta 2012 and New Brunswick 2011 elections. Many factors were at play and I don't think it fair to rubbish the model entirely. In Alberta it was unable to pick up a late shift in votes, which is understandable given the dearth of late polling results. In New Brunswick the vote projection was off by almost 5% for the Tories and 6% for the Liberals. In other words the vote projection for New Brunswick would have been an outlier or the 1/20.

For the Ontario provincial election a seat projection website that uses subjective data actually did better than Eric with a 91% accuracy rate. This doesn't mean Eric's methodology is wrong or inaccurate it could just be dumbluck on the part of the website or random chance. While generally the projection "evens itself out" in terms of inaccuracies it may also produce nonsensical or illogical results.

Calgary-Mountainview being a case in point. The Liberals won the riding with 41% of the vote. Eric's projections ranged from 18.3% (low), 23.3% (medium) and 28.3 (high) for the Liberal party candidate with Wildrose picking up the seat (low WR projection 30.5%). Eric predicted the Grits would get 9.9% of the Alberta popular vote compared to 25% in the last election. On the face of it Eric's projections for the Liberals in Mountainview appear spot on but, he has failed to account for the "known unknowns"- that is the Liberal vote will hold up better in Mountainview than most other places in Alberta simply due to the idiosyncracies of the riding and demographics and therefore it holds a disproportionate number of the total Liberal vote.

Was his projection wrong for Mountainview? Not entirely but, those of us on the ground and others with experience in Alberta politics would have Flagged Eric's prediction as odd-not impossible but, certainly unlikely since, by Eric's own provincewide projection WR was not headed for a landslide. On the ground there were other speculative indicators that conflicted with Eric's projection such as the large amount of signage for the Liberal candidate and relatively few for the WR.

Any good scientist must use his common sense when analysing results. That is why peer review and the ability to reproduce or replicate results is so important. Anonmallies are common and errors occur.

Having two projections one solely based on the numbers the other "tweaked" with personal judgement would no more undermine the credibility of the site than a prediction way off the mark. It may even improve the accuracy. If it didn't or became superfluous he could always drop it at a later date.

All adjustments are calculated based on past occurrences, including the floor crossers. I analyzed what happened in ridings where there had been floor-crossers to come to my adjustment.

Though I choose "star candidates" subjectively, the bonus they get was also calculated based on the experiences of past star candidates. And though I choose them subjectively, they are not chosen arbitrarily. The same standards are applied across the board.

So, in the case of Calgary-Mountainview any sort of tweak would have been completely arbitrary. How else could have adjusted for having a "large amount of signage"?

New Brunswick 2010 (not 2011) used an entirely different system, so it is not a good example. I disagree, however, that in Alberta 2012 the vote and seat projection model left much to be desired. Aside from missing the Liberal seats, the model performed very well when the right regional numbers were plugged in, despite Wildrose going from 7% to 34% in one election cycle. The vote projection model was at the mercy of the polls.

To question the legitimacy of a government at 10% in the polls- arguing they have lost the moral authority to govern is to question democracy itself. It is to say the government is bad in a metaphysical sense regardless of any policies, programs or misdeeds they may or may not have performed. It is to place a value judgement based only on popularity not deeds or actions. Many governments find themselves unpopular through making tough unpopular choices that in hindsight are judged correct or beneficial. The War Measures Act, participation in the Second World War, Free Trade would any of these policies receive high levels of popular support when introduced? To argue a government acts immorally without popular support irregardless of the cause is to negate morality and become enslaved by mob rule-it is barbarism! It is simply unjustifiable to argue an unpopular government would be morally right to shirk participation in the fight against Nazism or Rwanda or one may argue Syria today and it demonstrates why elections not polls confer legitimacy upon governments.

Whatever the benefit of polls and there are many- a sample size of 2,000 with confidence intervals +/- 2.5% 19 times out of 20 does not carry the moral authority to decide for all Canadians. In the first place it represents only .00006% of the population. Secondly, 5% of the time the poll may be wholly inaccurate.

The idea that unpopular (elected) governments are immoral leads to violence, rioting, chaos and justifies disregarding and breaking the law-in so doing the liberties and freedoms of others are trampled and lost. In Quebec we see such actions on a near nightly basis. Some argue M. Charest’s Government has lost the moral authority to govern; they do so not because the Government is unpopular but, through circumstantial (at this point) evidence that corruption, money laundering and out-right graft and bribery are endemic within the Quebec Liberal party if not the Government itself. It is far less easy to defend the actions of protesters who damage property, terrorized fellow students holding different opinions and disrupt the economy of Montreal. However, if unpopular governments are immoral violence is justified and legitimate political change can only emerge from the street not the ballot box. A philosophy such as this is inconsistent with democracy and the rule of law. Such violence is only justifiable in tyrannical jurisdictions without free and fair elections where the rule of law is nonexistent or skewed towards a particular group or class and where other avenues such as peaceful demonstrations or the press are denied or curtailed.

Nice view Senex, democracy in the end it's to let people decide, be informed, and stay civilized. Our system is surely not perfect, but Churchill was right: Democracy is the worst form of government, at the exception of every others.

Éric, I would simply say, you allow me, and many other I'm sure, to have a view of what Canadian think of their government, to translate meaningless number into an insight of what going on in the public opinion. You fallow the change between polls of the same firm and compare this change among various firm. In short, you are doing what everyone should do, but no one have time for.

I don't really care if your seat projection fail. Maybe one century ago someone could accurately predict elections, now it's just impossible with everyone having access to a lot of information. Continue your good work. You might fail at time, but a citizen should always inform his self from different source.

I don't think you need to justify what your doing, because you are doing it in the right spirit. Good job and keep at it.

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.

Details on the methodology of the poll aggregation and seat projections are available here and here. Methodology for the forecasting model used during election campaigns is available here.

Projections on this site are subject to the margins of error of the opinion polls included in the model, as well as the unpredictable nature of politics at the riding level. The degree of uncertainty in the projections is also reflected by the projections' high and low ranges, when noted.

ThreeHundredEight.com is a non-partisan site and is committed to reporting on polls responsibly.