Are conservative parties under-estimated in the polls?

Pratte argues that polls quasi-systematically under-estimate centre-right parties, most recently seen in last week's election in the United Kingdom. He also cites the provincial election in Alberta, the 2012 election in Quebec, as well as the Scottish referendum and the most recent vote in Israel.

He posits two explanations for this problem: conservative parties are supported by older people, who are less apt to express their views to strangers on the telephone; and right-wing views are portrayed unfavourably by journalists, artists, and intellectuals, who are important players in the public sphere. People don't want to publicly go against the grain.

We could add to this theory that if centre-right parties are disproportionately supported by older voters, they should disproportionately benefit at the ballot box, since older people vote in bigger numbers.

It is not a bad argument. Anecdotally, it certainly seems right. It feels right. But is it right?

I've looked at federal and provincial elections over the last 10 years to find out. I've compared each party's average performance in the last week of polls to the actual election result. I've included all major conservative parties (in addition to the PCs and federal Conservatives, I've included the ADQ/CAQ, the Saskatchewan Party, Wildrose, and the B.C. Liberals).

And since Pratte included them in his list, I have also included the Quebec Liberal Party. In the context of Quebec politics, the PLQ is arguably a centre-right party, and the 'shy Tory' effect Pratte cited has often been used to explain the Liberals' sometimes surprising electoral performance in Quebec.

The verdict? While there does seem to be a generalized under-estimation of conservative parties in the polls, the effect has been exaggerated and, in too many cases to make it a rule, has not happened at all.

You can see in the chart above that, in most cases, conservative parties do indeed out-perform the polls. But you might also notice that there are more than a few cases where they under-performed the polls, as well as many cases where they came so close to the polling average that we can chalk it up to sampling error rather than a methodological bias.

In fact, the average error for conservative parties has been just 1.6 percentage points. That isn't nothing, certainly. That is particularly the case when we compare this performance to that of the largest non-conservative party, which polls have over-estimated by an average of just 0.2 points (in other words, negligibly).

Perhaps most importantly, in 14 of the 24 elections I looked at there was no real effect on the outcome whatsoever due to any under-estimation of centre-right support. It didn't change the winner or the order of the major parties. In another two elections, it only changed who finished second and who finished third. In only three cases did it flip expectations of which party would actually form government, and in just two did it turn a minority victory into a majority one.

This means that in five of 24 elections, or just 21% of cases, did an under-estimation of centre-right parties in the polls have a major, lasting impact on an electoral outcome.

In 20 of the above 30 cases, a conservative party was over- or under-estimated by more than a point, meaning that in one-third of cases the polls were close enough that we can't seriously identify a systemic problem with the methodology.

But still, in 20 of 30 cases the conservative party was under-estimated by the polls. That means that if you're placing a bet, you'll be more right than wrong if you guess that the conservative party in the election will beat its polls. That still means, though, that you could be wrong one-third of the time.

In many cases, assuming that the polls will under-estimate the conservative party, as Pratte suggests journalists should do more often in the future, will lead you astray.

Let's say you assume that the polls will under-estimate the main conservative party by about four percentage points (Pratte cites, with numbers, the PLQ in 2012 and PCs in 2015 in Alberta, where the error was about that large). Here are some of the errors you would have made:

- Progressive Conservatives victories, rather than defeats, in Ontario in 2011 and 2014, in Manitoba in 2011, and in New Brunswick in 2014.
- The Official Opposition role, rather than third place, going to the Progressive Conservatives in Nova Scotia in 2009.
- Conservative majorities in the 2006 and 2008 federal elections, rather than minorities.
- Landslide victory, rather than a closer result, for the B.C. Liberals in 2005 and 2009.
- Minorities, rather than majorities, for the Liberals in 2007 in Ontario and Quebec.
- Would hardly have been helpful in Alberta in 2012, since the race was between two conservative parties.

Even if you assumed the under-estimation would be more modest, you'd still make many of these errors. Enough to make up for the successes? Really, it's a wash based on an assumption that is only marginally more right than it is wrong. Complicating matters is that some pollsters already apply weighting to their numbers to try to take this into account, so for some polls you may be doubling the effect.

So where does that leave us? These sorts of discrepancies are already taken into account in ThreeHundredEight.com's projection ranges.

More broadly, it is certainly something which should be considered when looking at the polls in the final stages of a campaign. The conservative or centre-right party might indeed be under-estimated, and there is more of a chance of that happening than the main centrist or centre-left party beating its polls. But it won't happen every single time, and it isn't a rule.

If you assumed the federal Conservatives would beat their polls, you would have been right in 2008 and 2011, but wrong in 2006. If you assumed the Quebec Liberals would beat their polls (as virtually everyone does in Quebec), you would have been right in 2012 and only marginally so in 2007 and 2014, but you would have been wrong in 2008.

A more responsible approach for journalists would be to remind readers that this can happen, rather than assure them that it will. If polls and human behaviour were so predictable, no one would ever get it wrong.

50 comments:

That is a very interesting piece. I would have loved to see how the center and center-left parties fared, which you only briefly mentionned. Is only the center over-estimated as people don't like to be seen as extremists? Or is it really only the left? Is it because left parties are, generally, smaller than right parties or is it because they are on the left? Ok, some of those are harder to prove, but just comparing right-center-left over/underestimation would be fairly easy.

I don't think it's conservatives in general that are underestimated, but a group of parties (many of whom just happen to be conservative) that use a particular set of strategies and tactics in campaigns. They target their voters very narrowly, collect mountains of data, work to suppress opposition vote with specific messaging, and have very efficient GOTV (Get-Out-The-Vote) operations. Many on the left feel that if they just communicate their ideas better, they'll win because they have better ideas that will earn them better results, while generally those on the right wing understand that it's all about tactics, marketing, messaging and ground operation. It also goes beyond general elections, to include leadership races. In Ontario, Patrick Brown had a very efficient operation that got out the necessary votes to win. It didn't matter if Christine Elliot had more caucus support, was generally better liked, or was more likely to win an election. Playing strictly by the moneyball rules, whatever tactics would increase the number of votes they would receive, that's what they do. Any discussion of ideas, policy, likability or ability to govern are not really relevant to the calculation. They also understand that most voters choose based on emotion and not on any objective calculation. All of this not to say that these tactics aren't occasionally used by those on the left. In the 2008 primaries in the USA, Obama's team moneyballed the outcome by doing the math to determine how many delegates they needed for him to win, and only doing those things which increased his delegate count, ignoring all other aspects of the race. Hillary Clinton's team was so oblivious to this that they continued to insist (and believe) that she was going to win even after they passed the point where it was mathematically impossible.

The polls in Israel didn't underestimate the right. They just underestimated the distribution of the vote between the different right wing parties. The polls correctly called a 67/53 split between the right-religious-Kahlon block and the centre-left-Arab bloc.

Exactly! I was going to say the same thing. One has to sum up ALL right of centre verses left of centre parties. What happens with both right and left parties on election day is that the minority left or right party will lose votes as voters hold their nose and vote for their second choice to keep their least favourite choice on the other side of the political spectrum out of power.

In the case of the Conservatives in the UK, UKIP voters held their nose, and voted Conservative to keep Labour out of power. Had the polls shown the Conservatives with a majority, they would likely have not benefited so much from last minute switcheroos.

The Aberta elections are quite interesting. In 2012 we saw huge numbers of Wildrose supporters jump ship, but it's not entirely sure which ship they jumped to, Clearly many left of centre voters jumped to the Conservatives to keep out Wildrose. Ironically, the reason why the Conservatives won huge is that the left played the fear card toward Wildrose, which I think accounted for much of the WR drop in 2012. This time around was quite different. The WR did not drop much without the fear card being played, and I think likely only did because some figured that the Conservatives were the best bet toward keeping the NDP out of power. It would be interesting to compare the demographic makeup of the Conservatives verses the Wildrose.

I know that this probably isn't news, but the impact seems to be much greater for conservative incumbents. The average bump for the sixteen incumbent parties listed is +2.2 points; the average for the other fourteen is just +0.9. Do you know what the typical incumbency bump would be, Eric?

Interesting. That average obviously includes conservative incumbents, but it still seems like, after accounting for incumbency, the conservative bump is probably going to be smaller than 1.6 points on average, probably closer to a single point or even a bit less.

Yup, this was the first thing I was going to ask, because an incumbency effect is anecdotally just as strong as a shy tory effect, so it's interesting to see that the numbers are also similar. Moreover, incumbency is unproblematic even in politcal climates like Alberta's, whereas here the shy tory effect becomes quite jumbled, as you also pointed out.

It seems to me, then, that the number that matters more than +1.6 for all conservative parties is Scott's figure of +0.9 for those which are non-incumbent. Informally, that still looks significant compared to the -0.2 for the next largest nonconservative party, and indeed, since the overestimation may be even higher for non-incumbent nonconservatives, this factor might just cancel out. But I'd still love to see this included fully in the analysis, though.

When you look at the breakdown of accuracy for the Alberta election, those with internet elements performed better than those with telephone elements. I don't think the elderly voter posit needs be included as a basis of the "shy Tory" theory. Furthermore, shouldn't there be an effect between the centre-right parties like the APC and far-right parties like WR, where we have seen in the last two elections the opposite, were the central Tory party was underestimated and the far-right party was overestimated?

I would say this is much more a voter action effect, where people intend and say they will vote for an alternative party and then revert to their default at the ballot box due to whatever personal reasons. Rather than looking at left-right, it should be compared to leader-second, centre-fringe, or established-wildcard to see where the effect is greatest.

I think the 308 approach of allowing a wider variance in the reported numbers is the most acceptable, as without isolating an actual source of inaccurance, it's just an assumption that going to muddle the numbers. Enough assumptions and you might as well skip the polling and just present your biased opinion as the facts.

Taking off from your point about Éric's "wider variance"... certainly, attempts to quantify human behaviour to the degree one might desire in polling is ultimately doomed to failure... polling analysis can only ever be an approximation, albeit an often informative approximation. But creating a model that works in all circumstances is inherently impossible because there are new variables in every election. Fortunately, each election's new variables are seldom so significant as to unduly distort the picture that polling can provide. The exceptions, however, can be pretty striking.

It's interesting that in the 2012 Alberta election, there was relatively little information to contradict the narrative of the polls. Actually, there was one clue that the polls might be wrong, and that was the murmur of Liberal, NDP, and other party supporters pitching in their lot with the PCs to prevent a Wildrose victory. But that talk was largely dismissed by the mainstream media and its significance underestimated. Otherwise, partly because the election was not accompanied by any great buzz, there was no basis to contradict what the polls were saying until voting day altered the picture altogether. As Éric has repeatedly said, had there been more polling right in the last days of the campaign, the final results might not have come as a surprise. Contrast that with this year’s election, when there seemed to be a fairly broad acceptance that the polls were accurately telling the story. Some people refused to accept the polling results (I’m not sure whether to take Andrew Coyne at face value when even on election day he continued to declare there would be a PC majority), but that refusal seems to have been largely based on ideology, not on an honest appraisal of the situation. The polls seemed accurate, not merely intuitively, but because there was an obvious phenomenon on the ground. There was too much anecdotal evidence to ignore (without, again, resorting to an ideologically-based refusal to confront reality). In 2012 the combination of a lazy faith in polls and a complaisant assumption that Wildrose would run away with the election led to a fatal lack of polling in the last days. In 2015 the combination of a lazy lack of faith in polls and an ideological refusal to confront the pro-NDP groundswell left (certain) pundits burned once again.

Interesting post Eric. One of your best in recent times. You have addressed an issue that has been discussed on this site's comment boards for a long time.

It does seem that the centre-right vote is underestimated more often that it is overestimated. But when pollsters do try to address the "Shy Tory" or "likely voter" categories there are instances where it becomes inaccurate (i.e. Ipsos during Ontario 2014).

I find that center-right parties are often underestimated at the expense of not the main center-left party but rather smaller parties.

In the recent Alberta election, it seems the PCs were underestimated at the expense of the Liberals/AP/Other vote and not the NDP. In many other instances, the center-right vote is underestimated while the Green Party vote is overestimated.

Then there is always the notion of what defines a center-right party. Alison Redford's PCs in 2012 weren't seen as a center-right party and most of the 10.6% boost likely came from the center/center-left.

Also, there is often scepticism from pundits if the major center-left party can actually GOTV due to demographics. More often than not they do GOTV (with the exception of BC NDP in 2013).

I think Pratte is thinking back to the '90s when Quebec pollsters (particularly Leger) seriously underestimated the Quebec Liberals in both 1994 and 1998. Leger predicted PQ victories of up to 10 points, when both elections ended up effectively tied.

To overcorrect, Quebec pollsters fooled around with "repartage des indécis" for a while in the '90s, which assumed that undecided poll respondents favoured the Liberals 2:1. That made no sense and you never hear of them doing that anymore. The problem was much more likely was that they were poorly weighting non-francophones, a problem that has been largely been addressed in the last 15 years (also, the non-francophone vote is not nearly so uniform as it was in the 90s, so poorly weighting them doesn't have as great an effect).

I wouldn't be surprised if this effect was related to the difference in smaller (in terms of population) ridings voting tendencies. In Alberta and UK, specifically England, there seemed to be a divide between the Wildrose party and NDP and a similar divide with Cons and Labour in the England. I also remember this phenomenon occuring in past elections. I could be wrong, but I'm not exactly sure where to find the data on historical elections in a convenient form.

Eric, with your statement "Landslide victory, rather than a closer result, for the B.C. Liberals in 2005 and 2009" would like to comment on same.

During 2005 and 2009, BC had two long-time, prominent CATI pollsters which always published quarterly results. Year after year. Mustel and Ipsos.

And look at the error in the BC 2005 and 2009 results (-1.2 and - 0.2).

Yet prior to the BC 2013 election, CATI pollster Mustel abandoned the BC political scene after ~20 years.

And prior to the BC 2013 election, Ipsos moved from CATI to "opt-in" online panel polling akin to Angus Reid.

In that vein, on the last day of the 2013 BC election campaign, both "opt-in" online pollsters Ipsos and ARS chimed in with late polls. Both showed a similar 8% - 9% BC NDP lead.

No CATI pollster in sight. Of course, the error with these 2 "opt-in" online panels pollsters was +13% or so. Quite obvious that major recruitment problems exist in BC concerning these "opt-in" online panel pollsters.

Even IVR in BC has major inherent errors. May show "momentum" but certainly not accurate figures IMHO.

Nik Nanos, a good national CATI pollster, previously stated in an interview (2008 or 2011 election?) that BC is a very hard place to poll. I agree.

And "opt-in" online panel pollsters as well as IVR pollsters exacerbate that problem. Mark my words - the province with the greatest "error" in your final federal aggregated result, compared to the actual result in 2015, will be BC. ;)

First they denied the Alberta NDP were competitive, then they denied the Alberta NDP could win, then they denied the Alberta NDP would win a majority, then they denied the Alberta NDP victory would have any effect on the federal scene...

It will have a very marginal effect on the federal scene (one poll doesn't make a trend in any case, so don't count your chickens before they hatch). Canadians are smart they know the difference between federal and provincial politics. Had you looked at the Ekos poll you would note the NDP rise is mainly due to an increase in Quebec and Ontario not Alberta!

I'm not saying the Alberta election did not have an impact but, we have no evidence to suggest it did either. It is pure speculation.

I think the NDP election in Alberta is going to have an impact, but not necessarily in Alberta. The impact is it validates the NDP is a viable option to counter the CPC. Right now, the only one seemed to be the LPC, but if Alberta, the conservative stronghold, can switch to an NDP government, than all other provinces can consider that change too. I think that's the impact, not necessarily a sudden NDP sweep in Alberta.

But one note on the fortune of the federal NDP in Alberta. As Eric noted elsewhere, polls have shown steadily increasing support for the NDP in Alberta over the last few months. there needn't be a sharp rise of Albertan interest in the NDP for things to change on a federal level - and even if the NDP took just 6 or 7 seats in Alberta that would have an impact on the Conservatives.

Éric, From your table, let's assume that Conservative support is underestimated in October by their average of the last three federal elections (boosting their popular vote by 1.73%). How would their seat count change?

First of all, we need to wait and see if this is real. An EKOS poll of 1,362 people without results released four days afterwards? Assuming so, we then need to wait and see if it's a statistical anomality or a new trend. Beyond that, we need to see where the NDP surge is and if it's enough to actually win seats.

Comparing this leak to EKOS last few IVR polls, it seems like it's mostly coming from CPC (4%) and other parties (2%), but not the LPC (1%). I would guess that the LPC still rules the east, the CPC rules the west, and any NDP seat gains are either urban or BC/ON/QC.

Such a result at a general election would (most probably) still leave the Tories with a small plurality and it is the Liberals rather than the NDP who would be the main beneficiary with the NDP gaining only marginally in terms of seats. There are more Conservative-Liberal ridings than Liberal-NDP ridings or NDP-Conservative constituencies. Therefore while such a vote would be the best result in NDP history in terms of seats and position and second best in terms of popular vote it would leave them as the Official Opposition with the Conservatives as Government and the Liberals as a very large third party.

It was the preliminary report, I would suppose that was before the weighting (and they probably polled a bit more, since they have more than "about half of their usual sample", but closer to two thirds).

Speaking to the three wag race topic, who thinks a coalition will happen for the first time in Canadian Federal history? I think one just might happen between the Conservatives and Liberals.

I would also like to argue that even with the NDP leading the polls the Conservatives will most likely win more seats because they tend to receive very low support in urban constituencies (I.e. Southern Toronto, Hamilton, Ottawa & Montreal) most of these constituencies give Conservative candidates less than 10% of the vote, While their support is much higher in suburban and rural areas. When the Liberals and NDPs's support is more centralized everywhere.

That will be political suicide for the Liberals! Why would they prop up a government that their base strongly dislikes?

Forget a coalition, the Liberals will lose a massive amount of support to the NDP, even if they prop up Conservatives in a minority parliament. The federal Liberal brand will be tarnished for a generation. The NDP will become the de facto center-left party.

It is very unlikely a coalition will happen (some may argue there was a coalition between English Canadian Liberals and Conservatives during the First World War).

The Liberals are a small "c" conservative party at base; constitutional monarchy, status quo Senate, Bay St. buddies etc...they share few policies with the NDP, I can't think of a single one but, others may have a better memory than I.

Most importantly Liberals will not want to make the NDP a "governing party" by giving them experience in government. Doing so would re-inforce and legitimise the split in the left-of-centre vote and the Liberals would in essence abandon their position as Canada's natural governing party and alternative to the Conservatives.

In BC for years people spoke about the 30 second Socreds - the NDP would clearly more popular for years and then on election day the Socreds would win again.

Including the BC Liberals and not all the others Liberal parties is can only be done because BC does not have a right wing party of any substance. The policies of the BC Liberals are no more right wing than the Chretien/Martin government or other Liberal parties in the country.

Yes, but the Chretien/Martin governments were quite far right in terms of economic policy - promoting privatisation, deregulation, sharp reductions in social programmes, etc. The BC Liberal government is, in fact, probably to the right of them - and why not, they are the successors to the Social Credit legacy in most respects.

The BC Liberals I would agree, are much like other Liberal parties across the country (Most of all the Ontario Liberals) but they still remain independent from either the Federal Conservatives or Liberals. The BC Conservatives are a Social Conservative Right-Wing party in BC and it looks like if they play their cards right they could win a seat or two in the next election.

The BC Liberals are the old Socreds by a different name. Most left of center voters view the BC Liberals in the same light as the Federal Conservatives, especially in rural ridings. The left flank of the BC Liberals jumped over the NDP after they were taken over when the Socreds jumped their own ship over the BC Liberals.

The Charest QLP was center-right, but under Couillard, it has taken quite a drastic right turn, they are pretty much sitting in ADQ/CAQ territory at the moment (which, I guess, for other provinces would fit right it with the Conservatives).

That's because there isn't really a Conservative Party in Quebec, I guees Le Coalition Avenir Quebec would be the closet to a Conservative Party. It can also really depend on who is leading the Liberals in Quebec like you mentioned.

The PQ are arguably social (nationalist) conservatives. In Scotland the SNP are called Tartan Tories and all nationalist parties have a bent of conservatism within them. After all they all want to conserve something; language, culture, social or ethnic group, way of life etc...

"Tartan Tories" is an insult thrown at the Scottish National Party by Labour. In Scotland such is the potentency of the word "Tory" that it gets slung back and forward.

The SNP is a complex party that has combined Scottish civic nationalism with various positions on the political spectrum. It was originally formed from a merger of a left wing and a right wing party and traces of both can be found amongst both the platform and ideology. Its growth in the 1970s coincided with the decline of the Conservatives (called Unionists until the mid 1960s with a separate party organisation) in Scotland with the SNP taking over chunks of the old Conservative vote, particularly in the north east. I think it was in this period that Labourites coined the term "Tartan Tories".

In recent years the SNP has broadly settled as a left-wing, anti-austerity party but it's had its dabbles with neo liberalism at times when countries like Ireland seemed the model to aspire to.

One of the problems I'm seeing in this country is how to actually see what a party is ??

It used to be that the Conservative party was definitely right on economic policy but distinctly barely right of centre on social policy. The liberals where centre right on economics and distinctly centre left on social policy. The CCF/NDP were hard left on social policy and distinctly left on economics.

With the exception of the current Reform/Conservative party which is hard right on everything the the rest are hard to place or pinpoint.

What is quite clear though is the the Canadian public is still basically centrist. So how to gain their votes is questionable.

Interesting, but it only measures the polls the last week of the campaign, by then most pollsters are upping their right totals, a better comparison would be to do the analysis, at least 10 days out, 14 is probably better, by then most people have made up their minds and you only have a marginal few in the last week deciding.

Anecdotally, I seem to remember in every election the pollster revising the right vote up in the last week.

I like most of your methodology in this study, but see one key flaw: When there are two centre-right parties in the same election, you treat them as two separate results. In the upcoming election, the centre-right vote will not be split. If we unite the centre-right results of each election, we end up with 24 separate election results, with six over-estimations, one right on, and 17 underestimations. This shows that a united centre-right is underestimated 70.8% of the time, and overestimated only 25% of the time. It also shows that as a whole, the centre-right outperform the polls by 2% on average.

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.

Details on the methodology of the poll aggregation and seat projections are available here and here. Methodology for the forecasting model used during election campaigns is available here.

Projections on this site are subject to the margins of error of the opinion polls included in the model, as well as the unpredictable nature of politics at the riding level. The degree of uncertainty in the projections is also reflected by the projections' high and low ranges, when noted.

ThreeHundredEight.com is a non-partisan site and is committed to reporting on polls responsibly.