Tuesday, November 26, 2013

By-election post-mortem

The four federal by-elections last night played out as ThreeHundredEight forecast, with the Liberals retaining Bourassa and Toronto Centre, the Conservatives retaining Provencher, and Brandon-Souris being a toss-up riding that leaned slightly towards the incumbent Tories. So slightly, in fact, that the Liberals were less than 400 votes short of winning it.

By-Election Barometer Record

This makes 22 by-elections without a wrong call by ThreeHundredEight, going back to June 2012. The Barometer will be put to the test again tonight in Carbonear-Harbour Grace in Newfoundland and Labrador. The model suggests the riding is leaning Liberal, though the PCs do have a slim chance of holding on to it. However, the forecast is for the Liberals under new leader Dwight Ball to take it by a narrow margin.

The results in Brandon-Souris, which were the most at-odds with the polls by Forum Research, showed the mettle of the By-Election Barometer. The forecast was that it was a toss-up between the Conservatives and Liberals, with the Tories considered to have a 52% chance of winning to 48% for the Liberals. In the end, the Tories won it by a single percentage point.

This was similar to the by-election this summer in Ottawa South, where the polls suggested the Tories were well-placed to steal former premier Dalton McGuinty's riding away from the Liberals. The model still considered it a toss-up that leaned slightly towards the Liberals, which turned out to be the case. It demonstrates the need to take into account the fundamentals in a riding, particularly when riding-specific polls are showing numbers that are wildly out of step with historical performances.

I assessed the performance of the polls for The Globe and Mail, and the results are not great for Forum Research. They missed Brandon-Souris by a wide margin and Provencher was off by a considerable degree as well. The polls were better in Toronto Centre and Bourassa, however. This record matches their performance in the Ontario by-elections this summer, where I gave them a score of 2.5 out of 5. Last night, they got 2.5 out of 4, for picking the right winner in Provencher but missing the respective levels of support for the Tories and Liberals.

Because of this very mixed record, I employed confidence intervals far wider than the reported margins of error of these polls. This turned out to be a good idea, but only captured some of the error. Of the 20 calls Forum Research made (one for each party + others in the four ridings), 60% fell within the 67% confidence interval and 85% fell within the 95% confidence interval. Clearly, these wide ranges will need to be stretched even further, but that is the usefulness of these extra data points.

The performance of the confidence intervals, based solely on Forum's polling, compared to the results can be seen in the chart above.

The misses at the 95% confidence interval were primarily in Brandon-Souris. The results for the Liberals and Conservatives were so off that a +/- 10-point confidence interval still did not capture the result. The Greens in Provencher also fell outside of the 95% confidence interval.

At the 67% confidence interval, only the Others in Toronto Centre and the Conservatives in Bourassa were wide of the mark. In Brandon-Souris, the Liberals and Conservatives fell outside the interval, while all parties fell outside of it in Provencher. In short, even compared to Forum's past errors in by-elections the ones in Brandon-Souris and Provencher were extraordinary.

Note: I inquired with Forum about their methods for Brandon-Souris in relation to the reports of multiple calls being received by some residents. I was told by Lorne Bozinoff that they used the same random dialing in all ridings, and that no panel of phone numbers was used to dial the same people over and over again. I was also told that if a number was called in consecutive nights, that person's results were dropped from the sample and that all three of their final polls (Forum also polled on Nov. 23, but did not publish the results which were similar to the poll of Nov. 24) were independent samples.

I think these by-elections clearly demonstrate the huge importance of the parties' get-out-the-vote ground game and how pollsters will be very hard-pressed to accurately predict results when the parties aren't equally competitive on this front.

Clearly the CPC focused all of its efforts on the MB ridings, particularly Brandon-Souris (as evidence by the highest voter turnout among the 4 ridings) which they were scared to death of losing. The Liberals had a strong showing in both ridings but clearly didn't come close to matching the CPC's ground game, hence the former's support was severely overestimated and the latter's severely underestimated in the polls.

The second highest turnout was in Toronto Centre, where the polls came decently close, probably because both the Liberals and NDP invested equally strong efforts in their ground game there.

Finally, the polls were also fairly close in Bourassa, where again I suspect that the Liberals and NDP invested roughly equal efforts, though less overall than Toronto Centre (as evidenced by the dismal turnout) because it was probably deemed by both parties as a safer Liberal riding than Toronto.

Also note that the CPC underperformed in relation to the polls in Toronto and Montreal, probably as a result of them investing little to no effort in these ridings.

On the one hand, I believe they were the very closest to predicting the result of 2011 Ontario election; they were the closest to predicting the 2012 Alberta election and the only pollster to register any hint of a last-minute closing gap between the PC and Wildrose; and they were the closest to predicting the result of the 2013 BC election and the only pollster to register the Liberals ≥40%.

But on the other hand, they mistakenly registered a surge in PQ support putting them in majority territory at the end of the 2012 Quebec election campaign, and their cumulative record in all of these by-elections is quite sketchy at best.

Their federal polls also seem to exhibit more volatility and wider ranges in party support than other pollsters.

Yes. I wonder how much of this has to do with the frequency of polling. They are active in virtually every provincial and by-election campaign, so they come out good in some and bad in others. Other firms are active in only a few campaigns, so it is difficult to know if they would do better or worse if they were just as active. What other firm was actively polling in NS, BC, QC, AB, and ON? None.

Harris-Decima, for example, is an excellent firm. But they haven't been active in a campaign since the federal one in 2011. How would they have performed in BC?

"Throwing darts" is a little extreme. Forum did register the surge of Liberal support in all the ridings, and did a decent job of predicting the results in several of them. We shouldn't neglect to accounts for the vagaries of by-elections and the importance of political organization, which the Liberals are only beginning to reconstruct out West. It's certainly paltry relative to the Blue Machine.

Plus, a few darts is better than no darts at all. None of the other polling firms dared to jump in publicly as far as I know. Who knows - maybe they would have done worse (or similarly). Personally I had hoped that Ipsos-Reid would have done something here. Their federal results, which are widely publicized, have been consistently and considerably different from what virtually all the other polling companies have been showing for a while now.

I have heard some kerfuffle about people being called multiple times by polling firms. I do not know that much about polling, but I know a lot about trying to reach people by phone. When one small riding is being polled so many times over a short period of time, randomized calling will produce a lot of duplicate calls. You typically have to dial 4 times to reach a live person. You might get as many as 10%-20% of those who answer their phone who are willing to answer your questions. So every poll that wants to hit a minimum of say 400 respondants has to dial between 8,000 and 16,000 numbers, (depending upon 'conversion rates' being 10% or 20%). You can see how if only a single company was polling, and they polled the riding 3 or 4 times, pretty well every telephone in the riding will ring at least twice. (assuming approximately 30,000 households)

So the moral of the story is that if you set the bar low enough you will succeed.

I predict that in 2015 the pollsters will show that the Cons will be ousted by an unholy alliance of Green, NDP and Liberals.

Using this predetermined information I predict that due to the polls being wrong by +/- 30% there will be 338 toss up and the incumbents will be narrowly favored in the 308 riding they currently hold and the Cons will be mildly favored in the 30 new riding's.

As I have clearly stated that I am not confident in any of the polling results ...... and have clearly predicted 338 toss-ups that someone will win I am mathematically certain that I will be 100% correct

There’s no spinning it — the Conservatives were spanked in Monday’s byelections

Andrew Coyne, today's National Post. Sometimes the manner"of winning is as important as the win. In these cases the Tories got hammered, even in seats they held. Plus big falls in the other two seats. "Manner" of win or loss matters !!

COMMENT MODERATION POLICY - Please be respectful when commenting. If choosing to remain anonymous, please sign your comment with some sort of pseudonym to avoid confusion. Please do not use any derogatory terms for fellow commenters, parties, or politicians. Inflammatory and overly partisan comments will not be posted. PLEASE KEEP DISCUSSION ON TOPIC.

Details on the methodology of the poll aggregation and seat projections are available here and here. Methodology for the forecasting model used during election campaigns is available here.

Projections on this site are subject to the margins of error of the opinion polls included in the model, as well as the unpredictable nature of politics at the riding level. The degree of uncertainty in the projections is also reflected by the projections' high and low ranges, when noted.

ThreeHundredEight.com is a non-partisan site and is committed to reporting on polls responsibly.