High Noon for pollsters in Ontario

It is hard to convey the ineffable excitement, even joy, a pollster experiences when opening up data tables from a fresh poll in the midst of an election. It’s like gazing into a shimmering orb whose insights are, for the moment, yours and yours alone. The eye races from number to number — men, women, seniors, the young, this region or that — each revealing some special insight into the mystical truth of the campaign.

And that thrilling moment, along with a pride in authorship, gives a pollster enormous confidence in the results. Even, it seems, when other pollsters are gazing into their own orbs and seeing something quite different.

The current Ontario election campaign has been striking for the degree to which different polls and pollsters have presented us with divergent narratives.

At EKOS, Frank Graves has seen Kathleen Wynne’s Liberals in a clear lead from the beginning — a gap he saw widening on the eve of Tuesday leader’s debate and narrowing slightly today. Here at iPolitics the other day, he gave a very robust defence of polls in general — and his own in particular — saying it was “nonsense” to suggest, as some political scientists have recently, that the polls are unreliable and misleading to voters.

Right now, if you average all the polls from all the pollsters, the Liberals appear to be in the lead — though by a smaller edge than EKOS gives them.

But here’s the thing: One of Canada’s leading polling outfits, Ipsos Reid, is telling a very different story.

From the start, Ipsos Reid has had Tim Hudak’s Progressive Conservatives ahead. Although it has shown the race tightening slightly lately, if anything Ipsos Reid seems more confident than others in the predictive power of its results. It has said that when it looks at “likely voters” — those who will probably show up to vote — the margin for the Tories is much larger.

After the debate the other night, which Ipsos Reid said Hudak clearly won, it hinted strongly that this should allow the PCs to increase their lead.

As Abraham Lincoln once remarked when contemplating the fact that both sides in the Civil War claimed to have God on their side: “Both may be, and one must be, wrong.”

(I should make a full disclosure here. I used to work for Frank Graves at EKOS. I also worked with the head of Ipsos Reid, Darrell Bricker, when I was a reporter at the Globe and Mail and he was the paper’s pollster.)

It may be that in the last week of the Ontario election, the polls will converge to a general consensus. If they don’t — or if they do, and the consensus turns out to be wrong — it could be another black eye for an industry already under attack.

Three issues come to mind when I try to evaluate the difference among the polls: one historical and two methodological.

Most polling organizations have so-called “house effects”, meaning that they tend to favour one party or another compared with the average of all the polls. At the federal level anyway, the poll aggregator Éric Grenier has calculated that EKOS has a tendency to show the Liberals somewhat stronger than other pollsters do. Ipsos Reid has an even stronger tendency to favour the federal Conservatives.

As a general observation, I’d say that when EKOS comes closest to the final result in an election, it is often when the Liberals do particularly well, and when Ipsos Reid outperforms the others, it is when the Tories run strongly. When they “blow” an election, as each of these pollsters have done from time to time, it is usually in the opposite direction.

So let’s just say in the Ontario election, both EKOS and Ipsos Reid seem to be on form.

Now, methodology.

We are in a period of great — some might say “wild” — experimentation with new polling techniques. Not long ago, the gold standard in polling was the phone survey conducted by human operators following a computer-generated script.

Now, almost no one is conducting such polls, largely because of the cost. Both EKOS and Ipsos Reid are using relatively novel methodologies and they are different from one another.

EKOS is using Interactive Voice Response (IVR) — so-called “robocalls” — in which a recorded voice leads you through a survey, and you punch in your responses on the keypad of your phone.

Ipsos Reid is using an online method in which people who were previously recruited to join a “panel” are polled via their computers.

Oddly enough, and for reasons I am helpless to explain, when you look at all the pollsters active in the Ontario election, those using IVR have tended to favour the Liberals, while those using online polls have tended to show the PCs running more strongly.

Myself, I give IVR more credence, but perhaps that’s because I worked with it at EKOS. The Gallup organization in the United States recently said tests they conducted suggested phone surveys were more accurate than online — but they were using traditional live operator methodology, not IVR.

The truth is that both IVR and online have had some conspicuous successes and spectacular failures when used by various pollsters both here in Canada and abroad. And different companies may handle the same basic technique with more or less precision or skill.

Then we come to the question of likely voters. With turnout falling in Canada in recent years, it has been increasingly important to assess who among those polled will actually vote. Many analysts think pollsters blew the last B.C. election because they failed to adjust correctly for turnout.

In the recent Quebec election, Ipsos Reid nailed the turnout — which presumably is what gives them such confidence in Ontario.

But Gallup, which got badly burned in the 2012 presidential election in part by using a turnout model that had worked for them for half a century, has now begun some intensive tests. Their conclusion? Sometimes these models work much better than others.

It may be that in the last week of the Ontario election, as sometimes happens, the polls will converge to a general consensus on the likely winner.

If they don’t — or if they do, and the consensus turns out to be wrong — it could be another black eye for an industry already under attack.

If there’s some hope for us all, it is that the polling industry is trying to learn from its mistakes — maybe not quite fast enough for those of us trying to make sense of all this.

Follow Paul Adams on Twitter @padams29

PaulAdams is associate professor of journalism at Carleton and has taught political science at the University of Manitoba. He is a veteran of the CBC, the Globe and Mail and EKOS Research. His book Power Trap explores the dilemma of Canada’s opposition parties.

The views, opinions and positions expressed by all iPolitics columnists and contributors are the author’s alone. They do not inherently or expressly reflect the views, opinions and/or positions of iPolitics.

17 comments on “High Noon for pollsters in Ontario”

Terrific piece Paul. My own doubts about polling especially in the US is that they have often been less than comprehensive (usually for reasons of cost-cutting), they don’t take into account the decline of landline use, and they are unable to weed out their own biases. Oh, and people increasingly lie to pollsters just to mess with them.

I have noticed how small the sample sizes are for some of these polls. Considering that they are polling a province with millions and millions of people, they usually only do a poll of 1,000 for the whole province, with very small sample sizes in the region. And Nanos usually only does polls of 500 people.

“Many analysts think pollsters blew the last B.C. election because they failed to adjust correctly for turnout.” It was the voters of BC who decided the election, not the experts whose polls were not accurate.

This polling obsession has become epidemic. Both sides of the pollsters claim victory, all done with different methods. I may be a layperson when it comes to the methodology of polling, but what I see are private consultants trying to decide the outcome of an election which, in my opinion, sets a bad precedent, as data can be manipulated to show anything.

We are inundated with enough political advertisements from parties and 3rd party advertising, never mind the latest poll. Having polls come out almost daily, pounced upon with breathless anticipation by the press, takes away from what the real issues are for voters and when they see that Party “A” is ahead of Party “B”, they are led to believe that a vote for their candidate does not count.

I believe the polls that say the media and Gotcha! Journalism is in large part responsible for the decline in voters and public participation in the democratic process. Also the one that says 75% of respondents do not trust the media’s ethics or honesty and in fact rate it below politicians and bankers. Now there’s some polling for you!

Paul: Excellent commentary on the state of polling today. Your observation on Ipsos tending to strongly favor Cons while EKOS tends to favor Libs agrees with my own perception.

A few comments:

(a) in regard to the Ontario election, Abacus which also uses the online, pre selected panel model as Ipsos Reid just reported the Liberals being significantly ahead among eligible voters, which agrees generally with all the other pollsters who have reported (EKOS, Forum, Nanos): http://abacusinsider.com/ontario-election-2/liberals-lead-tories-7-closer-race-voters/ This leaves Ipsos as the ONLY pollster to consistently claim a significantly lead for the Cons among eligible voters (there is agreement among all pollsters that the gap between the Cons and Libs are much closer among likely voters). Therefore, one cannot just say that it is due to different methodologies as both Ipsos and Abacus use the same general online panel model,

(b) it should be clear that having a pre selected, online panel is not the same as polling randomly, therefore, it would seem intellectually dishonest for polls using the online technique to somehow claim that they are reporting a truly representative/random result (even if they somehow claim that they can select “randomly” from their pre selected panel). It is even more of a stretch for polls using the online pre selected model to calculate margins of errors (MOEs) as if their samples had been obtained from a truly randomized sample. Hence Abacus had used what some would consider “weasel” words by putting in a disclaimer that says that the MOEs would have been such and such if the sample had been truly randomized. Most people and media, I suspect, would not have read the small print, and likely would not have understood its significance even if they did, and
(c) theoretically, at least, the IVR method as practiced by EKOS, or telephone polling (by Forum, I think) have the potential for reporting more truly representative/random polls than the online pre selected model. Of course the non online methods have to satisfy other important criteria as well (e.g. adequate sample size, competent/unbiased analysis, etc.). EKOS, for instance, appears to have made an effort to reach out to both cell and landlines.
In summary, it seems to be indeed high noon between the pollsters (apparently between Ipsos on one side and all the other pollsters on the other), unless they all converge in the remaining days of the Ontario election. Can badly carried out polls still forecast a correct result? Of course, even stopped clocks can be correct twice a day, no?

It’s funny how everyone goes on about how the polling environment has changed and we have to use new techniques to be accurate. Hogwash. The only polls that are accurate are the ones done the traditional old expensive way. The cheaper the polling costs, and many of these are done for free for the publicity, the more they are just junk masquerading as a scientific poll.

There is nothing wrong with polling the old way, with landlines and real people asking questions and calling back the same household over several nights. It works, as Christie Clark can attest. Traditional polling was within 1% of all the parties results in that election.

I’m no expert on polling, but I do know one thing. The pollsters in general, and in particular Graves/EKOS, were wrong on the last federal election and spectacularly wrong on the recent BC and Quebec provincial elections. There is ample reason to think they’ll be way off again in Ontario. We shall see.

The advent of a push polls, call display/screening, cell phones and lack of clarity with respect to voter intentions vrs likely voters has so muddied the waters that public polling is nearly useless. The parties, however, do more rigorous internal polling so they have a better idea of what the voters intend, so look to them for cues. In any event, most voters don’t become engaged until after debates, close to the election day.

Competing polls used to coalesce nearer election day and more or less mirrored the actual results; not so anymore IMHO.