Superforecasting: How I called the 2015 Election wrong

By Richard Forbes.Featured Image by Mark Blinch.

Nik Nanos humbly bit into a hat-shaped chocolate cake this week on CTV’s Power Play. The host, the colorful Don Martin, belly laughed through the segment. Nanos, one of the country’s most respected and trusted pollsters, had made a wrong prediction earlier this year when he had said he would eat his hat if Trudeau saw his preferred Prime Minister numbers go higher than they were this December when they stood at 53.6%. Five months later, however, Trudeau’s numbers have surpassed that number. 53.8% of Canadians now prefer Trudeau as Prime Minister versus the alternatives. It was a minor error – a splotch of subjectivity and common wisdom that perilously deviated too far from the story that the numbers have been saying about Trudeau for months: people like him a lot. Moreover, the specific question of “preferred Prime Minister” is vulnerable to changes in opinion regarding not only Trudeau, but his competitors, Ambrose, Mulcair and May, which makes it a more complex unknown to predict.

But Nanos’ mistakes pale in comparison to FiveThirtyEight poll analyst Nate Silver’s, who penned a contrite editorial last week titled “How I Acted Like A Pundit And Screwed Up On Donald Trump.” Silver, who has been under fire recently for having dismissed Trump’s chances, explained in the recent FiveThirtyEight piece that he had ignored available data to fit his pet theory, shared by many a pundit, that Trump just simply could not become the Republican nominee. Silver’s mistake, to act like a buffoon from the same papers now relishing the opportunity to criticize his methodology, has helped to fuel the myth of Donald Trump’s ‘impossible’ defiance of the odds. The odds, however – as astonishing as it is – were never particularly against Trump.

All across America, pundits have relished this opportunity to spill some ink against the reputation of Silver and fellow analysts; this was their chance to assert the supremacy of good old fashion punditry over data journalism.

Standing squarely on the side of data in this emerging debacle are the authors of Superforecasting: The Art and Science of Prediction.

Superforecasting, a recent book from two Canadian-born authors, Philip E. Tetlock and Dan Gardner, offers a new take on the art of forecasting events; Tetlock and Gardner explore just what it is that makes a forecaster better than average – a closer look at the skills and values of a forecaster and a profound philosophical discussion on their limitations. The book gained some much deserved attention in Canadian circles with the praise it’s received from Gerald Butts, the Prime Minister’s Principal Secretary, in addition to Dan Gardner’s fresh new appointment as an adviser to the Prime Minister. Gardner’s personal website announced this February, “Dan is being asked to put the methodology described in his book, Superforecasting, to work for (the) PMO.”

If Superforecasting is to be regarded as the working bible of the latest Trudeau administration it says quite a lot about the values of the Liberal braintrust now working in Langevin Block: numerate, detail oriented, measured, open-minded, flexible, collaborative, evaluative – “dragonfly-eyed”, as Superforecasting calls it – even if it risks being labeled by critics as indecisive and wild-eyed. Better Team Dithers than Team Stupid, would be a mantra worthy of a superforecaster. The book is aimed at the average New York Times reader, relatively well informed but not necessarily an elite by any means, the typical news junkie – and that target audience is no coincidence given the book’s superforecasters are an example of average folks whose careful eye on the news out-predicts leading experts; Superforecasting imparts on its reader with a promise that, with careful attention, they too can become better forecasters.

In many ways I expected a Malcolm Gladwell book – light on profound arguments, heavy on contradicting anecdotes – but I found Superforecasting to be far more concerned with conveying better skills of judgment, in a way where its success and confidence could be measured, than the usual easy-reading trifles that are peddled as bestsellers these days. Superforecasting never hesitates to point out to readers that they should be skeptical of its own argument; it’s keen, not simply to broach a subject in a light and entertaining way, but rather advance a defense of a new mindset that is thoroughly, never superficially, convincing. Superforecasting encourages its readers to predict only specific, measurable events and consider these events, not as unique circumstances, but as extensions of precedent, modified by context – the superforecaster breaks questions into sub-questions, measures doubt with numeracy and adapts to changing circumstances with a correction that is proportional to its significance.

But as I read Superforecasting, I couldn’t help but to continually reflect on its material with the 2015 Federal Election in mind – could superforecasters, using this book as a model, really have called the election correctly, I wondered? What it means to “call an election” is of course, vague and open to interpretation – it wouldn’t have taken a superforecaster to predict Justin Trudeau would win on October 18, the day before the election – and indeed, Superforecasting encourages its readers to continue to re-evaluate predictions as events unfold. But could a superforecaster have called the election correctly, say, three months out, when the election began – like our Prime Minister did? Or at the beginning of the year? That kind of judgment isn’t necessarily what Superforecasting promises its readers and, indeed, I hope this article engenders some well deserved skepticism on that point – the 2015 Federal Election was a terribly unpredictable political event, anyone suggesting that it could have been predicted using a neat and tidy little model should be met with considerable scrutiny.

And yet, despite the significant difficulty in predicting such an unpredictable event, that hardly stopped myself and many others from trying…

It was the tail end of last August when Justin Trudeau heard the news of Mulcair’s announcement with glee; the NDP had just committed to running a balanced budget for the next four years if elected.

During an interview with Bloomberg, Trudeau says he knew right then and there that the election was his. Many people attribute Mulcair’s decision and Trudeau’s decision to outflank him on the left with Trudeau’s rise to power. I’ll admit, I’m skeptical of whether that’s really the true narrative of what happened this fall, but what’s key here is the Prime Minister made a prediction that he would win – a prediction that many did not make, especially not at that precise moment. The Toronto Sun infamously ran a front page with the headline, “Political Suicide” the day after Trudeau announced he intended to run a series of deficits once in office. They’ve been eating those words since October.

Justin Trudeau announcing his intentions to run a series of deficits.

As a former Political Science student, I get asked during election season more about the odds of elections than I do about the weather. It’s the obvious conversation starter (3-1 odds for President Trump, before you ask.) If you had asked me how I thought the election would go, around, say February, I would have given you a rehearsed spiel: “Well, no matter what happens, it’ll probably be a bad night for Trudeau,” a weaselly way of avoiding choosing a definite winner between the NDP and the Tories, “and, I think Harper will defy the odds and sneak in another majority.” This prediction, despite being couched in fluffy language like “probably” and “I think,” was still horribly wrong – the actual election result was virtually the exact opposite of what I predicted happened. I’m not sure I could have been more wrong on any measurable basis.

Indeed, I had erroneously rejected lots of positive approval ratings and favorable polling for Trudeau from 2014 on, what was effectively, a hunch that Trudeau would struggle in a real election because I regarded his oratory approach to be amateurish compared to Harper and Mulcair; I also was skeptical, despite some fairly negative polling, that Harper’s opponents would actually be able land a proper punch on him in an election because a number of the issues they had run with previously felt like they were more concerned with process rather than outcome – sleazy cheques, cooked books, election fraud – and I simply wasn’t convinced that those kinds of campaigns had resonated with voters in previous campaigns. Canadians have demonstrated an incredible ability over the years to turn a blind eye to an incumbent’s shady tactics and strong arming in the spirit of the horse race.

All in all, my bullish prediction dismissed most empirical data and put an overly heavy emphasis on Harper’s most recent polling numbers, which were more positive after the attack on Parliament Hill, while dismissing Trudeau’s chances especially – ignoring the available evidence for a pet theory was my great sin here, and it allowed me to entertain some wild theory that Harper would defy expectations like David Cameron did in 2015 (and yes, I was prematurely vindicated when Lynton Crosby, who oversaw Cameron’s win, supposedly began advising Harper in September.)

I expected Stephen Harper to drag out the old sweater vest from the closet, blow off the dust and reassuringly hand-wave any and every problem, from climate change to gender gaps, with a “Canada leads all other G7 nations in job creation” boilerplate lullaby to tranquilize his supporters and detractors alike. But this election was no sleepwalk – and that wasn’t the only mistake I made in predicting the election.

Back in February still, I was taking classes at Waterloo – when my class was asked by the professor whether they believed Stephen Harper would call a spring election prior to the Duffy trial (beginning in May,) everyone raised their hand with the sole exception to me. I fretted. Was I out to lunch, I wondered? Everyone else seemed certain Harper would call a snap election – and for good reason, nearly every pundit and every whisper on the Hill was suggesting just that.

Professor Esselment furrows her brow: “why not?” she asks. I went on to explain, now feeling the pressure of being on the hot seat, that I didn’t think the trial itself would matter in the minds of Canadians months after the trial concluded given Canada’s usual political amnesia. On the contrary, I suggested Harper would run a “summer election,” beginning around July 30 and concluding right before labour day – I cited Charest’s near win in 2012 as an example of how an election run in the sleepy “dog days” of summer would benefit an incumbent: minimizing attention to the election and reducing voter turnout (i.e., cutting the cottage-goers from the party bases.)

Pressed to explain how Harper could justify such a move, I glibly said Harper could use any half-baked reasoning he liked, including some nonsense about “doing an election before the start of parliament.”

As it turns out, the class was wrong, Harper didn’t call the election in the spring – he called the election three days after I expected him to, August 2. His reasoning? Even more tortuously fallacious than the one I came up with in February (some crap about the election costing less.)

But that doesn’t mean I was correct, or that my reasoning was sound.

For one thing, Harper didn’t call a short 36 day election as I expected, he called the longest election in Canadian history since 1872: 78 days. Calling a short 36 day election, as I had expected, would have meant that voters would go to the ballot box in the thick of the Mike Duffy trial (if it hadn’t have been recessed on August 25) and at the height of the NDP’s support. Two unexpected events in May led to these changes in circumstances: Notley pulled off a historic win in Alberta, resurrecting Mulcair’s chances, and the Duffy trial was delayed til August 12. Calling a long campaign allowed the Conservatives to ride out the storm, or so they thought; the Duffy trial would have long concluded and the NDP, as the frontrunner, would be scrutinized more closely for months til October 19.

I would also dispute my reasoning for my prediction. Even though I turned out to be correct in that Harper didn’t call a spring election, I had made a common error in my reasoning. The authors of Superforcasting warn readers against supplanting easy questions for hard ones. I was asked “will Harper call a spring election?” but I answered the easier question, “would you call a spring election if you were Stephen Harper?” – my personal thoughts ended up being closer than our understanding of Stephen Harper but it was still, nevertheless, a careless prediction. By all accounts, Stephen Harper was risk averse and ran a frightfully paranoid PMO – short pants boys running around like decapitated chickens at the sound of every minute controversy – my prediction, although sharing a similar cynicism as the Conservatives, differed in its approach to politics to Jenni Byrne’s “hillbilly Malcolm Tucker on amphetamines” approach. Not overreacting to the threat of a Duffy trial, as I suggested, was out of character for an administration that had prorogued parliament for less.

In not considering the character of the Harper administration, I failed to consider its faith in spending to deliver results – long boasting about the party’s wealth and the size of its “battle-chest” – I’ve never been a big proponent of “election spending equals election results”, you can burn a lot of money supporting a dead campaign, but Harper and his party insiders had long believed that the strength of their war machine was dependent on its ability to buy airtime and carpet bomb the airwaves with negative advertising and attack ads. This contrast helps to explain why I would have run the shortest campaign possible and Harper would have run the longest campaign possible, yet we would have made these diverging calls with the same intention: to minimize risk to our respective campaigns.

Stephen Harper and his son talk behind the scenes at the Maclean’s debate. Gerald Butts talks with Justin Trudeau, top left. (Photograph by Liz Sullivan)

But nevertheless, there’s still the nagging question in my mind whether a superforecaster really could have used the available evidence and predict an event like, whether Stephen Harper or Thomas Mulcair or Justin Trudeau would be Prime Minister after October 19, before the election began. I believe the answer, despite Superforecasting’s strengths, is no. Indeed, a quick glance at the Good Judgment Project, a website promoted by Superforecasting, shows forecasters’ predictions were favoring the Conservatives until October 14. Although, I did find it odd though that the Good Judgment Project began its predictions in late September, more than half way through the election – the exclusion of the rest of the election would likely save any forecaster some face.

Superforecasting would suggest, when making such a prediction, to begin by considering the precedent. For example, I could have found, if I had bothered to dig up the numbers, that incumbents in Canadian federal elections win 60% of the elections they challenge (I found no significant difference in terms of how long the incumbents had been in office versus their chance of re-election.) If the incumbent has a 60% shot at winning any given election, it follows then that the official opposition has a 37.5% chance of beating them for the win, after factoring for the slim chance of a third party win. I calculated a third party’s odds of winning at 39 to 1, 2.5%, on the basis that only one time in Canadian federal history has a third party jumped from third to first in an election: Arthur Meighen’s Conservatives in 1925. Of course, as fate would have it, 2015 would once again favor a third party despite the odds.

With these odds, nonetheless, we could have modified them to consider the probability of a win according to the latest polling averages (however, it’s worth reminding readers: polling only provides a “snapshot” of a public’s mood at the time of polling, it’s not meant to have much predictive value.)

Such a forecast model would predict either a Conservative or an NDP for most of 2015, let alone the election – the Good Judgment Project’s average trend followed similar numbers in the election’s final days but did a better job at dismissing the NDP’s chances of a win – to better this makeshift forecasting model I’ve proposed, along the lines of what the Good Judgment Project’s participants were doing, I could have continued to increase the significance of current polling data as it neared election day – that would provide results that better matched the Good Judgment Project. Regardless however, that still leaves us with a forecast that would dismiss a Liberal win until, at least, October – and I would be skeptical of any prediction of great confidence in a Liberal win made more than two weeks prior to election night. The final “stretch” of the fall election was extremely decisive.

I would also express great caution in associating causal links between events and the rise of the Liberal Party in the federal election. A common remark, along with lamenting Mulcair’s “balanced budget” promise, is that the Tories’ chances dwindled with the niqab debate and its islamophobic undertones – but I would suggest there’s a more complex causal relationship present. When the niqab debate emerged in late September, the Tories’ numbers did not go down, they went up – surveys that I saw at the time suggested it was an effective wedge issue – but I do believe that this apparent success faced a countervailing effect. Disheartening a lot of Canadians, a values debate, while consolidating Harper’s base, encouraged many voters to back whoever in October seemed to have the most momentum, Mulcair or Trudeau. In early October, the NDP campaign had cooled significantly in terms of its presence and its messaging, as if it were running on cruise control. That was in stark contrast to Trudeau, running the campaign of his life, whose campaign was barraging the airwaves with two of the most effective political ads I’ve ever seen – by October 19, any television-watching Canadian could recite Trudeau’s “better is always possible” stump speech verbatim and do a cracking good Hazel McCallion impression (*Stephen*, do I look scared to you?”.) The Liberal party benefited from running a better campaign at the most opportune moment.

Superforecasting, while not necessarily preparing readers to forecast the kind of explosively volatile, multi-party democracy we were entertaining last fall, does provide other lessons for its readers to take with them. Tools of better judgment cannot give us the clairvoyant powers necessary to have accurately (or confidentially) called the 2015 Federal Election a few months out – but these tools can make us better predictors overall and could have even improved my own judgment on the election. Whereas I made predictions that weren’t asked for, Superforecasting urges us to stick to the question asked, how ever hard. Whereas I sometimes dismissed data that didn’t jive with my pet theories, Superforecasting urges us to slay our sacred cows and subject them to considerable scrutiny. A shift towards this kind of careful and numerate mindset in the Prime Minister’s Office is an exciting development, especially for those of us who have been calling for more research-driven governance in Ottawa.

Both Trudeau and Harper appeared in competing party conventions this week. One a loser, the other the election’s victor. Stephen Harper took the stage Thursday night over the sound of ACDC – music as awkward a fit as the man it introduced – passé and fake in its testosterone. The former leader, pale and tired, smiled meekly at the adoring crowds, skirting around discussing the election, as he has in the past, except admitting it had not ended as he would have hoped. He spoke with a cavernous distance between him and the party as if he were already a footnote, fading into the twilight as he spoke on stage. Indeed, textbooks are no doubt already being printed, characterizing the 2015 Federal Election as a major electoral shift, and Stephen Harper’s defeat, a near certain event from the onset.

But, as Superforecasting warns its readers, history fools us as time whizzes past. Our uncertainties becomes certainties and our fortunes become destinies. The truth is if fate is to be disbelieved and data canonized, Stephen Harper could very well have attended the winner’s convention this weekend rather than Trudeau.

Published by The Ribbon

4 thoughts on “Superforecasting: How I called the 2015 Election wrong”

I love all the head scratching around the failed predictions for the 2015 election!

In my opinion though, the major oversight is in not understanding and accounting for the 2011 election results.

To illustrate, the NDP were only elected in such numbers because a) Ignatieff was weak and did not connect with the electorate, and b) Jack Layton connected with voters. Also, pundits generally failed to understand Quebec voters, who were not signalling a major shift towards the NDP, just a fondness for Jack Layton, and only a short-term one at that. Substitute Tom Mulcair for Jack Layton, and you have a very different NDP, and all bets are off. Plus, don’t forget — in Quebec, voters were more familiar with Tom Mulcair as a politician than the rest of Canadians, and may have had mixed feelings about him.

Liberals numbers in 2011 were only so low because Ignatieff was personally rejected by Canadians, not because Canadians became less centrist. In 2015 there was every indication that Trudeau was connecting with Canadians. Time and again, he shattered low expectations of his performance — that should have been a clue.

Lastly, anyone who spent any time on social media in the past few years would have noticed how despised Harper was as PM, and how determined people were to voting him out — for months, and years even, people posted about how they would vote for the candidate with the best momentum to unseat the CPC candidates.

I don’t think that superforecasting can take all these factors into account — only a clear-eyed pundit can.

And there is the rub — most MSM pundits are too biased to be able to recognize the views of Canadians. Witness the reaction to elbowgate: almost uniformly the MSM punditocracy declared the PM damaged goods, that this was a disaster he could not recover from. Not one of them focused on the antics of the NDP. Polling data since has shown Trudeau’s favourability ratings went up as a result, while Mulcair’s declined, demonstrating that the MSM punditrocracy is completely out of touch with Canadians, and is withering in its own toxic bubble.

That’s an interesting way to put it, Monika. Certainly the bleeding of the NDP’s support in Quebec had a lot of people surprised – 308’s Eric Grenier was predicting Quebec was an orange “fortress” until October in a few pollcasts. The polling just didn’t show the break (although it did show Mulcair’s weaker personal numbers) that was on the horizon.