Menu

Putting elections under the microscope

Kevin Bonham has written a very detailed summary of the performance of seat-level betting markets in the last election. It’s well worth reading — go check it out. We’ve really enjoyed reading Kevin’s blog and his analysis. In a spirit of friendly debate, we wanted to respond to a few comments that are directly relevant to this blog. Kevin writes:

electionlab in their final analysis considered that the most likely culprit in the errors made by seat betting markets was their modelling and not the markets themselves. In my view, there was nothing significantly wrong in their model’s read of what the seat betting markets were thinking – rather, what actually happened was that seat betting markets themselves were in fact wrong. Different modelling assumptions regarding covariance and so on greatly affect the spread of modelled expectations, but they have little impact on the mean. The seat betting markets were collectively expecting Labor to win fewer than 50 seats at the end. There is no way to remodel the final odds to find 55 seats for Labor in them because it is just not true that those markets thought Labor would win that many seats. Or at least, if someone “finds” such a way to read that result into the markets, the next time they test it I can pretty much guarantee the post hoc overfitting in their new model will cause it to blow up.

We disagree here, for two reasons.

First, changes in covariance structure don’t just change the spread of the seat-count distribution; they fundamentally alter the shape of the distribution, too, and this has important implications for estimating seat counts from the betting odds. Assuming independence between seats results in a unimodal, bell-shaped distribution; on the other hand, assuming maximum covariance between seats (as constrained by the betting markets) results in a bimodal distribution, with very little density in the middle. More generally, it makes intuitive sense that the distribution might become multi-modal if you bring in covariance between seats. We agree that the mean is relatively insensitive to covariance structure. But for bimodal distributions, the mean is a very bad point estimate and lies in a low probability region of the distribution, meaning it is not a very useful tool in this situation (in hindsight, we should not have used the mean as a point estimate for our maximum covariance model because of this). This is really important, because it affects how we make inferences using the distribution derived from the betting odds. There is nothing sacred about the mean, and it appears that a point estimate centered on the mode (or two point estimates, centered around the two modes) may make more sense here. Or perhaps a point estimate is just a bad idea, and we should look at 95% credible intervals instead. We don’t know the true covariance structure, so we don’t know the true underlying seat count distribution. But we have very good reason to believe that the mean will be a poor point estimate in this case.

Second, a final Labor seat count of 55 seats is absolutely consistent with the betting odds, as long as you expect there to be moderate amounts of covariance between seats. According to the maximum covariance model, there was a 95% chance Labor would obtain somewhere between 32 and 64 seats. Of course, the maximum covariance model is an extreme case, but it’s not hard to show that for more moderate covariance structures, the 55 seats result is within a 95% credible interval. Almost the only way to obtain a prediction with 55 seats outside the 95% credible interval is to assume independence between seats. But as we know, that’s unlikely to be a good assumption.

It baffles me that experienced statisticians attempt to determine how many seats betting markets think parties will win by looking at an indirect and problematic measure (aggregation of implied probabilities concerning particular seats) when there are more direct markets available on seat total events and their past track record has been excellent.

This is a reasonable question: why did we derive predicted seat counts from seat-level betting odds, rather than just directly looking at the seat count betting markets? Why use an indirect method when there is a direct one? We did this because we wanted a general approach that allowed us to look at interesting seat-level scenarios (e.g., this and this). The seat count predictions were something we could readily do with our more general model, so we had a go. Unfortunately, the election ended up being a landslide and, for many of the scenarios we thought might be interesting, the betting markets ended up giving pretty obvious and boring predictions! So the seat count predictions ended up being prominent. We agree that, if your sole aim is to predict seat counts, the seat count betting markets are the way to go. But our ultimate goal is not to just get good at predicting seat counts. We want to use the seat-level betting odds to find interesting stories that can’t be revealed without seat-level data.

The election is finally over! At time of writing, the Coalition stand to hold 89 seats and Labor 57 seats, with the remaining 4 seats going to a Green and independents. It seems likely these numbers may well change in the coming weeks as postal votes come in for a few close electorates, but we’ll go with them for now.

Leng and I have been sorting through the entrails of our predictions, looking at what worked and what didn’t. What did the betting markets get right, and where did they fall down? What could we have done better?

What worked

The betting markets successfully predicted the Coalition would win government with a large majority. This bears repeating, even though by election day pretty much everyone was predicting this. In contrast to the polls, at no point in the campaign were Labor anywhere close to being the favourites in the betting markets. The high-water mark for Labor in the polls was around July 8, when Newspoll recorded a 50-50 2PP. In contrast, our AFR analysis of the electorate-level betting markets on July 11 still put Labor well behind the Coalition, with an expected Coalition-Labor seat count of 84-60. The closest the parties ever got was 81-66 in our July 30 analysis. We aren’t criticizing the polls here — the polls and betting markets measure different things — but this clearly demonstrates that betting markets do more than just parrot polls, contrary to a common misconception.

What didn’t

Assuming the 89-57-4 seat count holds, our predicted expected seat count of 99-48-3 was off by 10 seats. While this was well within the margin of error of our maximum covariance model, this is worse than we’d expected! There are a few possible reasons for this.

Starting with the most general, could this be considered strong evidence against the underlying theory of prediction markets? Not really. Since prediction markets give probabilistic forecasts, they have to be assessed over multiple campaigns. Results from any one campaign don’t tell you much about predictions of underlying probabilities, just as rolling a die once doesn’t tell you enough to know if it’s loaded or not.

Could this result be due to violations of some of the underlying assumptions of prediction markets? That’s certainly possible. We have no information on the amount of money wagered in each seat, although we have been told that around $250,000 was wagered on Sportsbet’s individual seat market between January and the election, with most of it on marginal seats. Nevertheless, we don’t know how that money was distributed, or if it was sufficient, so it’s possible that this had something to do with the seat count error. Longshot bias does not appear to have had any large, obvious impact on the result, with the betting markets ultimately underestimating Labor’s seat count rather than overestimating it (while this may be partly due to our rather crude correction for longshot bias, this doesn’t appear to have altered the results much). Our understanding of these biases is very crude. Predictions from seat-level betting odds have little heritage and it will take time to be able to observe and eventually correct for these biases.

However, we think the most likely culprit is in the modelling, not the betting markets themselves. Using electorate-level betting odds to predict seat counts requires knowledge of the underlying covariance structure between electorates (ad hoc methods, such as counting up the number of seats where Labor has > 50% probability of victory, make no statistical sense). There is no obvious way to estimate this, so we have to make some assumptions about this structure in practice. To try and get an idea of the impact of this uncertainty, we used two models for our estimates: a model that assumed zero covariance between seats (the ‘independent seats’ model), and another that assumed the maximum possible covariance between seats, conditioned on the betting odds (the ‘maximum covariance’ model). These models both generated distributions of seat counts for each party.

Covariance is key

The independent seats model is commonly used, but there seem to be good intuitive reasons to believe it is badly wrong. The final seat counts were outside the margin of error for the independent seats model’s predictions; that is, there was less than a 5% chance of this election outcome occurring by chance, according to the independent seats model. We aren’t big fans of hypothesis testing, but if you were to use the independent seats model as the null hypothesis, you would probably reject this model given the outcome (p < 0.05). The value was well within the margin of error for the maximum covariance model, however. We don’t suggest the maximum covariance model is necessarily a good representation of the real world. But it demonstrates the sensitivity of the results to the assumed covariance structure.

Why did the two models yield the same expected seat counts, even though their seat count distributions were so different? The maximum covariance model represents a scenario where a large, nationwide swing occurs in one direction, resulting in a symmetric distribution that yields the same expected seat count as the independent seats model. However, more subtle covariance structures, such as ones where different states have different magnitude swings (as occurred in this election), may result in asymmetric distributions that shift the expected seat count. So understanding the role of covariance structure in both expected seat counts and the overall distribution is essential. Covariance is key, and it’s something Leng and I want to look at in more detail.

But for now, we’re going to take a break! We’ve had a great time making this blog. We couldn’t have done it without the support of a lot of great people. We want to thank Bob Chen Ren and Jeff Chan for their assistance with the blog, Edmund Tadros and Jason Murphy at the Australian Financial Review for writing up our work, Kevin Bonham and Simon Jackman for their insights, and everyone who showed an interest in this blog!

Ed Morrow had a nice sign off. We dont have one of our own so we just thought we’d steal his. This will be our last blog post until tomorrow when we either crow or eat crow. Either way, we’ve had fun doing the stats and writing stuff. Its 1am now and we’re done. It’s time to eat pizza. Thanks for reading. Good luck to all candidates and people making predictions.

Since predictions based on individual seat probabilities are always subjective (do you predict a candidate will win when they have > 70% probability of victory? 80%? 90%?) and prone to misinterpretation, we don’t intend to make predictions about who will win in all 150 seats. As we discussed in a previous post, it is easy to jump to wrong conclusions with these odds. In particular, just because a party has the highest probability of victory for a particular seat doesn’t necessarily mean the market is confident they’ll win: for instance, if Labor has a 51% chance of victory in a seat, the market still expects them to lose roughly half the time. This is why counting up the number of seats in which Labor has > 50% probability of victory is a bad way of estimating the number of seats they are likely to win.

One big thing we’ve tried to stress on this blog is the role of uncertainty in making predictions. This post is about the uncertainty around our final seat count predictions. With our Monte Carlo simulations, we simulate 100,000 elections and the predictions are the average number of seats each party will win. But as we’ve shown before, there is a distribution of outcomes. The shape of this distribution depends on the covariance structure between seats. We don’t know what this is, so we run two models representing two extreme cases: an independent seats model and a maximum covariance model. Here is the distribution when we assume independence between seats:

What we’d like to do now is provide the 95% credible interval for the predictions. What the hell is a 95% credible interval? Most of you would have heard of a 95% confidence interval which is based off repeated sampling. We don’t have that here. A credible interval is much simpler to interpret. We had 100,000 simulations – a 95% credible interval tells us the range of outcomes which cover 95% of the simulations. The 95% credible interval for the independence model:

a. ALP: 43-53 seats

b. Coalition: 94-104 seats

So in 95% of our simulations, the ALP will win between 43-53 seats and the Coalition will win between 94-104 seats. Really should have put money on the result a few weeks back!

The distribution under the maximum covariance model is a bit more funky looking. It’s not a classical bell shape, with the extremes having the highest probability. The reason for this was discussed in a previous blog post. Unsurprisingly, the variance of this distribution is greater than under the independence model.

The 95% credible interval for the maximum covariance model

a. ALP: 32-64 seats

b. Coalition: 84-114 seats

With this maximum covariance model, the credible intervals are wider.

Note that there is zero probability of a Labor victory under both models.

The difference between our predictions and those based on national polls is that national polls have to assume uniform swings. For example, extrapolating from the latest Roy Morgan poll, the Coalition is predicted to win 89 seats. Hopefully having data on every seat and not relying on uniform swings mean our predictions are more accurate. We’ve made a pal over the internet – Kevin Bonham. He’s been a good guy re commenting on our blog etc and he’s also got a nice blog of his own. Below is a summary of the predictions that lots of different parties have made found on his blog.

So while the election result looks like a foregone conclusion, the test of the predictive powers of betting markets is going to rest on predicting the number of seats each party will win. Compared to all others, our predictions for the ALP are the most dire. More to come.

It’s election day! The campaign has been only 33 days. I think K and I have managed to post on about 8 of those days so I dont think anyone’s going to give us jobs as bloggers any time soon. But the good news is that he is visiting me from MIT and we’re going to try make some predictions in the coming hours.

Like K said, it seems pretty apparent from all the polls and other analysis that the Coalition is going to get up. So our first goal of showing the effectiveness of betting markets in predicting the overall election isn’t going to be really tested. One thing we can point to is that the betting markets were definitely a lot more aggressive with a big Coalition win a lot earlier than a lot of the polls. Just two days after Rudd called the election, the betting markets pointed to the Coalition winning 84 seats to the ALP’s 65. At that time, the national level polls were calling it pretty close. The Newspoll 2 party preferred on both Aug 4 and Aug 11 was 52-48 in favor of the Coalition. So we werent complete muppets.

Anyway, we want to make amends re our poor blogging record. In the next few hours, we’re going to post

We posted a few weeks ago about the likelihood of a complete Labor wipeout in Tasmania: our models gave it a probability somewhere between 5% and 25%. Even a probability of 5% seemed pretty high at the time, given Labor currently hold every seat in Tasmania, except for Denison (held by Andrew Wilkie).

With the election one day away, we wanted to revisit this scenario. The odds should have considerable predictive power by now. Using the same methodology as the previous post, we estimate the probability of a Labor wipeout in Tasmania to be somewhere between 22% and 35%! Conservatively, that’s roughly the probability of flipping a coin twice and getting two heads. It’s more likely not to happen, but there is still a very considerable risk it will.

Looking at the individual seat probabilities, Bass and Braddon appear to be comfortable Coalition gains. Wilkie appears likely to retain his seat in Denison. Franklin and Lyons are the two seats that could still go either way, with Franklin leaning towards Labor (61% chance of Labor victory) and Lyons leaning towards the Coalition (58% chance of Coalition victory). If the implied probabilities of Labor victory drop in Franklin, the probability of a Labor wipeout will increase substantially. We’ll take another look at this scenario when we publish our predictions in the next day or so.

The betting odds suggest the ALP is facing a massive defeat on Saturday. But even worse for Labor, their expected seat count is still deteriorating.

Expected seat counts using Sportsbet betting odds from September 5

Our latest analysis in the AFR (a link will be up shortly) was conducted earlier in the week, where our analysis showed the betting markets predicting Labor would win 51 seats. Underdogs often regain a little ground leading into the final few days of the election campaign, so we thought that number might increase before election day. In fact, as of September 5, the markets now expect Labor to win more like 49 seats.

Barring an act of God, the overall election outcome looks like a foregone conclusion. Both our independent and maximum covariance models give Labor literally zero probability of victory. This is astonishing, since the maximum covariance model had been giving Labor a small but non-zero chance of winning up until very recently. The reason for this change appears to be many of the probabilities moving towards a clear victor in some of the seats that were relatively uncertain previously. As the seat probabilities move closer to 1 or 0, the choice of covariance structure has a smaller impact on the predicted result.

Leng and I will be posting our final predictions shortly. Putting aside any political views on a big Coalition victory, an easily predictable election is not a good test of our methodology! So we’ll be making predictions about individual seat results, and possibly a few scenarios, too. Stay tuned.