Why you’re wrong to get excited about “60%”

August 4th, 2014, 2:35pm by Sam Wang

Some people are excited (positively or negatively) about Nate Silver’s column today giving a probability of a GOP takeover at 60%. To cut to the chase: I do not think that number means what you think it does.

Here are three big things to think about.

1) Five coin tosses. At this point, Senate control comes down to as few as five* races: AK, CO, IA, KY, and LA. Think of these races as coin tosses. Then Democrats have to win 3 out of these 5 tosses to retain control. (I’m simplifying matters, but not by much.) These coins are not perfectly fair, and the overall situation is a little unfavorable to Democrats. That is basically the amount of uncertainty expressed in Silver’s probability.

Fundamentally, any probability in the 40-60% range is a numerical way of saying “I don’t know.” (Just to poke at the scar a bit, “I don’t know” is what Silver should have said when he intimated that Brazil would probably beat Germany in the World Cup. We all know how that turned out.)

2) The certainty fallacy. Silver has done something common among paid writers, which is to do what it takes to attract eyeballs. He has rounded a probability that is barely over 50% to make the statement that one side is ahead.

This problem is endemic to journalism. Nuance with probabilities is just lost on most people. But writers are just trying to appeal to their readers. The real headline is “WE DON’T KNOW” but somehow that is not what gets the lead. Really, he’s caved in to all of you!

Basically, whenever you see a probability like that, you should mentally say “plus or minus 20%” just to get the right idea. Of course, if the probability gets above 80% that doesn’t make sense any more. At that point, it’s OK to say that one side is ahead.

If Begich really is ahead, that translates to Democrats only having to win 2 out of 4. That gives a pretty accurate intuition for where the Senate race actually is as of today.

Make no mistake: at the moment, the battle for Senate control is super-close. If national opinion swings as little as 2 points in either direction, uncertainty about Senate control will evaporate. As I’ve written before, even if that happens, don’t be surprised if both sides end up with 49 to 51 seats each in 2015.

*Just for purposes of building up a clearer intuition about probability, I’ve left out AR, GA, and NC. Of course, here at PEC our Meta-Analysis considers all races at once. But the bottom-line answer is exactly what I’ve said. So for purposes of discussion today, I’ll leave those races out.

34 Comments so far ↓

Had a lot more respect for 538 last cycle when the discussion was mostly politics and the search for America’s best burrito was left to the Food Channel where it belongs. I do miss Intrade though for it’s daily ( sometimes hourly ) entertainment for political junkies . Still think following the betting lines give a more accurate picture of the Race than the talking heads who need to ‘ keep it close ‘ to interest viewers and sell advertising . Thanks for all your hard work Sam sure glad you haven’t changed !

I’d get more excited if we didn’t already know one side can pretty much kill anything it wants in the senate. Couple that with a president that doesn’t have election pressures for two more years. Our government has become like a hockey game but instead of a puck the ref drops a dead horse labeled “change” and the two sides proceed to whack at it with toothpicks. Like two large companies engaging in ad wars they both win from the added publicity the ‘rivalry’ brings.

Even when the Democrats had a clear large majority the best they could get was a pretty flawed, but much better than nothing, health care bill. I’m not getting excited about slim majorities for either side.

The thing is, how much can really happen? Measured in Senate control probability, a lot — but that’s only because we’re right on the edge. Measured in terms of public opinion, the likely range of movement is smaller.

Ok Sam, play “Dear Abby” for a moment. I have a conservative friend who will most certainly try to throw the new 538 number in my face as soon as he picks up on it. So, do I pre-empt his move by sending him a link to the above right now or do I wait and use it as a rebuttal?

The key to understanding what Silver and I do is that it’s not rocket science. All one needs is a method for looking at all the data at once, one that does not add too much extraneous bullshit. Punditry is mostly extraneous bullshit.

All I am doing today is pointing out that Silver said something mostly defensible as a number (60%), which people are foolishly rounding up to 100%.

Moreover, as subsequent research has indicated, Intrade was “gamed.” That is, a pro-Romney guy with deep pockets intentionally bought up Romney to Win shares on Intrade with the intent of driving up the price, probably to make Romney look “good.” See here: http://bit.ly/1kBGasB

Dr. Wang,
Again your insights are highly appreciated.
Two quick questions about this mid-term election.
1. Do you detect any trends in the Senate races, both individually and overall?
2. After Cantor’s debacle in VA, where an important part of the story was disastrous polling, what is your confidence level in the polls you are seeing? I assume that state-wides are far better than congressional district polls, but are minority populations still being under-represented?

Yes poll aggregation is more accurate but Intrade is more accurate than just one poll I believe . Current betting line on 2016 : Clinton 5 /4 Rubio 10/1 Christie 10/1 Paul 14/1 . I see Paul as an underlay currently as he’s had a better year than all other Republicans .

Well, I agree with that. On short time scales, Betting markets perform like a noisy poll average – about as well as if one looked at a HuffPost polling page for three seconds and then estimated an average. On long time scales, markets collect the wisdom of individual bettors. That amounts to conventional wisdom, which is often right.

There’s one wild-card in the Senate control discussion that so far, I have seen little about. While he currently caucuses with the Democrats, Maine Senator Angus King’s “loyalties” have often appeared to be up for sale. At the moment, with Democrats in control, he caucuses with them, but with a 50-50 divided Senate, could he throw the whole thing to the GOP? I’m sure they’d try and offer him the moon and stars if he would. The only problem with that is that he would likely face the wrath of Henry Reid, and the Democrats are likely to retake control in 2016. King isn’t up for reelection until 2018, and 2014 will be long distant memory by then.

60% +- 20% sounds about right. Silver would get an F if this was a lab report. We do not take kindly to a reported measurement not accompanied by its uncertainty.

Looking at your previous post comparing polls with results (which showed that results skewed D+2 compared to polls in close races), and keeping in mind that midterms have low turnout and even a modest bump in voting likelihood among a certain demographic can have a serious effect, I would add ~15% systematic uncertainty in quadrature for pollster mis-weighting. (For a total of a 25% uncertainty)

Yes, people like the African American cohort that voted in the MS runoff for Thad Cochran. The articles I’ve read state these voters carried Cochran, but none discuss the technical details of how their vote was mobilized.

At some point there needs to be a study of GOTV effectiveness in the era where everyone (but me) is connected to social media on their phones.

Effective education/mobilization helps both sides of course, so Sam is probably right to trust the median of the polls for now. In the long run I expect the pollsters will change their demographic weighting and likely voter screens towards Dems, since apathetic voters tend to skew D.

Whatever happens in 2014, the Democrats will have a much better chance of regaining the Senate in 2016 when the Republicans have to defend a large number of seats. As stated above, a 51-49 GOP Senate for the next two years will mean little except for judges.

To me, the single most interesting item in Silver’s column is his forecast of an 80-20 probability of victory for McConnell in Kentucky. That is non-congruent with PEC’s estimate, and I thought with what the polls were showing. Have I missed some statistical info that would support Silver on this one? He could just be weighting McConnell’s incumbency very heavily, but one doesn’t know since he gives no discussion for this assessment.

It probably has to do with campaign finance, or maybe older polls. Those are not crazy reasons. But I try to keep such handicapping separate from the PEC estimate, which is set up to just use the last 3 polls / N weeks in a blind fashion – a minimalist approach. One thing that is true is that over the course of the campaign, Grimes has had difficulty breaking past 47%.

Any polls-only based estimate should be more like 50-50. As they state, they are using prior assumptions based on pre-campaign conditions, which is what NYT and WaPo do. That’s where they get their data. In particular, I am skeptical of the WaPo approach. I am against such assumptions on the grounds that they are less accurate than fresh polling data.

For example, other aggregators are calling Mitch McConnell the favorite. In that case it could be prior assumptions. In that case, my gut says they have a point: Grimes’s HuffPost trendline is fairly flat and and she has had trouble getting above 47%.

Generally, if one looks at long-term trends, incumbent job approval, and other factors, the impression that emerges in the nine races I listed a few days ago is as follows:Republican advantage: Georgia, Kentucky, Louisiana, MontanaDemocratic advantage: Alaska, Colorado, Iowa, Michigan, North Carolina

This particular scenario leads to a 50-50 split. Now, if you call this a prediction I’ll deny it…for now.

At the end of the 2008-2012 campaigns, I would often increase the time window on each state until I minimize SEM, i.e. I choose the time window where SD/N is smallest, where N>=3. This secret sauce worked rather well. I wouldn’t do it now – conditions are too fluid.

Dr. Wang is it possible a quiet giant is waking up and this election could see an increase in voter turnout? The reasons why I ask: women’s health care, voters’ rights, immigration, and a few other issues. I feel like this election will see an increase in voter turnout. Just wondering about your thoughts on this.

No idea. It would probably show up in polls. In 2004 I thought there’d be some “bonus” above what the polls were saying. Boy, was I wrong. So…just watch the aggregate up there in the banner, and in the graphs in the right-hand column.