Here’s what Nate does: He compares the change in polls between successive periods.? If there were momentum, he argues, a rise in the polls this month would make it more likely there is another rise next month, while a fall in this month’s polls would likely yield another fall next month.? That is, he believes that momentum will cause successive changes in a time series to be positively correlated. Instead, he finds the opposite to be true. (His graph is?here.)

But this isn’t a fair test of the momentum hypothesis.? Here’s the problem: Imagine that there is some measurement error in the poll taken this month – perhaps the pollsters just interviewed too many Democrats, or too many Republicans. If there’s any kind of measurement error, this could drive Nate’s findings.? There are two cases to consider:

If the estimate for this month is too high, it will make the change between this month and last month look too large, and it will make the change from this month to next month look too small (or perhaps negative).

Or, if the estimate for this month is too low, it will make the change between this month and last month look too small, and it will make the change from this month to next month look too large.

Either way, if there’s any kind of noise in your estimate for this month, you get a?negative correlation between adjoining changes in polls.? But in my example, this negative correlation is driven by measurement error in the polls, not by the presence or absence of momentum.

The problem is that he’s using the same measurement – this month’s poll – in constructing both of the variables he’s analyzing.? And this is likely responsible for much of the (negative) correlation he observes.? So perhaps this negative correlation is in fact disguising true momentum in political races.

Perhaps a simple example will help.? I’m going to use data on the black unemployment rate to make my point, just because these are the data I have handy:

It’s crystal clear that there’s substantial momentum here: When black unemployment has been rising for a few months, it usually continues to rise; when it has been falling, it usually continues to fall for a few more months.

But let’s see what happens if we perform Nate Silver’s test, analyzing successive monthly changes in this black unemployment rate?

Instead of finding positive momentum, we find that this month’s change is negatively correlated with next month’s change in unemployment! This would lead Nate’s approach to (wrongly) conclude there’s no momentum.? The reason is that the black unemployment rate is measured with error, and by construction, these errors cause this month’s change and last month’s change to move in opposite directions.

There’s a simple solution: Analyze changes where you don’t use the same measure – this month’s unemployment rate – in constructing both your dependent and independent variables.? I try this alternative test in the next graph, showing the change in the black unemployment rate between next month and this month versus the change between last month and the previous month:

And now this analysis shows what was obvious from the first graph: Yes, there is momentum in the black unemployment rate.

Is this what is going on with Nate’s analysis of polling data?? I don’t know for sure, because I don’t have his database to test this idea on.? But I’m willing to bet it is.? Nate: Here’s my proposal.? Re-run your analysis, but instead of analyzing the relationship between changes over periods A and B as a function of changes between periods B and C, analyze them as a function of changes between periods C and D.? I’m confident that you’ll find less evidence of negative correlation; you may even find evidence of a positive correlation.

In fact, let’s bet a fancy dinner on the outcome – I reckon you’ll find that non-overlapping changes in polling are in fact positively related. That is, there probably is momentum in political races.

Fly on the wall.

Why don't you be quite specific as to which races we can watch to see who is buying whom dinner?

Ari Caroline

October 22, 2010 @ 3:21pm

Another possible solution is to use a long-term weighted trend such as that created in exponential smoothing forecasts.

Granted, if you define momentum as month-to-month, this doesn't help as much. Still, I think most of us looking at these charts have a longer term view of momentum.

jim

October 22, 2010 @ 3:33pm

If its a question of noisy data, wouldn't a smoothing method be more appropriate?

Tristan

October 22, 2010 @ 3:41pm

When I read Nate's post I didn't take away that his conclusion was that there was negative correlation (even though his graph shows a slight negative pattern), but that he said there was no impact from one period to the next.

I would agree that not using independant variables would cause a slight negative correlation, but it appeared to be at a level that wasn't significant.

I believe that the point Nate is trying to make is that when news personalities report on polling data they make a claim for a strong effect of momentum, one that's been shown not to exist, certainly not in the magnitude that they imply.

Even looking at the data you use to show an "obvious" case of momentum I struggle to see it in the first graph and remain unconvinced that it's an important factor after looking at the scatter plot. If I were to report on black unemployment numbers and talk only about the momentum effect that would miss out on the real story, which is most likely much to complex to discover from just looking at these graphs. In the same way that trying to tell the story about popular support for politicians is much too complex a story to tell just based on looking at graphs of polling numbers. In both cases it's relatively easy to fall in to the trap of assigning too much importance to momentum just because there's no other background available.

Read more...

Andrew

October 22, 2010 @ 4:37pm

A few points:
1. The point of the piece was not to show that there was a negative correlation for momentum between two periods. Rather, the point was that the correclation was insignificant.

2. Nate does not compare strictly month-to-month. Period A (right before the election) is only 10 days long, whereas Perid D (two months out+) is 90 days long.

3. His data points are not based on single polls, but rather the average of multiple polls. He "require[d] that at least 2 polls were conducted in each period for the race to be included."

Also note that the relevant timescales for elections (a few months) and the black unemployment rate (a few years) are very different.

Mantonat

October 22, 2010 @ 5:37pm

I think a definition of what you mean by momentum is in order here. Momentum in terms of polls doesn't just mean a change over time, at least not to me. What I think people mean when they say momentum is that there is change that accelerates as the deadline approaches, change that becomes harder and harder for the opposition to overcome. So I don't see how the word momentum even applies to the graph of unemployment. There's no deadline or goal date, there's no opposition. It's simply raw numbers. When coming up a with a statistical measure of momemtum, one must also be able to measure the impact of obstacles. After all, doesn't momentum mean how hard an object is to stop once it gets going? In electoral races, someone may have so much momemtum that only the worst news about the candidate will change the polls. Others may have just a little momentum, so that the opposition can use strategy to slow popularity. With unemployment statistics, momentum can't really bowl over mitigating factors. Unemployment increases or decreases because of external factors, not in spite of them.

Read more...

Dr J

October 22, 2010 @ 5:42pm

The correlations on both those plots look terrible - as a physicist I wouldn't claim any particular significance or usefulness in either of them - you claim 12%(!) of the month to month variation can be attributed to momentum - that is very little - clearly, month to month variations in unemployment are dominated by noise, hence to claim momentum based on month to month variations (as our news media often does) is specious. I think Nate has a point - even though there is an underlying signal (obviously) you can not tell much from the month to month variations - maybe you should do a fourier analysis and look at the power spectrum for various frequencies to get a feeling for the time scale that the large scale variations are taking place - or do your correlation analysis over various time scales if you don't understand fourier transforms, like averaging over a several month's worth of data... so far the depth of your analysis does not impress me.

Read more...

Peter R

October 22, 2010 @ 5:46pm

Another area where people obviously search for "momentum" is the stock market. Isn't there some method we can borrow from them to do this a bit more rigorously?

Tristan

October 22, 2010 @ 6:50pm

Also, if Nate doesn't want to bet, you can always take this data to the prediction markets and find plenty of people to bet against you in races where the price doesn't fully reflect the impact of momentum that you expect.

I'm sure clearing a profit equal to the cost of a couple fancy dinners would be a good way of making a point and showing the usefulness of this analysis. Of course if your trades went the wrong way it would show that momentum might not be the most useful prediction tool - so... win win?

CK

October 22, 2010 @ 8:00pm

Why not email Nate. He loves this stuff and I'm sure he would respond to your response directly.

Phil Birnbaum

October 23, 2010 @ 4:13am

Very nice!

Josh

October 24, 2010 @ 7:01am

I don't know about any of this, but I read the word "political" and started thinking of how I can't freaking wait until midterm elections are over.

www.philstockworld.com

shaman

October 24, 2010 @ 4:33pm

Seems like an opportunity to use a non-parametric statistic, like Wald-Wolfowitz to test for the significance of runs.

JDK

October 25, 2010 @ 1:57pm

The crux of the issue is "How do we operational define momemtum".

As I had pointed out in my comment to Silver's blog, his "definition" fails so as you point out his analysis fails.

But I'm not sure your operational definition is any better. Who knows?

I think its time to re-imagine polling data via the lens of statistical process control, aka Shewhart and Deming.
The use of an IndX/mR chart and "Western Electric" rules for trends as a likely special cause easily fits the bill of a better definition of "momentum".

And for the most part, it can be done pretty easily with pencil and paper!

Daniel

October 26, 2010 @ 9:55pm

To Dr J, at #7:

Social scientists have much lower thresholds for significance than physical sciences. In political economics, if you manage to explain 35% of your data, people start naming laws after you. 12% is not fabulous, but it's still acceptable in this context.

More generally, I think Wolfers misread Silver's original post. It seems like Silver has already taken steps to mitigate bias by using a combination of polls, and furthermore, he admits his results are not significant. In effect, Silver's not saying that there's never momentum in an election; instead, he's arguing there's no reason to suggest it's a factor in general elections.