In Jobs Data, ‘Surprises’ Mean Bad News

Friday’s jobs report was suggestive of an economy at stall speed. The United States neither added or subtracted jobs in August, according to the initial estimates from government’s establishment survey. The numbers hadn’t been good before, but this was the first time in ten months that the payrolls figure had failed to be positive.

The report contrasted with the expectations of economists surveyed by organizations like Bloomberg and Reuters, who had forecast about 64,000 jobs to be added. In the shorthand of financial journalists, the result was deemed to be “unexpected.”

It seems like when there are these “surprises” in the jobs report, they usually turn out to be negative. Behind two out of three economic doors, it seems, lies a goat, and not the shiny new automobile that will drive us along on the road to recovery.

Then again, it’s also my impression that negative economic reports, fairly or not, tend to receive more press attention than positive ones. So I decided to check the record. This is a pretty simple study. I went through the Google News archive to compile reports of consensus forecasts of payroll growth made immediately prior to the release of the government’s monthly employment report. If there were different figures reported in different surveys of economists, I simply averaged them together, although usually the differences didn’t amount to much. I then compared the forecasts against the net nonfarm payrolls figure as initially reported by the government.

It turns out that since the start of the recession in December 2007, the consensus forecasts have been too optimistic on 28 occasions, and too pessimistic on 17 occasions. So there have in fact been more negative than positive surprises. On average, economists have overestimated job creation by about 18,000 jobs per month, or about 823,000 jobs total over the entire period.

If you focus only on the most serious misses — those where the Wall Street consensus was off by at least 50,000 jobs in either direction — there have been 18 negative surprises and 10 positive ones.

That sounds pretty bad, and it is — especially since people’s livelihoods are at stake. However, the results are only just on the verge of being statistically significant.

So I decided to expand the study to include every jobs report in the new millennium: everything from January 2000 onward. (Because the Google News archive gets spotty as of about mid- 2001, I supplemented the data with reports from CBS MarketWatch for the early years.)

It turns out that this pattern of too-optimistic forecasts also existed prior to the recession. Over the past ten years, there have been 89 jobs forecasts that were too optimistic, versus just 51 that were too pessimistic.

Eliminating the close calls where the report was within 50,000 jobs either way, the tally becomes even more lopsided: 24 positive surprises against 57 negative ones.

With the larger sample size, we can now say almost without question that the jobs creation estimates put forward by economists have been biased upward. Negative surprises have been about twice as common as positive ones over the past 12 years.

What’s interesting is that this pattern is fairly persistent across time: the study covers portions of three expansions and two recessions, and the optimism bias is present in all five periods. If anything, it’s gotten a little bit milder recently: economists were especially flummoxed when the jobs engine of the 1990s began to screech to a halt at the start of the 2000s.

The more benign explanation for this, I suppose, is that the economists surveyed by organizations like Reuters, Bloomberg and CBS MarketWatch are a bunch of unrepentant bulls, whose voices often tend to be heard more loudly in the financial newswires. The past decade or so has been associated with lots of volatility in stock prices — but in general, stock prices have remained expensive relative to earnings, and as compared to longer-term trends, despite a litany of economic problems in the broader economy.

The scarier possibility is that something really has changed in the American economy: since 2000, the average payrolls report has shown the addition of just 17,000 new jobs (or only 4,000 jobs after revisions). If these models are trained on data going back several decades, but there have been structural changes in the U.S. economy since then, that could explain why these forecasts are consistently missing to the high side.

Another potential explanation — or complication — is that there’s quite a bit of error in the jobs numbers that the government puts out. So far, I’ve been focusing on the jobs numbers as initially reported by the government each month. But they are subject to substantial revision, both in the two months after the government releases the initial number and then again at later periods.

Since 2000, slightly more of the revisions have been to the downside, although they can vary significantly from period to period.

Most notably, the government’s initial estimates significantly underestimated the magnitude of job losses during the most recent recession. They now total almost 7.4 million jobs, according to the government’s latest estimates, versus an aggregate of 5.1 million in job losses as reported originally, a difference of 2.3 million jobs.

Most of these jobs losses — about 1.7 million out of the total discrepancy of 2.3 million — came prior to the design and passage of the stimulus bill in February 2009. (Other economic statistics like G.D.P. have also been subject to significant downward revision over this period.)

I don’t know whether seeing the figures as they read now, as opposed to how they printed at the time, would have compelled Barack Obama and the Congress to pass a more substantial stimulus package. But the economy was much worse than originally believed.

On average, the absolute value of monthly revisions since 2000 has been about 70,000 jobs, plus or minus.

The margin of error, or 95 percent confidence interval, is generally about 2.5 times as large as the average error. So in this case, it’s about 180,000 jobs. If the August employment report, which showed no overall change in payrolls, seemed too darned equivocal for you, do not fret: it will probably change. Unfortunately, recent history suggests that these changes will probably be for the worse.

Media reports probably treat the government’s initial estimates of job growth with too much credulity. The employment report is based on a pair of surveys, which like any surveys are subject to significant sampling error. Taking survey data and using statistical models to translate them into estimates of job creation introduces more error.

The error in economists’ forecasts of the payrolls number, and the error in the figure as originally reported by the government versus the revised number, tend to have a compounding effect. Since 2000, economists’ initial estimate of job growth has been off by an average of 90,000 jobs as compared with the revised figure. That implies that when you see an economist’s forecast of monthly job growth, it has a 95 percent confidence interval of plus or minus 232,000 jobs.

Nate Silver is the founder and editor in chief of FiveThirtyEight. @natesilver538