Legitimate Reasons To Question September’s BLS Household Survey

As my posts yesterday (see here and here) should make clear, I tend to agree with Steven Taylor regarding the absurdity of the reaction to yesterday’s jobs report from many on the right. The idea that the numbers were consciously manipulated is, quite simply, absurd and it betrays the kind of conspiracy theory mindset best left to the people who think that there are aliens being kept at Area 51 and that Elvis and JFK are running a diner in Montana. The fact that it’s being pushed by people like Jack Welch, the former CEO of one of America’s most prestigious corporations, is all the more troubling. Say what you will about Welch, but he’s no Donald Trump, and I’m not sure whether or not he simply tweeted something without thinking (believe me, it happens) or whether he actually believes that the Obama campaign is manipulating economic data. On the other hand, I’m not at all surprised to see people like Allen West pushing this theory, his time in Congress to date has not exactly been auspicious on a number of counts and he seems to attract most of his support from the most radical wings of the Tea Party movement.

All of that said, though, that doesn’t mean that there aren’t legitimate reasons to question some of the numbers that we get from the Bureau of Labor Statistics. As I noted in my posts yesterday, there are actually two completely separate surveys that make up the jobs report that gets released every month from the BLS. As Harvard Economics Professor Greg Mankiw explains, there are issues with both surveys that need to be taken into account:

One might expect these two measures of employment to be identical, but that is not the case. Although they are positively correlated, the two measures can diverge, especially over short periods of time. A particularly large divergence occurred in the early 2000s, as the economy recovered from the recession of 2001. From November 2001 to August 2003, the establishment survey showed a decline in employment of 1.0 million, while the household survey showed an increase of 1.4 million. Some commentators said the economy was experiencing a “jobless recovery,” but this description applied only to the establishment data, not to the household data.

Why might these two measures of employment diverge? Part of the explanation is that the surveys measure different things. For example, a person who runs his or her own business is self-employed. The household survey counts that person as working, whereas the establishment survey does not because that person does not show up on any firm’s payroll. As another example, a person who holds two jobs is counted as one employed person in the household survey but is counted twice in the establishment survey because that person would show up on the payroll of two firms.

Another part of the explanation for the divergence is that surveys are imperfect. For example, when new firms start up, it may take some time before those firms are included in the establishment survey. The BLS tries to estimate employment at start-ups, but the model it uses to produce these estimates is one possible source of error. A different problem arises from how the household survey extrapolates employment among the surveyed households to the entire population. If the BLS uses incorrect estimates of the size of the population, these errors will be reflected in its estimates of household employment. One possible source of incorrect population estimates is changes in the rate of immigration, both legal and illegal.

(…)

More important than the specifics of these surveys or this particular episode when they diverged is the broader lesson: all economic statistics are imperfect. Although they contain valuable information about what is happening in the economy, each one should be interpreted with a healthy dose of caution and a bit of skepticism.

That last point is an important one, I think. Much like polls are merely a snapshot in time and any individual poll can be subject to any number of variables that could turn a reliable poll in to an outlier, the twin employment surveys can be flawed for one reason or another in particular months.This is one of the reasons that the BLS routinely revises it’s figures for at least two months after making the initial report. So, to some extent, we shouldn’t put too much import on a single month’s report, especially if it is inconsistent with other, more reliable, economic statistics.

All of this brings up the question of just how reliable some of our economic statistics actually are. For example, for years economists have been criticizing the model that the Commerce Department uses to calculate the Consumer Price Index, arguing that it doesn’t reflect reality anymore largely because it’s based on a model developed decades ago that doesn’t reflect the budget of an average family in the 21st Century an that it doesn’t account for the fact that increases in cost often come with increases in product quality. The issues are different with regard to the BLS employment survey’s, of course, because they deal with questions of the methodology of the survey, but as Jazz Shaw points out, making the survey more accurate could be problematic:

First of all, the survey is not just done by BLS workers. It’s conducted in conjunction with the census bureau. And how many people would they need to interview to produce more statistically meaningful results? Ten times as many? Polling professionals will tell you that it’s hard to get people to complete even a moderate size survey. Take a look at the current survey being used. The labor statistics section alone is more than 20 pages long. If you wanted to get a vastly larger sample, the government would need to be calling millions of people every month. It could turn an already bloated bureaucracy into an unmanageable behemoth.

Jazz seems to question the need for the unemployment reports altogether. I’m not sure I agree with this sentiment. In a modern economy, there’s some importance in the government, and specifically institutions like Congress and the Federal Reserve, having at least some idea of what the state of the economy might be. While it’s true that there are private surveys done by entities like Gallup and ADP that also survey the jobs market, they suffer from their own methodological problems and may not be any more reliable than the BLS survey. Taking the information the BLS provides away, as flawed as it might be, doesn’t strike me as the best response to this situation. Instead, I’d suggest that we should find ways to make the survey more reliable while at the same time making it more clear that a single month’s results aren’t nearly as definitive as the media makes them out to be.

This report is a tale of two labor markets. The establishment survey (payrolls) painted a picture of moderately growing employment over the last three months but at a marginally slower pace than over the last year. At this pace of job creation, the unemployment rate should be barely drifting lower given underlying demographic trends. In contrast, the household survey painted a picture of a sharply falling unemployment rate—down 1.2% points over the last 12 months. Such a rapid decline in the unemployment rate would be consistent with 4%-5% real economic growthhistorically but much of the decline is accounted for by people dropping out of the labor force (over the last year the employment-population ratio has risen to only 58.7% from 58.4%). We believe part of the drop in the unemployment rate over the last two months is a statistical quirk (the household data show an increase in employment of 873,000 in September, which is completely implausible and likely a result of sampling volatility). Moreover, declining labor force participation over the last year (resulting in 1.1 million people disappearing from the labor force) accounts for much of the rest of the decline. With this report, the ISMs, and vehicle sales, the September economy is off to a better-than-expected start but nowhere near as good as suggested by the decline in the unemployment rate.

(Emphasis in original)

As we already know, we aren’t in an economy where the economy is growing at a real rate anywhere near 4% to 5%. Indeed, the final revision of second quarter GDP ended up revising growth downward to 1.3% and the estimates for third quarter GDP, the first report of which comes out later this month, has it in the same general range but in any case no higher than 1.5%. So either the GDP numbers are way, way off, which seems unlikely, or there’s some statistical flaw in the Household Survey. Of course, once you take into account the fact that a large number of the new hires were apparently part-time and seasonal jobs, the numbers start to make more sense. More importantly, though, the last time we actually saw anything close to 800,000 jobs created in a single month was in an era when GDP was growing at a mind-blowing 9.3%. Based on all of this, it seems more likely than not, the Household Survey numbers for September are a statistical outlier that we are likely to see adjusted downward, with a subsequent upward tick in the Unemployment Rate, in the coming months.

Related Posts:

About Doug MataconisDoug holds a B.A. in Political Science from Rutgers University and J.D. from George Mason University School of Law. He joined the staff of OTB in May, 2010 and also writes at Below The Beltway.
Follow Doug on Twitter | Facebook

The weird thing is that no one used to dealing with these numbers was going to get very excited by just one of them. In all the data series, in all the world, one moved from 8.1 to 7.8. As I’ve noted, that isn’t even a big change, just 3.7% variation.

No one would normally spend as much energy as you’ve just done worrying about whether the change will stick. They’d maybe shift outlook slightly, and then wait for next month’s data.

As I’ve been emphasizing, this is all a drama only because the leading digit changed. 7.8 is below 8 and that is big news, even though it is only below by 0.2 and 0.2 is not much.

So the bottom line is to wait a month or two, and don’t get all nuts about the leading digit. If you want to round to a single digit, 7.8 still rounds to 8.

I do not think there is a conspiracy or some major fault with the figures as given( Actually there were some things that happened at 51 that the government will keep classified forever, and they are not related to ufo’s) What are resources that give more information about the jobs statistics? I would want to know breakdowns by age, salary, benefits, and type of work. Where do the numbers come from? How do they compare to previous years? What trends do we see? How is the math done? When there is more data, there can be a clearer picture of what is going on.

Only because of the timing, right? U3 spiked from 9.5 to 9.8 in November 2010, and then dropped to 9.4 the following month. That was a one month 0.4 drop, but no one cared. Oh, and all that variation in 2010 was within “the nines,” with the same leading digit.

The household survey tends to show greater enthusiasm than the establishment one, which is why the leftstream media were so adamant about using the establishment one back during Bush 43. The drop in the U3 simply does not jibe with the number of jobs created unless the U6 took a similar jump. Did it? You’re basing your statistics on a survey of people that may not properly understand the parameters, just so you can say “oh look, U3 is under 8!” Is it merely coincidence that this is just in time for the election?

Blame the media. The BLS provides much of the nuance in the supporting tables that accompany the jobs release.

I’ll explain. The BLS says there were 114.000 jobs created in Sept. That is not true. The 114,000 figure is a statistical estimate within a 90% confidence interval. In other words, based upon the BLS models, the “real” number of jobs created lies within plus or minus one standard deviation of 114.000. How large is the standard deviation? The tables suggest it is about plus or minus 100.000. A more accurate way of describing it would be, “The BLS has determined with 90% confidence that the number of jobs created in Sept. was between 14.000 and 214.000″. Note that also implies there is a 10% chance that the “true” job creation number was less than 14,000 or greater than 214.000. The only thing we know for sure is that the 114.000 number that makes every headline is a fictional number. I am not saying it is dishonestly calculated or there is anything nefarious going on. It’s arrived at using sound statistical science. But it’s simply not accurate to say there were 114,000 jobs created. As the saying goes, “there are lies, damn lies, and statistics”.

It borders on journalism malpractice for reporters to present the number with a false sense of precision and make no effort to explain part of the science and the numerous assumption that go into its calculation. For example, the raw data are adjusted for seasonal adjustments. And as Mankiw correctly points out, the BLS uses something called the “birth/death” model to estimate the number of jobs created by new startups and the number of jobs lost from small business closures. People have been criticizing the “birth/death” model for probably at least a decade. And I’m not talking about internet cranks. I’m talking serious people who analyze this stuff for a living like John Williams at Shadow Statistics, investment analysts, and serious financial bloggers.

I’ve been fascinated by this debate the last few days. I think JP has good perspective.

All of the various purely random statistical problems, sampling statistical problems and the hoary birth death estimate have been well hashed out.

The elephant in the room is the poor absolute jobs number, or in the case of the big one, it’s part time component. This is consistent with an abysmal absolute number of jobs created pattern for months now. Forget the statistical mumbo jumbo of a single month. It’s all political posturing, and pure analytical crap. This economy is nowhere.

Welch is another billionaire in the tank for Romney. So in an attempt to save his boy he said something transparently stupid. He’s an old man, and obviously he made a mistake. I hate to see him lower himself to Donald Trump status.

It makes me nervous — I’m hoping not to suffer major IQ drops as I age. But the fact is you do start seeing little bits of decay around the edges. My night vision ain’t what it used to be. As far as work goes I’m compensating for loss of stamina with the efficiency that comes from experience. But yeah, age comes for us all, and it’s come for Jack Welch.

One thing I hope to avoid is the stubbornness of old age. I’ve never been stubborn, but who knows?

Why is it assumed that a Tea Party individual such as Allen West, or even Jack Welch, for that matter are necessarily biased? I find this position insulting. Both men are raising valid questions. And why does a survey that seeks to ascertain who is working and who is not require 20 pages? There should be an investigation of the methodology and the specific results of this month’s survey need to be looked into. Someone not associated with the DLS should look at the calls made, what was said and compare those to the stats. If there is no audit trail, that should raise some suspicions. Secondly, what are the party affiliations of the people working there? If they are all Obama supporters then there should be some interviews of these people. Thirdly, how many people have been hired here over the last few years and what are the party affiliations of those people? And finally, why are part time jobs counted as jobs? Why aren’t they counted as half a job? Why is it partisan to question this organization? It seems to me that a statistical analysis that leaves open the possibility of mere assumptions has to be suspect. It is hardly scientific. The possibility of swinging an election that is as important as this one, either by means of deliberate cooking of the numbers or even by means of chance requires that we ask questions. I’m not persuaded by a mere assertion that everything is on the up and up.

They are career bureaucrats. They are not party hacks. There may be reason to doubt their methodology but not their honesty. It’s well known that Nixon tried to get to them and failed. There has never been so much as a whiff of scandal associated with these public servants, so the reason Welch is being called out is that he made a crude insinuation without any evidence whatsoever. It was the equivalent of me saying, “I don’t know what’s going on with Bobby V, but there sure are some dead hookers. I’m just sayin. . .”

I’m being kind suggesting Welch is senile. The alternative is that he’s an assh0le who trashed hardworking people without any any evidence and solely in order to further his selfish political agenda.

If we accept that there are flaws, the real question is are these flaws unique to the Obama era, or have they existed across previous administrations?

As I have suggested repeatedly, those flaws were pointed out when the left was chanting “Jobless recovery” under the Bush administration., and charging that the use of U3 numbers was deceptive. So, no, they’re not particularly unique, of themselves. What is fairly unique is the way the Obama WH and the press are touting these numbers as if Obama worked some kind of miracle… just when he needed to to save his butt after the (Lack of ) performance in Denver.

In any event, funny how now suddenly the bets get hedged and there’s now suddenly legit reasons to question the reports.

I suppose that a jobs report that more conforms to the pattern of the previous 44 months of Obama is in the offing, to be released mere hours before the polls open… At which point, we’ll hear charges from the left abot how the whole thing is rigged against Obama.

If this were not simply crap that you made up, you might have a point. But that quote only exists between your ears. Back here in the real world, Obama has said more times than I can count that we have made progress, but times are still hard and we have a long way to go. That you are unable to give Obama any credit for the modest gains that have been made and lie about how they present them is consistant with your pathology.

Personally I will take tepid job growth over the 500K a month job loss rate that Obama inherited.

@Scott: WRT ordinary citizens understanding statistics and science: This is precisely why it matters that the state of Texas is deliberately dumbing-down our children. This is why it matters that “Honey Boo-Boo” is on the Learning or History Channel or whatever the capitalist substitution for PBS is this week.

When our population is stupid, they are easily manipulated. It’s our modern equivalent of the Southern Strategy.

What happened to you to make you so hopelessly naive about how government spending really works?

I have a relative who is severely disabled. The government actually does stuff for people like that. I have a mother and a mother in law who are elderly. They actually need Social Security and Medicare.

I went to a public college and university. Got me out of a dead end job and into a well paying career.

Why don’t you get back to me when you have a little more life experience under your belt?

Sorry, but I call bullshit on “Harvard Economics Professor Greg Mankiw”. I seem to recall he had a more prominent role a few years ago. President Dush? Gush? Something like that. I read that line and went “right. Done now. Its a Mataconis piece full of bad faith arguing’

Mankiw is an interesting combination of things. He’s a bright economist, who likes to lie now and then, and runs a blog without comments for that reason. When he’s reasonable (carbon tax) that’s fine, when he goes full partisan he doesn’t want push back. Luckily other econ blogs bring it.

I can easily see why people could suspect that an administration which is directing companies to explicitly ignore worker protection laws requiring layoff notices be sent out (and offering to pay the legal costs and fines these companies may incur from ignoring the law) might also put their thumb on the scale to influence employment reports to help their re-election campaign.

Major defense companies said this week they will not send out layoff notices to warn of big job cuts in January, taking away the prospect of embarrassing layoff notices right before the November elections. That’s led to charges that the White House overstepped when it told the industry the notices are not needed.

The White House went a step further, promising to cover any legal costs the industry might face because workers can sue if they’re fired without proper notice.

“Nobody has talked to workers or their unions about the impact of this,” says Thomas Buffenbarger, president of the International Association of Machinists and Aerospace Workers.
Buffenbarger says the White House took care of management’s concerns, but ignored workers, who still might lose their jobs next year. He says the White House is trying to manipulate his members.
“With a pink slip in one hand and their ballot in the other, how do you think they’re gonna vote?” Buffenbarger says. “Well, I would suspect they would vote against whoever the incumbent is, Republican or Democrat.”

Vast amounts of federal spending does nothing but take taxes paid by people who have less money and give it to other people who have more money. Ag Subsidies are a good example of this and they represent a tiny amount of that type of federal spending.

The college education that helped you is burying the youth of today in crushing, inescapable, life ruining debt.

A degree does not guarantee you or your children a good job anymore. In fact, it doesn’t guarantee you a job: last year, 1 out of 2 bachelor’s degree holders under 25 were jobless or unemployed.

We keep telling young Americans that a bachelor’s degree in history is as valuable as, say, a chemical engineering degree — but it’s just not true anymore.

All degrees are not created equal. And if we — parents, educators, entrepreneurs and nonprofit leaders — maintain this narrow-minded approach, then we are not just failing young indebted Americans and their families. We are harming the long-term vitality of our economy.

I don’t subscribe to the notion that GDP needs to grow by 5% for unemployment to fall .3%.In fact it has a whiff of totally made up statistic. I will note that when the GDP fell by 8% unemployment grew by 4%.

Vast amounts of federal spending does nothing but take taxes paid by people who have less money and give it to other people who have more money.

So let’s end corporate welfare and send the money to people that actually need it. There are a lot of them.

A degree does not guarantee you or your children a good job anymore.

Anyone who expects a “guarantee” of anything in life is a fool. Look, the world has changed. For much of my life, we had 5% of the worlds population consuming roughly a third of it’s resources. Pretty sweet if you were an American in the post WW2 world. Well, that world is gone, and Americans can’t simply pass Go and collect $200 anymore..

Don’t blame it on the liberal arts, and don’t blame it on the government. History is trying to resume its normal course. 1-2% of the population owns pretty much everything, there is a small elite that serves the owners/rulers and does well, and everybody else is a peon. The question is, will Americans just sit back and let it happen.

@Duracomm: So…closing a few federal agencies and departments, changing how military items are procured, and means testing (it would help if you could be a little more specific with that last one) will help to significantly balance the budget? Hmm, these things alone don’t seem to be nearly enough…meanwhile, we have all these people like you telling us that cuts are definitely going to be made…and yet, we’ve been living with significant debts since Reagan, and somehow, we are still here…

I can easily see why people could suspect that an administration which is directing companies to explicitly ignore worker protection laws requiring layoff notices be sent out (and offering to pay the legal costs and fines these companies may incur from ignoring the law) might also put their thumb on the scale to influence employment reports to help their re-election campaign.

Yes, keep telling yourself that it is all a conspiracy…when the President wins reelection, that will be part of the conspiracy too…

In the end cuts are going to be made because Washington spending is outrunning the economies ability to fund the politicians spending addiction.

No cuts made can balance the budget, assuming that is the goal. You forget there are two components to the federal budget, the discretionary component and the cyclical component. The discretionary component is what government chooses to fund during a given fiscal year (the military, EPA, Department of Justice, Forest Service etc.). The cyclical component is sensitive to other factors such as tax revenues and welfare payments. Our current deficits are primarily a product of the cyclical component.

Attempts to reduce the deficit by adopting a contractionary discretionary stance will simply exacerbate the cyclical components by reducing revenues, generating additional unemployment and causing welfare payments to rise. Even if government were to make cuts to welfare payments this would not solve budget deficits because the cuts would drive down revenues as spending contracts.