advice from a fake consultant

Tuesday, November 17, 2009

On Determining Impact, Or, How Stimulative Is Stimulus?

We strive to be, if anything, a participatory space around here, and I’ve had a question come to my inbox that is very much deserving of our attention.

To make a long story short, our questioner wants to know why, on the one hand, despite the passage of the American Recovery and Reinvestment Act of 2009 (ARRA, also known as the “stimulus”), unemployment in the construction industry continues to increase, and, on the other hand, why there is such a giant disparity, on a state-by-state basis, in the cost of saving a job?

They’re great questions, and, having done a bit of research, I think I have some cogent answers.

A few facts will help set the stage:

I post on numerous sites, one of those being Blue Oklahoma, and about ten days ago I received an email from a reader who wanted me to know that he had data up regarding how effective stimulus dollars are at creating construction jobs.

He also wondered if I would be willing to blog about his work, which is itself posted in the form of a blog, with handy charts and graphs; I’ll quickly summarize what he had to say for your dining and dancing pleasure:

Although the goal of the stimulus was to create construction jobs, today’s data suggests that roughly 10 times as many jobs were lost in the construction industry in the recent past 12 months (September 2008 – September 2009) than were created by the stimulus efforts this year to date.

“The major question surrounding the ARRA and the construction industry on this reporting deadline is: How many construction jobs has the stimulus bill actually created or retained?”

In the blog he reports that if you were to go to the Recovery.gov website, download the state summary data located there, and then do a bit of quick math, you’d find that:

“…the ARRA has created or saved 73,352 construction jobs across the nation at a total cost of $15.8 billion since the bill was signed into law.

That’s $222,107 per construction job.”

He has also created a chart, that is intended to show, on a state-by-state basis, the cost per job—and there is enormous variation in the results, from a low of $47,536 in Minnesota to a high of $535,171 in California.

As a result, he’s come to this conclusion:

“Jobs are being created and saved but nowhere near a rate that will allow the stimulus bill to claim victory over construction unemployment.”

So the question for us becomes: how solid is his analysis?

In order to get a better answer, I decided to examine some of the underlying data supporting his conclusions—and to put it as gently as possible, the numbers that we’re seeing today are a bit…squishy.

There are a couple of reasons why, which, naturally, require a couple of quick explanations. (This is a “quick and dirty” education; there are exceptions to some of what you’ll see described below.)

Right off the bat, it appears that identifying exactly how many jobs are being saved is more difficult than it seems—but before we can really understand that, we need to take a moment and understand exactly how jobs are counted.

If you work 40 hours a week, which is the equivalent of a full-time job, you would equal one (warning: technical term ahead) “Full Time Equivalent”, also known as an FTE. Two people, each working 20 hours a week, are also one FTE, as are any other combinations that you can come up with that get you to 40 hours a week. From here on, when we use the word “jobs”, we also mean FTEs, and vice versa.

The rules of the stimulus program are unique unto themselves, and one of the unique rules, at least for the moment, is that overtime hours don’t count when counting FTEs; since we’re talking about the number of construction jobs the stimulus might be creating, and about 25% of construction jobs involve overtime work, this rule is probably distorting the outcome.

States are also having problems translating FTE tracking systems they already have in place into the new Federal FTE definitions being used to figure out how many jobs are being created with stimulus funds.

An example of this problem is laid out in a document from the University of Connecticut (UConn) describing how their FTE reporting is going. The State FTE tracking system uses “cumulative” reporting, the Federal system, “incremental” reporting; the only thing you need to know about the two systems is that, quoting from the report:

“…At no time would the state and federal FTE figures match.”

There’s another issue in play here: this is a brand-new bureaucracy, and everyone is still “finding their way”, on both the State and Federal sides. Here’s another quote from the same UConn progress report:

“Note #2: We have experienced challenges in reporting, primarily with formatting issues. Solutions include working directly with OPM [the State’s Office of Policy and Management] and the respective OSPs [the University’s Office for Sponsored Programs] to enhance timeliness and formatting accuracy. The reports submitted June through September 2009 were definitely part of the learning process. Currently, we are working directly with the respective OSPs to ensure the correct reporting templates are used for state reporting purposes.

Put all that together, and you have a collection of “structural” issues that will probably cause the “real” construction FTE numbers to be somewhat different from today’s “reported” numbers by some currently unknown amount that can probably be “estimated out” later on.

The biggest distortion in statistics, however, is a “timeline” issue, and it’s because trying to estimate the “cost per FTE” at the beginning of construction projects is inherently problematic.

To illustrate this point, let’s drill down to one individual project and see how things work:

Award number OK56S09550109 was granted to the City of Shawnee's Housing Authority to modernize the HVAC (Heating, Ventilation, and Air Conditioning) system at a public housing development. The current reporting is that $856,585 was awarded for the work, for which 2.5 FTEs have been reported.

However, as of the reporting date only $61,674 has been expended (or $70,084--both numbers appear on the same webpage); that money going to Childers-Childers, Architects.

The 2½ FTEs are .5 each of two administrators and 1.5 architects.

Obviously there will be more jobs created as this project moves from design to construction, and the estimate of roughly $340,000 per FTE that could theoretically be cited as accurate today will no longer be valid once a bunch of people show up and actually start installing stuff.

In fact, it could be reasonably argued that the "correct" number is $24,669 per FTE (or $28,034), based on the amount expended and jobs created to date.

This “timeline issue” is a statistical problem that Thorman himself acknowledges in his blog:

“With 73,352 jobs created/saved during this reporting period, the number will undoubtedly go up in future months as more projects begin and as more projects enter more labor-intensive phases. The construction jobs created/saved by the stimulus will likely get better before they get worse.”

(Just for the record, a third method you could use to count FTEs would be to divide total grant awards against total estimated construction employment throughout the lifetime of these projects.)

You may recall that the reason we’re having this discussion is because we are trying to come to some conclusion about what impact the stimulus is having on creating jobs—or, alternatively, creating even more geeky FTEs.

Well, having looked at the thing all the way down to the individual project level, it may be that the best answer that’s available…is that there’s no answer yet available.

With that in mind, my conclusion is that we will need some time to create a large enough “statistical universe” of completed or nearly-completed projects before we can begin to make useful extrapolations about the stimulus’ future success, and my guess is that it will be six to 12 months before that threshold is reached…which means I have no idea whether the stimulus is creating or will create a sufficient number of construction jobs relative to its budget, and It may well be summer of 2010 before we do know.

And that, my fellow political observers, has the potential to make the ’10 Congressional midterms very, very, interesting.