It’s hard to know how computer models work

David Brooks

The computer model is the Delphi oracle of the modern age, using software and mathematics rather than vapors and animal sacrifice to create mystical answers that guide our actions.

We hear predictions all the time about what is going to happen with the economy, the climate, the election, the ecosystem, the behavior of the entire universe, all based on what computer models say. But we never see how these models work, just as ancient Greeks never got to see how the oracle came up with her predictions in the Delphic grotto.

So I jumped at a chance to pick the brains of the author of one software oracle, in hopes of getting some insight.

The model is called the Connect NNE Economic Scenario Model, and it is designed to help planners predict what will happen to jobs, commuters and other economic aspects of the county or state if they do various actions, like attract a certain type of industry.

The model uses jobs and income data from the federal Bureau of Economic Analysis or Bureau of Labor Statistics. You type in scenarios you want to examine, including numbers of new jobs, types of industry arriving or departing, proposed construction projects, and the like, then see how the number of jobs, earnings and economic output changes for places as small as a single county. (Economic models usually cover whole regions or metropolitan areas; a key benefit of Connect NNE is it handles places as small as Coos County.)

FairPoint Communications is handing out free copies and holding training sessions to help people use it, in hopes of goosing the region’s economy and therefore the amount of phone and Internet service that gets bought. The fact that it burnishes FairPoint’s image among industry and government folks doesn’t hurt, I imagine.

My first surprise is that the computer model is based on plain old Excel, Microsoft’s ubiquitous spreadsheet software, and isn’t that huge a file, at 4 megabytes. You filter it with plain old pull-down menus and tabs, and type numbers into little spreadsheet cells.

Yet it is sophisticated enough that Dennis Delay of the New Hampshire Center for Public Policy Studies has used it twice to inform legislators about such things as the likely economic impact of expanded gambling on the state.

“I’ve found this modeling system useful for answering what-if questions, and I think the answers that you get are reasonable,” said Delay. “It’s a lot easier to use, in a spreadsheet as compared to an esoteric modeling system, and they populated it with data for northern New England counties.”

So how does it work?

Mark Madsen, senior economist for the firm VitalEconomy, who oversaw the development, spent an hour on the phone with me last week, giving me the layman’s walkthrough.

The model is built on the Nobel Prize-winning work of Wassily Leontief, whose development of linear algebra-powered input/output analysis – how to quantify the effect that changing one economic item has on a different economic item – is a foundation of the field. (A measure of Leontief’s notability: No fewer than three of his graduate students also won economics Nobels, including Paul Samuelson of Econ 101 textbook fame.)

“Economic models can range from the very simple to the very complex. ... When we say we’re going to model something, we have to narrow down all these different data sources we have, all these different behavioral matrices we have,” Madsen said.

In this case, the key statistic is jobs, which is well-measured and whose effect on other parts of the economy via what are called “multipliers,” is well-understood. Other factors that are harder to pin down, such as consumer behavior, were avoided.

Why are the ripple effects of jobs well-understood? Because scads of research economists, at universities, think tanks and companies, have examined the issue for decades.

It’s complicated because economics can’t be tested in the double-blind experimental format that makes “hard science” so successful. You can’t, for example, freeze all hiring in Hillsborough County as a control group to examine the effect of hiring patterns in Rockingham County.

So professors design small experiments and try to extrapolate the results to the real world, or try to tease out hidden relationships from economic information gathered by governments.

“There is always some professor who has two or three graduate students who are testing data,” said Madsen.

They look at figures from the past and present, they’ve compared results in different places and different times, they have gone back and double-checked predictions from other models, and most importantly, they’ve written about their findings in esoteric, complicated, obscure publications, where they can be studied by people like Madsen.

And that is really the secret behind computer models – not the specifics of the algorithms or input patterns, but the accumulated knowledge base that judges this algorithm to be legitimate while that one isn’t, or this usage of the algorithm to work well while that usage is uncertain.

For those of us who want to understand these computer models, that’s sobering. It means that we can’t really judge a model without investing more time than is realistic.

Madsen, who spent a year developing Connect NNE on top of roughly five years of general computer-model development, is sympathetic.

“How does a lay person know whether an economic model is valid? It’s very difficult,” he said. “That’s when people have to say, either I’m going to invest my own time and look up citations and see how this was put together, or ask somebody who is an expert in the area and believe them.”

This opacity has the unhappy effect of empowering people to ignore computer models if they don’t like the results. This happens notably in the climate-change debate, where the phrase “computer models can prove anything you want them to” get tossed around, as code for “I’ll ignore results that don’t reinforce my belief.”

Unfortunately, my chance to peek inside an oracle’s cave indicates that there’s no easy way to combat that argument. Computer models can’t prove anything you want and we do need to pay attention to them – not if they’re good computer models.

The hard part is telling which ones are good. Maybe an animal sacrifice would help.

Granite Geek appears Mondays in the Telegraph, and online at www.granitegeek.org. David Brooks can be reached at 594-5831 or dbrooks@nashuatelegraph.com.