Site Mobile Navigation

Is Our Adults Learning?

In 2009, we had a big debate about whether to pass a stimulus package. Many esteemed and/or Nobel Prize-winning economists like Joseph Stiglitz, Larry Summers and Christina Romer argued that it would help lift the economy out of recession. Many other esteemed and/or Nobel Prize-winning economists like Robert Barro, Edward Prescott and James Buchanan argued that positive effects would be small and the package wouldn’t be worth the long-term cost.

We went ahead and spent the roughly $800 billion. What have we learned?

For certain, nothing. The economists who supported the stimulus now argue the economy would have been worse off without it. Those who opposed it argue that the results have been meager. It’s hard to think of anybody whose mind has been changed by what happened.

This is not entirely surprising. Nearly 80 years later, it’s hard to know if the New Deal did much to end the Great Depression. Still, it would be nice if we could learn from experience. To avoid national catastrophe, we’re going to have to figure out how to control health care costs, improve schools and do other things.

Jim Manzi has spent his career helping businesses learn from experience — first at AT&T Laboratories, then as a consultant with Strategic Planning Associates and then as founder of Applied Predictive Technologies, a successful software firm.

In his new book, “Uncontrolled,” Manzi notes that many experts tackle policy problems by creating big pattern-finding models and then running simulations to see how proposals will work. That’s essentially what the proponents and opponents of the stimulus package did.

The problem is that no model can capture enough of the world’s complexity to yield definitive conclusions or make nonobvious predictions. A lot depends on what assumptions you build into them.

In “Uncontrolled,” Manzi looks at two celebrated model-building exercises. Larry Bartels of Princeton produced a model finding that presidential policies exercise the single biggest influence on income distribution. The authors of “Freakonomics” produced a model showing legalized abortions subsequently reduced crime rates.

Photo

David BrooksCredit
Josh Haner/The New York Times

Manzi argues that by slightly tweaking the technical assumptions in these models, you eliminate the headline-grabbing results. He also points out that regression models that try to explain crime rates have not become more accurate over the past 30 years. All this model-building hasn’t even helped us get better at understanding the problem.

What you really need to achieve sustained learning, Manzi argues, is controlled experiments. Try something out. Compare the results against a control group. Build up an information feedback loop. This is how businesses learn. By 2000, the credit card company Capital One was running 60,000 randomized tests a year — trying out different innovations and strategies. Google ran about 12,000 randomized experiments in 2009 alone.

These randomized tests actually do vindicate or disprove theories. For example, a few years ago, one experiment suggested that if you give people too many choices they get overwhelmed and experience less satisfaction. But researchers conducted dozens more experiments, trying to replicate the phenomenon. They couldn’t.

Businesses conduct hundreds of thousands of randomized trials each year. Pharmaceutical companies conduct thousands more. But government? Hardly any. Government agencies conduct only a smattering of controlled experiments to test policies in the justice system, education, welfare and so on.

Why doesn’t government want to learn? First, there’s no infrastructure. There are few agencies designed to supervise such experiments. Second, there is no way to conduct a randomized experiment to test big economywide policies like the stimulus package.

Finally, the general lesson of randomized experiments is that the vast majority of new proposals do not work, and those that do work only do so to a limited extent and only under certain circumstances. This is true in business and government. Politicians are not inclined to set up rigorous testing methods showing that their favorite ideas don’t work.

Manzi wants to infuse government with a culture of experimentation. Set up an F.D.A.-like agency to institute thousands of randomized testing experiments throughout government. Decentralize policy experimentation as much as possible to encourage maximum variation.

His tour through the history of government learning is sobering, suggesting there may be a growing policy gap. The world is changing fast, producing enormous benefits and problems. Our ability to understand these problems is slow. Social policies designed to address them usually fail and almost always produce limited results. Most problems have too many interlocking causes to be explicable through modeling.

Still, things don’t have to be this bad. The first step to wisdom is admitting how little we know and constructing a trial-and-error process on the basis of our own ignorance. Inject controlled experiments throughout government. Feel your way forward. Fail less badly every day.

A version of this op-ed appears in print on April 27, 2012, on page A27 of the New York edition with the headline: Is Our Adults Learning?. Today's Paper|Subscribe