Saturday, January 25, 2014

Chris House sticks up for macro

In a recent article in The Week, I claimed that a lot of econ fields are doing better than macro. Now Chris House, my old macro sensei, jumps in to defend his field from his disgruntled ex-student.

Before we get into the back-and-forth, there's the interesting question of who you should believe, an experienced insider or a bomb-throwing outsider. Chris is a successful macroeconomist at an excellent university. I am not a macroeconomist at all; my experience is limited to two years of grad school macro and a number of academic conferences. Insiders tend to be biased, because people go into fields that they enjoy and believe in, and they also have a vested interest in promoting the field. But outsiders tend to be ignorant. So after I respond to Chris' points, I'll bring in a recent critique of macro from an accomplished insider in the field.

Ha. I kid. Actually I do not have it in for macroeconomists. A lot of the economists I admire most are macroeconomists - Markus Brunnermeier, for example, or Martin Eichenbaum, or Ricardo Caballero. Miles Kimball and Mark Thoma and David Andolfatto, who are among my favorite bloggers and are awesome people besides, are macroeconomists. All of my favorite classes at UMich were taught by macroeconomists. I enjoy going to macro conferences much more than I'd enjoy going to a game theory conference or a labor econ conference. There are certain macroeconomists who do annoy me, of course, but that will be true in any field.

What annoys me (warning: pretentious phrasing ahead) is the intellectual culture of macro. Which lots of macroeconomists are working to change from the inside (which of course is more admirable than simply running away and throwing bombs over their shoulder, as I did). By helping to focus popular annoyance away from econ as a whole and toward macro in particular, my hope was to help the internal critics make headway toward reform.

Anyway, back to Chris:

Macro is quite productive and overall quite healthy. There are several distinguishing features of macroeconomics which set it apart from many other areas in economics. In my assessment, along most of these dimensions, macro comes out looking quite good.

First, macroeconomists are constantly comparing models to data...Of course it is a problem that the theories are so soundly rejected...In many other areas in economics the theories aren’t rejected because...the theories simply don’t exist. There are many purely empirical studies in which there is little theory to speak of.

Sorry, but I fail to see why "purely empirical" studies with "little theory to speak of" are worse than "theories [that are] soundly rejected." That just doesn't make any sense to me. What is wrong with looking at the world without first trying to force it into some mental framework that you came up with?

And what are macroeconomists doing with all those rejected theories? Are they throwing them away? Are they keeping them on the shelf in case they're needed? Do people go on believing and using them anyway because they seem cool? (Actually Chris partially answers this later on...)

Also, Chris is overstating his case when he says that macroeconomists are "constantly comparing models to the data". Sometimes these comparisons are not particularly impressive. For example, many macro models are compared with the data by "moment matching", which is just about the lowest possible bar for models to pass. But lots of macro papers are satisfied with "moment matching".

Chris also mentions that in some branches of econ, theories are rarely compared to data. I assume he's talking about pure game theory. But pure game theory is really a branch of mathematics, not an empirical science, so it doesn't need to test itself. Applied game theory is continually tested against data, and many applied game theory models have been so successful that they have found widespread application in industry. And other than pure game theory, I'm pretty sure all branches of econ compare their theories to data.

Back to Chris:

Second, in macroeconomics, there is a constant push to quantify theories. That is, there is always an effort to attach meaningful parameter values to the models...This is again one of the ways in which macro is quite unlike other fields.

So, theories that are "soundly rejected" by data, but which have very well-quantified parameter values? What is the point of precision without accuracy? I mean, it's nice, why not be precise, but what's the point of measuring the 5th decimal place of a parameter that probably doesn't even represent any real thing?

It's kind of like "Oh, if this theory were right, you could boost growth by 1% a year by cutting taxes to 20%, but according to our data this theory is nowhere close to right." So who cares if the 1% number is big? Again, I kind of don't get it.

Back to Chris:

Third, when the models fail (and they always fail eventually), the response of macroeconomists isn’t to simply abandon the model, but rather they highlight the nature of the failure. This is again a good research habit because mistakes and rejections have value – knowing the nature of the mismatch between the model and the data helps you to refine the theory. There are many “puzzles” in macroeconomics (the excess sensitivity puzzle, the risk-free-rate puzzle, the equity premium puzzle, the international comovement puzzles, and so on, …). At a superficial level one might be tempted to conclude that the prevalence of such puzzles shows that the field is in a constant state of disarray. In fact, these mismatches between theory and data serve as an important guide to how to modify the theories.

Ah, so this is what happens to all those theories that don't match the data! They become "puzzles", which then generate more macro papers.

Look, I can make up any number of theories that don't match reality. I can generate "puzzles" for you all day long. For example, my theory is that recessions are caused by a giant flying mongoose named Skippy that appears whenever investment is low and breathes a magic Breath of Sadness that makes people not want to work for a few years (note that this model matches the negative correlation between investment and unemployment). Sadly, we have not yet managed to observe Skippy the Giant Mongoose, but that's OK; we can spend the next ten years of our lives publishing possible solutions to the Mongoose Deficit Puzzle.

Silly. But only silly because Skippy the Mongoose sounds implausible to you. How about the idea that stocks consist of claims on the output of fruit trees that give a random amount of fruit each year in an i.i.d. normal distribution, and that people care only about the amount of fruit they eat each year? That's the setup in the Equity Premium Puzzle. Sounds a bit more plausible than Skippy the Mongoose, I'll grant you, but not necessarily so plausible that its failure to match reality should have been hailed as a great theoretical triumph.

The Equity Premium Puzzle did get a lot of finance people interested in investigating stuff that eventually proved very interesting, I'll grant it that. And it spurred people to think some very deep thoughts. So it is a very important paper. But if all you ever get are puzzles and puzzles and more puzzles, then at some point maybe the real puzzle is why you're still at it.

Finally, back to Chris:

Lastly, unlike many other fields, macroeconomists need to have a wide array of skills and familiarity with many sub-fields of economics. As a group, macroeconomists have knowledge of a wide range of analytical techniques, probably better knowledge of history, and greater familiarity and appreciation of economic institutions than the average economist.

I agree, that is a reason many macroeconomists are awesome.

Anyway, so I think this argument - which, as you might guess, is basically a replay of discussions Chris and I had in his office back in 2007 - pretty clearly demonstrates my problems with the intellectual culture of macro. The Macro Way of Thinking just values stuff that I don't value very much. I guess that's a matter of taste.

But now it's time for me to point out that a number of actual, practicing, successful macroeconomists are saying stuff similar to what I'm saying (though in more upbeat, polite, and constructive ways). For example, Ricardo Caballero of MIT wrote this comment about "Macroeconomics After the Crisis", which I pretty much agree with 100%. The abstract:

In this paper I argue that the current core of macroeconomics—by which I mainly mean the so-called dynamic stochastic general equilibrium approach—has become so mesmerized with its own internal logic that it has begun to confuse the precision it has achieved about its own world with the precision that it has about the real one. This is dangerous for both methodological and policy reasons. On the methodology front, macroeconomic research has been in “fine-tuning” mode within the local-maximum of the dynamic stochastic general equilibrium world, when we should be in “broad-exploration” mode. We are too far from absolute truth to be so specialized and to make the kind of confident quantitative claims that often emerge from the core. On the policy front, this confused precision creates the illusion that a minor adjustment in the standard policy framework will prevent future crises, and by doing so it leaves us overly exposed to the new and unexpected.

And an excerpt:

To be fair to our field, an enormous amount of work at the intersection of macroeconomics and corporate finance has been chasing many of the issues that played a central role during the current crisis, including liquidity evaporation, collateral shortages, bubbles, crises, panics, fire sales, risk-shifting, contagion, and the like. However, much of this literature belongs to the periphery of macroeconomics rather than to its core...The dynamic stochastic general equilibrium strategy is so attractive, and even plain addictive, because it allows one to generate impulse responses that can be fully described in terms of seemingly scientific statements. The model is an irresistible snake-charmer. In contrast, the periphery is not nearly as ambitious, and it provides mostly qualitative insights. So we are left with the tension between a type of answer to which we aspire but that has limited connection with reality (the core) and more sensible but incomplete answers (the periphery).

Caballero is describing the "precision without accuracy" problem. Macro's current culture values very precise, detailed, deep knowledge about models that have so far given us no reason to believe in them. But that knowledge is not likely to ever be valuable to people outside the field.

(His proposed solution - that macroeconomic theorists should focus more on picking away at the problem, understanding macro-relevant "microeconomic" phenomena instead of trying to swallow the whole business cycle in one big arrogant gulp - seems right to me too. But that is a topic for another post.)

Anyway, so if you really want a critic of macro who is an eminent macroeconomist rather than a disgruntled bomb-thrower who quit the field after two and a half years of grad school, go read Caballero.

"Second, in macroeconomics, there is a constant push to quantify theories. That is, there is always an effort to attach meaningful parameter values to the models...This is again one of the ways in which macro is quite unlike other fields."

In a social science you do not apply reality to theories. Theories come after looking at the reality. Of course you can fit the world to your theory if you want to. But that is very dangerous.

Other social scientists know that truth is often what you make of it. That is why you start every time from the primary historical material and the raw statistical data. You then examine the context of that data and what it means. After all that is done you can look at competing theories - Keynesian, Marxist, Marshallian whatever as reference points only. But NEVER let a theory guide your analysis (or do things like take theories out of another discipline like mathematics) - because inevitably it will influence the result.

Isn't the fundamental issue one of data? You can never really dismiss macro models because there are always two easy responses to a lack of empirical evidence. What about the counterfactual, and the data was rubbish.

Which leads me to another question. How can you build a complex model with rubbish data? The more complexity, the more precision needed (since estimation errors multiply) and therefore the better data needed. How many billions of data points do physicists use to conduct experiments on complex systems, or weather or climate models? And at least the thing they are measuring is pretty clear cut. This clearly isn't the case for even the most basic of macroeconomic datapoints like GDP, inflation and unemployment.

Also why do macro economists insist on building models with assumptions they know to be false? Why not build models with assumptions you know to be true. Or if you are not sure to be true, test if they are true first before building a huge complex model based on dubious assumptions?

Do you know how modeling works in complex systems? Which of the billions possible assumptions should one make? All of them? How would we ever resolve what is true assumption or not if we have limited data? How would you know which assumptions matter and which don't?

Isn't theory supposed to guide research, suggest questions, and result in useful conclusions? Empirical studies without theory would be blind guesses in the dark. Imagine a life scientist trying to determine how cancer cells function who refuses to use evolutionary theory and just runs batch file regressions on empirical data. How far would that get you?

"What is wrong with looking at the world without first trying to force it into some mental framework that you came up with?"

Hindsight bias, publication bias, and worthless t statistics. Publication bias in this case favors two kinds of results: Spectacular results and results favoring commonsense.And yes, I believe that many economists are "doing random batch file regressions on empirical data". Maybe not individually, because each one of them has an idea how the world works, but as a profession, they do that.

No., a theory is an explanation that ties a wide body of evidence together and has widespread support in the scientific community as the correct explanation. It is not a hypothesis. For example, evolution is a theory in the life sciences and relativity is a theory in physics. A hypothesis is just an educated guess.

In the absence of theory what else would scientists do BUT run batch file regressions? Professor House is pointing out that Macro has a wide body of theory to inform research, so batch file regressions are neither needed nor appropriate. The theory helps him ask good questions that lead to models with testable implications. The data can then be used to modify the theory or even falsify it if the implications do not agree with the evidence. No one scientist does this. Only widespread consensus in the scientific community results in such a modification to theory.

You mistakenly suggest that you have a theory you call Skippy the Giant Mongoose. Sorry, Noah, that would at best be a hypothesis. You would have to convince a wide body of macroeconomists that Skippy explains the data better than any other explanation before it could be called a theory. Good luck with that, as you don't believe it yourself.

No., a theory is an explanation that ties a wide body of evidence together and has widespread support in the scientific community as the correct explanation.

Well that can't be right. Theories exist before they gain widespread acceptance, and they exist after they have fallen out of favor. I think you need to rethink your definition of "theory".

Actually, I can't tell the difference between a theory and a hypothesis, when I think about it.

In the absence of theory what else would scientists do BUT run batch file regressions?

This question should make you think very carefully about House's critique. Obviously, the people he describes as "not having a theory" are NOT running batch file regressions. No one is doing that (I hope). So what are they doing? What the heck is House talking about?

Think about it.

You mistakenly suggest that you have a theory you call Skippy the Giant Mongoose. Sorry, Noah, that would at best be a hypothesis.

No, it's a very fully fleshed-out theory, I just didn't tell you all of the details. ;-)

You would have to convince a wide body of macroeconomists that Skippy explains the data better than any other explanation before it could be called a theory.

Again, this is just silly. Open the AER and pick out any theory paper, I almost guarantee that that paper will not have convinced a wide body of macroeconomists of anything. Because a wide body of macroeconomists is unlikely to have read it.

Well then, by Ben Wheeler's chosen definition, none of the "theories" discussed by Chris House are real "scientific theories", since as Chris House notes, none of them are supported by vast bodies of evidence.

Not my definition. This is the definition of the National Academy of Sciences.

Macro theory is certainly not as well developed as atomic theory or plate tectonics. It is full of holes. But that doesn't mean there isn't a foundation to macro that has wide support amongst macroeconomists. Doesn't David Romer's book enjoy widespread support amongst macroeconomists?

I love this post. I think House nailed what is totally wrong with macro with this clause " refine the theory". He just assumes that current theory should be refined not thrown away. The possibility that say the Euler equation is not part of a useful effort to understand consumption is too horrible for him to contemplate.

The results are two. First that some of the basic assumptions used by macroeconomists are totally invulnerable to data because it is assumed, a priori, that that they belong in the model and that the failure of the model means something else has to be added somewhere. The second is that whatever is added somewhere to solve the puzzle is not added when other questions are addressed. The failed model is assumed to be a good first order approximation, because it is the dominant model with no regard for reality.

In the history of thought, some models have just been abandoned. House asserts that this is a bad thing to do. He made that assertion with no qualifications. His defense of macroeconomics makes sense only if he thinks chemists should still be refining the phlogiston model.

But he is polite and it is very nice of him to discuss things with a renegade former student.

Say what you will about macro but it's a late bloomer field. It's been three hundred years of walking blind folded with only noise, smell, and light to guide the way. But fold is lifting very slowly. I feel that in the future it'll be indispensible to society. When the blindfold is lifted and data is insanely bountiful and measurable, then it'll be pretty damn close to seeing the future, like Foundation. Huh, now I get where Paul Krugman got the macro bug.

" Anyway, so if you really want a critic of macro who is an eminent macroeconomist rather than a disgruntled bomb-thrower who quit the field after two and a half years of grad school, go read Caballero."

As a group, macroeconomists have knowledge of a wide range of analytical techniques, probably better knowledge of history, and greater familiarity and appreciation of economic institutions than the average economist.

No. Well maybe better than other economists but the way macro-economists look at history is appalling to a real historian. An historian never uses hypothesis testing or models, for very good reasons.

A good example of what could be considered a "seminal" macro paper but a very questionable piece of historical analysis is Sargent's "End of Four Big Inflations". In that paper he argues that hyperinflation and its stabilisation was linked to guaranteed reserve/money supply ratios.

In fact it had a lot more to do with things like the countries in question had lost major wars and were basically failed states.

Aaaauuuugggghhh. The financial crisis was not the result of something new and unexpected. We have "known" for hundreds of years that bankers descend into speculation and fraud unless they are watched like hawks.

The experience of the last ten years is not just a rebuttal to macro economics - it is an indictment.

I like House's piece. And you're not going to dig yourself out by pointing the way to Caballero. Caballero is essentially saying: "what I see under my nose doesn't tell me a lot about what I want to know." In some cases where Caballero is arguing that people aren't doing something, you can find plenty of capable research where people have been doing the thing they were supposed to have not been doing. So Caballero knows more macro than you do, but in his article unfortunately he's apparently ignorant of much that matters.

And you're not going to dig yourself out by pointing the way to Caballero.

Steve, to dig oneself out requires that one starts buried. Curiously I find myself with nothing out from under which to dig myself.

In some cases where Caballero is arguing that people aren't doing something, you can find plenty of capable research where people have been doing the thing they were supposed to have not been doing.

Allow me to make the humble suggestion that it might be a fruitful exercise to read Caballero's piece with greater attention to detail, for he argues not that his preferred methods are absent from the field, but that they reside within the field's "periphery" rather than its "core"...

On Caballero, I think "peripheral" might mean what is discussed outside the NBER fluctuations group. There is plenty of "peripheral" stuff published over the last 40 years in the mainstream journals, on the topics he mentions, for anyone who cares to look.

Caballero's piece was a complaint written more than 3 years ago. Since then, it's hard to call this stuff peripheral, even in Caballero's sense. We now have more papers on financial frictions and financial crises, than we can shake our collective sticks it. So why you want to think about Caballero now is not clear. And Barro??? Talk about flogging a dead horse.

Caballero's piece was a complaint written more than 3 years ago. Since then, it's hard to call this stuff peripheral, even in Caballero's sense. We now have more papers on financial frictions and financial crises, than we can shake our collective sticks it.

So you're saying that in the last 3 years, the macro field has changed substantially, along the lines that Caballero called for?

And Barro??? Talk about flogging a dead horse.

True. And I don't even know why they keep replaying Martin Luther King's speeches on the news...that was like, half a century ago.

The big point Chris house is making is that macroeconomics knows something, uses it to build more complex models, tests them against data and refines accordingly. However, the big question is: does macroeconomics today know anything? Know in the sense of having powerful evidence for I whatever it is. Not in the sense, like with string theory, that it's so beautiful and there are no alternatives.