The blow-up over the Reinhart-Rogoff results reminds me of a point I’ve been meaning to make about our ability to use empirical methods to make progress in macroeconomics...it's about the quantity and quality of the data we use to draw important conclusions in macroeconomics.

Everybody has been highly critical of theoretical macroeconomic models, DSGE models in particular, and for good reason. But the imaginative construction of theoretical models is not the biggest problem in macro – we can build reasonable models to explain just about anything. The biggest problem in macroeconomics is the inability of econometricians of all flavors (classical, Bayesian) to definitively choose one model over another, i.e. to sort between these imaginative constructions. We like to think or ourselves as scientists, but if data can’t settle our theoretical disputes – and it doesn’t appear that it can – then our claim for scientific validity has little or no merit.

There are many reasons for this. For example, the use of historical rather than “all else equal” laboratory/experimental data makes it difficult to figure out if a particular relationship we find in the data reveals an important truth rather than a chance run that mimics a causal relationship. If we could do repeated experiments or compare data across countries (or other jurisdictions) without worrying about the “all else equal assumption” we’d could perhaps sort this out. It would be like repeated experiments. But, unfortunately, there are too many institutional differences and common shocks across countries to reliably treat each country as an independent, all else equal experiment. Without repeated experiments – with just one set of historical data for the US to rely upon – it is extraordinarily difficult to tell the difference between a spurious correlation and a true, noteworthy relationship in the data.

Even so, if we had a very, very long time-series for a single country, and if certain regularity conditions persisted over time (e.g. no structural change), we might be able to answer important theoretical and policy questions (if the same policy is tried again and again over time within a country, we can sort out the random and the systematic effects). Unfortunately, the time period covered by a typical data set in macroeconomics is relatively short (so that very few useful policy experiments are contained in the available data, e.g. there are very few data points telling us how the economy reacts to fiscal policy in deep recessions).

There is another problem with using historical as opposed to experimental data, testing theoretical models against data the researcher knows about when the model is built...It’s not really fair to test a theory against historical macroeconomic data, we all know what the data say and it would be foolish to build a model that is inconsistent with the historical data it was built to explain – of course the model will fit the data, who would be impressed by that? But a test against data that the investigator could not have known about when the theory was formulated is a different story – those tests are meaningful...

By today, I thought, I would have almost double the data I had [in the 80s] and that would improve the precision of tests quite a bit...

It didn’t work out that way. There was a big change in the Fed’s operating procedure in the early 1980s...

So, here we are 25 years or so later and macroeconomists don’t have any more data at our disposal than we did when I was in graduate school. And if the structure of the economy keeps changing – as it will – the same will probably be true 25 years from now. We will either have to model the structural change explicitly (which isn’t easy, and attempts to model structural beaks often induce as much uncertainty as clarity), or continually discard historical data as time goes on (maybe big data, digital technology, theoretical advances, etc. will help?).

The point is that for a variety of reasons – the lack of experimental data, small data sets, and important structural change foremost among them – empirical macroeconomics is not able to definitively say which competing model of the economy best explains the data. There are some questions we’ve been able to address successfully with empirical methods, e.g., there has been a big change in views about the effectiveness of monetary policy over the last few decades driven by empirical work. But for the most part empirical macro has not been able to settle important policy questions...

I used to think that the accumulation of data along with ever improving empirical techniques would eventually allow us to answer important theoretical and policy questions. I haven’t completely lost faith, but it’s hard to be satisfied with our progress to date. It’s even more disappointing to see researchers overlooking these well-known, obvious problems – for example the lack pf precision and sensitivity to data errors that come with the reliance on just a few observations – to oversell their results. (emphasis mine)

This is the clearest and based statement of the problem that I've ever seen. (Update: More from Thoma here.)

I'd like to add one point about the limits of time-series econometrics. To do time-series, you really need two assumptions: 1) ergodicity, and 2) stationarity. Mark addressed the ergodicity problem when he talked about trend breaks. As for stationarity, it sometimes matters a lot - for example, if technology has a unit root, then positive technology shocks should cause recessions. But the statistical tests that we use to figure out if a time-series has a unit root or not all have very low power. There are some pretty deep theoretical reasons for this.

Anyway, that's just yet one more reason macro data is uninformative. That problem isn't going to be solved by gathering more accurate data, or by seeking out new macroeconomic aggregates to measure (though we should probably do both of those things anyway).

So what are the implications of this basic fundamental limitation of macro? I think there are three.

1. Beware of would-be prophets from outside the mainstream. There are a number of people, usually associated with alternative or "heterodox" schools of thought, who claim that macro's relative uselessness is based on an obviously faulty theoretical framework, and that all we have to do to get better macro is to use different kinds of theories - philosophical "praxeology", or chaotic systems of nonlinear ODEs, etc. I'm not saying those theories are wrong, but you should realize that they are all just alternative theories, not alternative empirics. The weakness of macro empirics means that we're going to be just as unable to pick between these funky alternatives as we are now unable to pick between various neoclassical DSGE models.

2. Macroeconomists should try to stop overselling their results. Just matching some of the moments of aggregate time series is way too low of a bar. When models are rejected by statistical tests (and I've heard it said that they all are!), that is important. When models have low out-of-sample forecasting power, that is important. These things should be noted and reported. Plausibility is not good enough. We need to fight against the urge to pretend we understand things that we don't understand.

3. To get better macro we need better micro. The fact that we haven't fond any "laws of macroeconomics" need not deter us; as many others have noted, with good understanding of the behavior of individual agents, we can simulate hypothetical macroeconomies and try to do economic "weather forecasting". We can also discard a whole slew of macro theories and models whose assumptions don't fit the facts of microeconomics. This itself is a very difficult project, but there are a lot of smart decision theorists, game theorists, and experimentalists working on this, so I'm hopeful that we can make some real progress there. (But again, beware of people saying "All we need to do is agent-based modeling." Without microfoundations we can believe in, any aggregation mechanism will just be garbage-in, garbage-out.)

After the financial crisis, a bunch of people realized how little macroeconomists do know. I think people are now slowly realizing just how little macroeconomists can know. There is a difference.

124 comments:

One of the tragedies here, in my view, is that there isn't an absence of data overall - just a specific absence of aggregate time-series data. Even moving slightly toward disaggregation gives us plenty of additional opportunities to weigh competing theories. I view the obsession with estimating or testing macroeconomic models using aggregate data to extraordinarily counterproductive and wrongheaded. There are so many other sources of evidence, and the time series we have are just so terrible for inference.

(Fun fact: as Mark Thoma mentions in his post, there was a very important structural break in Fed behavior in the 80s. If you try to obtain impulse responses to a "monetary shock" using much more recent data, for instance, you get results that look absolutely nothing like the ones to which empirical macroeconomists are accustomed -- in fact, most of the time you get nothing, as impulse responses are quite close to zero. Sometimes I wonder how long everyone will take to realize this. Of course, it is not too surprising given theory: if a central bank is inflation targeting, and doing its job well, then inflation damn well better be unpredictable from all data, including the Fed's own behavior. But it underscores that there is no reason we should expect aggregate data to display the exogenous variation necessary to identify our models.)

In many ways, my view here does not differ so much from Ed Prescott's stated views on "calibration": I think it is silly to shoehorn all empirical macroeconomics into a single, estimable aggregate framework, and that instead we should flexibly rely on a wide range of sources for evidence. In fact, I think Prescott failed precisely because he did not take his own philosophy seriously enough. When it came time to choose parameterizations that determined the Frisch elasticity or elasticity of investment demand, he and his followers effectively worked backward from the aggregate data rather than disciplining themselves with micro evidence. (Chetty et al. in the 2012 NBER macro annual provide a masterful takedown of Prescott and Rogerson's claims about labor supply elasticities; and as far as I know, the absence of significant investment adjustment costs in RBC models has always had absurd and indefensible implications once you step away from matching a single quantity moment.)

"it is silly to shoehorn all empirical macroeconomics into a single, estimable aggregate framework, and that instead we should flexibly rely on a wide range of sources for evidence."

Well, you are slowly working you way towards the truth, at least: this is all history, and the only remotely valid explanations of these events are historical explanations, which take account of the unique character of each episode.

Another 50 years or so of macro failure (and failure of "microfoundations"), and I think this will be widely acknowledged.

"with good understanding of the behavior of individual agents we can simulate macroeconomies"

Is this really true? does a really good understanding of an individual soldier help model how a battalion will act? or even how any one soldier will act in a battalion?

You have made this point once before in a blog and I am not quite sure it works. A crowd of ten is a thing all by itself, it is not only 1+1+1...=10.

I liked Mark's post, and your additions to it.

The main point is important, its a complex and vast subject. So compared to the size of the topic little is yet known, yet, it's also true that tremendous progress has been made in the field and a lot is known.

As a physicist I dont quite understand single-agent microfoundations for macro, however. I kept thinking that macro ought to be some sort of statistical mechanics but that dosent seem to be the case. Thoughts? Any pointers on good discussions?

Ofcourse if the data for macro is poor, none of this matters, but it IS intellectually fun...

Such as... gvt spending doesn't crowd out private investments when you're facing a deflationary AD shock. Interest rates don't rise either due to bond vigilantes or whatnot. The ZLB is real and slightly negative real interest rates are not enough to get people to borrow/consume or invest...

Stuff like that.

I read Mark Thoma's article yesterday, I've read and agree with a lot of what Noah added (some of it is beyond me, stats-wise and I won't pretend the reverse) but I am still unsure of his main conclusion about needing better micro foundations. Sure, like better data or different theories, it cannot hurt.

Yet, I also believe in what Gene Callahan says above. Macro remains quite an history-related 'science'. I've tried to say a few things myself but it's pretty hard to make predictions:

and http://theredbanker.blogspot.com/2013/03/refreshing-macro-part-4-profit-matters.html

for example.

Still, here is one prediction: We won't achieve trend/above trend growth (i.e. 3.0% and above) unless and until either 2 things happen. One is some technological change of some kind radically alterating the world as we know it. The other is people regaining, one way or the other, some income growth (i.e. catching up with past and keeping up with present productivity gains or a good chunk thereof). Valid alternative to the second: All those productivity gains get reflected in lower prices. That too would work.

Failing either of those 2 things, we will remain in a slow growth mode.

Ok, if you start with the individual and work up, clearly you need a theory of aggregation similar to the theories for aggregation of individual stocks into portfolios.

So there's a systematic or group component of behavior and then the individual specific non systematic behavior plus an error term.

Of course, you'll have all the same problems, which for the most part are due to correlations not being stable.

But it would be interesting work. (and why isn't there more work on forecasting correlations? or have I just missed that?)

As far as the question about what economists know - too much to say, but I prefer economists today to those of say the year 1350. There is an old quip that its not what you don't know that hurts you, its what you know that ain't so that does. Surely we have a better idea of what we don't know at the very least, don't you think?

"As a physicist I dont quite understand single-agent microfoundations for macro, however. I kept thinking that macro ought to be some sort of statistical mechanics but that dosent seem to be the case. Thoughts? "

Statistical mechanics is too weak, as agents are not molecules.

However, there is a complete theory of computation, isomorphic to Turing machines, based on "agents" interacting by passing "messages."

So, worst case, we are left with modeling the behavior of a Turing machine, which mostly can't be done any faster than simulating it. Which is what agent-based modeling is, with micro foundations defining the behavior of the agents.

So there is theoretical reason to believe that agent-based models are a useful model, and in some sense a maximal model. But in terms of actually inferring things about the universe we live in, we don't have enough information to select between models, nor is it really possible for us to do so. I see this as related to the difficulties of the halting problem, and the sensitive dependence on initial conditions chaos thing.

In short, sometimes I feel like there is arbitrarily bad news for macro here. You'd need to simulate the world accurately, and good luck with that.

However, I think short term "weather forecasting," inquiry into robust control systems, and reasonably reproducible interventions are all still theoretically possible, and I think worth looking for seriously.

I also think economics has often lost track of the idea of asking ourselves what we want for humanity, by choosing a purely positive approach, but that's my non-rational agent talking.

The people don't really matter, the ideas are what matter. Just because people make good criticisms of bad microfoundations doesn't mean that those same people are likely to have better macro theories!

Great post Noah (or should I say Mark). I completely agree. On the third point, I strongly believe that we will not fully understand how the financial crisis led to the great recession until we find a way to model Knightian uncertainty separately from risk aversion. Recent attempts include ambiguity aversion. However, they are very new. In any case, microeconomists could be doing more work on this.

Work on ambiguity aversion isn't exactly new, it dates back to at least the late 1980's (and perhaps earlier depending on exactly what you want to include).

Admittedly, the literature on AA outside of decision theory is relatively small but there are people working on it. I can speak from experience that it is hard - once you add AA, it is often the case that almost anything can happen. Don't expect any major breakthroughs soon, but we're trying.

"On the third point, I strongly believe that we will not fully understand how the financial crisis led to the great recession until we find a way to model Knightian uncertainty separately from risk aversion."

No, that will get us nowhere at all. We might understand how the LAST crisis led to a recession, but each historical event is unique. The fundamental error is this: economists usually think we can't possibly understand individual historical episodes unless we can place them in some general category and under some general law. But the exact opposite is true: there is no way we could place historical events in general categories and draw them together under some law unless we already understood them individually!

For example, CA above say we cannot comprehend "how the financial crisis led to the great recession until we find a way to model Knightian uncertainty separately from risk aversion".

Really? Really? So the banking system had a heart attack, MM funds freezed redemption sending God knows how many real companies into tachycardia, cancelling on investment plans,firing people, consumers saw their main saving asset, their houses, take a 30% hit and lost jobs...

Do you really need a way to model Knightian uncertainty separately from risk aversion to understand how the 2007-8 financial mayhem led to real-economy traumas 6-12 months down the line? I don't think so. By all means, go for it but, again, people don't go around with VaR models in their heads. They just react.

"No, that will get us nowhere at all. We might understand how the LAST crisis led to a recession, but each historical event is unique. The fundamental error is this: economists usually think we can't possibly understand individual historical episodes unless we can place them in some general category and under some general law. But the exact opposite is true: there is no way we could place historical events in general categories and draw them together under some law unless we already understood them individually!"

Absurd. Epidemiologists/meteorologists do precisely this now reasonably well. Medical and weather analogies are the cure for Miseans broken generalization-engines.

That is fine and good but if the macroeconmists can know so little why are they allowed to have such an impact on the economy? why are so many of them employed in jobs for life at six figures? under this premise they are literally worse than having no one there because they provide a fig leaf of 'science' to some conservative cretins.

As opposed to writing postmodernist interpretations of literary work? Or coming up with false theories regarding the evolution of galaxies? (http://news.ufl.edu/2011/10/10/galaxies2011/) Or suing the driver who run over your cat? Or designing and trading financial derivatives? Or hitting a small ball with a bat? As opposed to what?

I don't know if this is the appropriate place to ask, but given these difficulties studying real macroeconomies, has there been much academic interest in studying artificial ones, like those in large multi-player video games? Of course these economies are quite different than the real one, but they are also much more observable, and it is possible to experiment on them. I think there would be a lot to learn. Is this something that is being seriously pursued?

I once worked on a problem in high energy physics called the EMC effect that has a similar data problem (insufficient data to definitively say one model is better than others). My advisor jokingly said that EMC stood for "Everyone's Model is Cool".

http://en.wikipedia.org/wiki/EMC_Effect

The data in this case is actually really good and the signature is clear! The problem seems to be that fundamental calculations (using QCD, the microfoundations of nuclear physics) are too difficult so people cobble together all kinds of effective field theories (the DSGE models of nuclear physics) but the data doesn't really distinguish between them.

I think it's the lack of meaningful, exogenous variation in all the data economists use ... economics will always suffer (thankfully) from the fact that we do not experiment in people's lives and that key institutions keep changing. The problems in macro may jump up and down more loudly, but they are present in all of economics. We can improve our research practices to accept our reality as a social science and stop aping physical sciences that we will never be. Macro questions are important...that's why they get asked and we need to keep plugging away at them. The first step in grad school research was to imagine the perfect data set (ok so this was in my labor classes) and then the second step was to figure out how to work with what data you have. Policy makers are going to act (or not) and we as economists can provide them a range of analysis (or not). The 'or not' seems like a less desirable outcome in general, though of course we should be raising the quality and diversity bar.

It is worse than just not having good enough data. Even with good data we will see it differently.

I think we a have evolved to be hard wired to see things differently across any given spectrum. It makes sense if we realize that anything people do will have the seeds of its destruction sown into it.

I am a liberal but I know that if my fondest wishes were to be made real, eventually the unintended consequences of my policies would out weigh the benefits of them. As that happens traditional Conservative criticism will become more valid. Conservative solutions will become more effective. The liberal ideas that worked so well for a time become discredited. Conservative Ideas become ascendent and then entrenched. But they come with their own unintended consequences, and then liberal Ideas start to look good again... Rinse. Repeat.

If we had better models maybe we could see this. Maybe economists would see that some approaches are appropriate for one point in our Lib/con cycle and others are better for other times. Maybe Economist would stop falling into camps promoting the "one true way".

But I doubt it.

I am just a econ novice but consider myself a Keynesian. I was a Keynesian before I ever heard of Keynesians or John Maynard Keynes. As a kid I tried to ponder econ. Of course I had no clue. (As a 9 year old I thought that Nixon's price controls were a great idea !) But the more I thought about econ, the older I got, the more I thought I was onto something. When I finally got a little education and encountered John Maynard Keynes I felt like he was saying everything I was tying to say. I felt like he had a the Grand vision of something I only had a glimpse of.

I bet a lot of people have had the same kind of affinity experience when they discovered their brand of econ...

A lot of Keynes' stuff seems axiomatic to me. From conversations on threads like theses I know that no matter what the point of view, Market Monetarists MMT'ers, Austrians... whatevers, they all think somethings are axiomatic that I think are plainly not.

I'm not sure what you mean by "weather forecasting", and I imagine this may be it, but it seems like the only reasonable way to test a model is to make as many short or medium term predictions as you can across as many economies as possible, and then just wait a few decades and aggregate the results. Obviously not perfect, but I'd love to hear a better, ethical means of empirically testing econ models.

Have any of you thought about adapting your models or aspects of them to Massive Multiplayer Online Games test them? Not a perfect substitute for real life transactions but still.... Who knows, you might even evolve a workable P2P virtual currency from them.

Noah, I think you seriously underestimate the inaccuracy of macro variables.

You've got it the other way arround. The biggest problem is not small samples, it is very crude data. Small samples gives you large standard errors. If you do not worship R2 and statistical significance, as you should not, then you can live with that.

But if you do not have accurate data for the analysis you need, then it does not matter how sophisticated the anlysis is, it is going to be wrong. With the data we have now, there are a lot of questions that could not be answered even if you had one million observations. It could even be seriously misleading, because you would have very "significant" but wrong results. You would be precisely wrong.

"it would be foolish to build a model that is inconsistent with the historical data it was built to explain"

Au contraire, it is absolutely necessary to do so. That's how you test the model. Condition your model on some of the historical data, and test it against the rest. Do so with different models and using different data for conditioning and testing. This is not a new idea.

We have a crisis of unemployment today. We know how to create jobs today. Simply hire more people to do what needs to be done. We lack the political will. Conversations that distract our governing elites from the problem at hand contribute to the lack of will.

It is incredible that some PhD economists have no understanding of the very real risks of extended unemployment creating an underclass of permanently unemployable with very large social cost and future economic drag. These real risks have been pushed out of the conversation. The current crisis has been pushed aside by prima donnas and malefactors of great wealth for Chicken Little discussion of low probability problems 5-20 years out. A failure of Macro is failure to focus on the large risk on the doorstep and instead focusing on potential risks that are miles away. Fixing the unemployment crisis today will make it easier to fix other problems down the road.

Our policy elites are not focused on the crisis at hand. They are overly concerned about low risk long term events. Their policy for the unemployment crisis: "Let them eat cake!"

There is a huge disconnect between what our academics are suggesting and what our lawmakers are doing. While you are 100% correct about the reality of the situation, the blame needs to be laid at the feet of political ideologues who want to break our system of public supports.

Perhaps the problem is that the profession rejected the path of disequilibrium macro and hysteresiseffect macro for the lovely coherence of equilibrium macro. One of my favorite pieces by Ken Rogoffis about the Dorbusch overshooting model. He described how heroic Dornbusch was to build sticky prices into his model when the profession, as led by Lucas and Sargent, denigrated the ideathat goods markets prices might clear slowly. Rogoff describes how graduate students research naturally gravitated towards what would get them published and find them jobs.

While I'm fairly certain disequilibrium macro is going to come back with a vengeance (with cool results like, if 31.2392% of your agents can learn with an algorithm from the Sargent-Gul set, you'll get stagflation! see Kautsky et. al 2024 for a partial explanation), the neuro and behavioral econ will probably take a long time to get right.

I think both this and Thoma's post on the subject are excellent and need to be repeated more often. That said, I disagree with both of you. Yes, the data is poor and its hard to distinguish between models. That is true in many fields and there are successful people in those fields who follow a rational process. That does not seem to be the case in macro.

As just one example, take Warren Buffet and the stock market. I think the stock market has many of the same data issues as macro making it hard to distinguish one model from another. So what does Buffet do? First, he doesn't use extremely complicated models (because you don't have sufficient data to distinguish between them). He uses very simple mental models. Second, most of the time he does nothing and says nothing (because there isn't enough data to do or say anything). Third, when he sees something really glaringly screamingly obvious he does something or says something. Soros could be another example. I personally would also include Keynes but there is so much controversy around his theories that it is easier to leave him out and just use Buffet as the canonical example.

In my mind, the main problem with macro is a lack of humility/perspective/taste. I believe there are a few useful simple guides and principles with reasonable data support. Instead of focusing on these, people focus on ridiculously complicated models and econometric issues.

The has a number of bad results. First, the respected members of the field are often skilled in something that is not actually useful in understanding the economy (e.g., skilled in fancy math, skilled in writing papers, skilled in coming up with clever but overly complicated models which cannot be validated, skilled in writing textbooks, etc.). This is bad because when lay people or government officials look to the field for advice, they are generally getting advice from people with the wrong skills and without the right insights.

Second, people spend a lot of time and energy researching, discussing, debating, and developing overly complicated stuff that can't be validated by the data. This not only takes away resources from the few things that do matter but it makes it much harder to find the few things that do matter because of all the chaff.

So how do we fix the problem? I have a thought but its probably not going to work. Generally when you see large complicated systems that work in a certain way that you see as suboptimal its because there are many things in the system which push it to that suboptimal point. It's tempting to try simple fixes to the system but these usually don't work because you really don't understand the complex system.

Still, if I had to pick one thing it would be objective accountability. Right now the key difference between macro and say the stock market is that the latter has some form of accountability and the former does not. Note the key word *objective* here. Peer review is valuable for some things but hasn't really seemed to work in macro (or at least there are some gaps). For example, have people, universities, schools of thought, etc., make actual predictions and measure their success rate in a public manner. That would be a way to distinguish which models work and which don't. It would also shift incentives for people to use resources on simple models which provide better predictions than doing things like making overly complicated models to show off their cleverness, writing papers to support political interests or commercial interests who provide funds.

I Had not seen this comment earlier, and thus posted something below. I read Noah's blog since I do believe that he is a really smart guy.

I was thus surprised, to see this entry: one which misses the obvious point that macroeconomics is supposed to address, big issues in the economy like unemployment, inequality and its detrimental effects, inflation, interest rates and the like.

It should not be about identifying "correct" models (because you most likely can't). This is one discipline which aids the country and its leaders (like the President and Congress, perhaps?!!) to perform the most basic duty. That of providing an economic environment which employs resources and engenders social stability. DSGE models and the rest of it, should be relegated to a position that best fits them: Frustration outlets for some academics suffering from deep envy of physical sciences and trying to show-off their "mathematical" skills. Period.

My take on the problem is a combination of no ability to have controlled experiments, and Upton Sinclair's Law ("It is difficult to get someone to believe something when his salary depends on his not believing it.")

Let's say your typical macroeconomist comes up with a model that says "If we do X, Y will happen", the country adopts X as a policy, and Y fails to materialize, you have two ways of responding:1. Rethink the model to figure out what's wrong with it.2. Claim that some other factor Z was why Y didn't happen.

The problem is that macroeconomists are modeling systems so complex that you can always find some plausible Z that explains away the failure of the test. And because macroeconomists make their living by making correct predictions, it is in their personal interest to find that Z every single time and not once say "Whoops, I was wrong, back to the drawing board."

End result: Bad macroeconomic theories that utterly fail to predict reality stick around forever, rather than getting consigned to the dustbin along with the many other failed theories of science.

I think what is not emphasized enough is that with the advent of ever better microdata fitting your model to macro time series is no longer the only way to evaluate macro models. Rather, I think the way to go in the future is to test the assumptions on agent behavior with microdata. After all, aggregate results directly follow from micro-level assumptions.

A lot of things were like this - limited data, weakly informative (if at all) - in the early days of any science. Sometimes it needed a new angle or new insight into what sort of data was useful or important; many times it needed a better method of collecting data; sometimes it simply needed a better analytical protocol. The mistake is usually in thinking that "new" automatically means "better". Lots of things look like a breakthrough until we start looking at them more closely and with larger data sets. The big problem is with us; we're a Procrustean animal - we force the disorderly world to take on a the semblance of order we think it ought to have, no matter how we have to brutalize the information. It's a rare scientist who asks the data what they have to say instead of forcing the data to reveal the hidden secrets that we know it must have. There's not much money in being nice and politely inquiring these days. The real money is in telling our patrons what they want to hear, and being quick about it.

"Macroeconomists should try to stop overselling their results. Just matching some of the moments of aggregate time series is way too low of a bar. When models are rejected by statistical tests (and I've heard it said that they all are!), that is important. When models have low out-of-sample forecasting power, that is important. These things should be noted and reported. Plausibility is not good enough. We need to fight against the urge to pretend we understand things that we don't understand."

How much of this do you suppose is due to economists trying to tailor their research for general public consumption? Not so much with their academic literature, but as far as blogs, op-ed pieces, books, and other mediums that have allowed economists to have a larger audience, but a larger audience that gets bored with the minutiae of economics/law/policy/etc. and likes absolutism (not Sithlord like, but who/what is right and who/what is wrong).

I would think this more problematic for those in the public-eye/who are monetizing their research and, more importantly, reputations. For instance, Krugman is one of the most well known economists, and I have to wonder whether the probability that many in the general public who read him might think the Phillips curve has something to do with the design of a light bulb, and if that dictates how he 'sells' his opinions, research, etc.--to some degree I believe this is true because he will on occasion label his posts as "Wonkish", forewarning his readers.

Frederic,You need a mechanism that maps out exactly how each of the events you describe affected employment and production. None of the models I have seen, including Krugman and Eggersson, are convincing to me. The phrase "MM funds freezed redemption sending God knows how many real companies into tachycardia" is not a theory. It is like saying that uncle Billy had a heart attack because the cat that jumped through the window scared him. Well, on a very supreficial level this may be true. But no medical doctor would be satisfied with it. Of course people react, but knowing how they react and to what is pretty important!

Gene,one of my favorite quotes is by Mark Twain: "History does not repeat itself, but it does rhyme."

I think the solution to the lack of test-ability and macro data systems is going to found at the creative nexus between macro-economists and virtual/game worlds.

For example, Eve Online should be on every macro-economist's radar. The most recent article is this: http://www.gamasutra.com/blogs/RaminShokrizade/20130405/189984/How_I_Used_EVE_Online_to_Predict_the_Great_Recession.php

These games are not currently designed with macro-economic theories in mind, but I can easily see how economically informed testing/modelling around "game created events" could be incredibly useful in generating models and testing those models against actual results in reasonable time frames, and given some good game design and an appeal to sandbox style players, a profitable enterprise.

Seems to me that this post expresses a rather idiosyncratic notion of the nature of macroeconomics, that it is all about business cycles. If the central core of macroeconomics is growth theory and this is just a sideshow then we have pretty good theory and pretty good data.

Why? Growth data is data is much more accurate then cycle data! Cycle data is a subset of growth data.

Now, if you are refering to data on "institutions" etc. Then I agree with you. They should forbid the zillions of papers "showing" that "institutions" is " significant", "culture" is "significant". They are just artifacts of severely misspecified models and variables with really bad measurement errors.

We should forbid papers based on data with measurement errors? Hm, then we should close down 3/4 of journals publishing empirical work. It is not only data on institutions that contain errors. One can only hope that these errors are relatively small and uncorrelated with the other explanatory variables.

"After the financial crisis, a bunch of people realized how little macroeconomists do know." No. After the financial crisis we realized who had drunk Wall Street's Kool-Aid and who hadn't. Also, using historical data need not leave one unable to distinguish between competing models. Figure out what the models major predictions are given a base set of assumptions and compare to future data whenever it appears to fit said assumptions. Repeat enough times until there is a clear winner. When Einstein released his theory of special relativity, he rejected the fact that his equations predicted black holes. and yet, they did, and black holes exist. Validation.

Actually *some* theories can be tested. Onl crapy theories cannot. So there may not be enough medical data to say if humours theory is better tha fluid theory but there is enough data to tell modern cellular theory from them both.

Same with econ: MMT predicted the Euro debacle ( I mean actual scenario, not that monetary policy eill be ineffective as per OCA), the GFC, the recession due to austerity, low borrowing costs and ineffectiveness of QE. So although you cannot tell DSGEs apart or them apart from neoclasical, you can damn well tell apart MMT from all of those.

Except right now isn't it obvious that macro (overall) is a long way from being limited by anything *except* politics? Why has Krugman been right and Chicago been wrong, for pushing four years now? Is Chicago so short on people who know economics?

What R&R tells us is that we need serious quantitative theory to analyze macro issues because most regressions are misspecified.This is exactly what many have been doing under the label "DSGE", which is by the way not understood very well by blogers.The quantitative theory approach starts from primitives and uses micro foundations to build models that have been successful in many dimensions, though there is still work to be done.

That's were you can make progress. This by the way applies to a bunch of "applied micro" work that runs misspecified regressions without any theoretical foundations. So macro is at the forefront of the discipline as a matter of fact!

I'm an outsider so please forgive me if this is unfair. I'm very troubled and puzzled about the whole "dismal science" theme regarding macro-economics. I don't see why macro-economics should be any harder than geology or astronony/cosmology. You can't do plate tectonics or a supernova in a lab any more than you can do macro-economics. Nevertheless geologist and cosmologists don't flounder about in the way that seems to bedevil macro-economics. To me macro-economics sometimes looks a bit like biomedical science would if it were confined to a bitter ideological struggle between homeopaths and acupuncturists. Basically the same approach seems to be taken as is taken by those schools of biomedicine. The data for macro-economics seems on the face of it to be massively more available than for any other science (you just download government collected economic data off the web). New events do occur at much faster pace than in geology so unfolding events provide a test for existing theories. You do have immutable laws with which to anchor theories- those immutable laws are accounting -eg if I owe you something you are owed it by me etc.I'm afraid that to me it simply looks as though the problem is that data is ignored, the laws of accounting are ignored, when unfolding events disprove theories, that is ignored etc etc.

I don't see why macro-economics should be any harder than geology or astronony/cosmology.

I do. The reason is that the microfoundations of cosmology and geology - chemistry, physics, etc. - are on pretty solid ground, while microeconomics (the foundation of macroeconomics) is not well-understood.

Geologists and astronomers have easier questions to answer. For instance, imagine you went to an astronomer and asked them: "What happens if I get close to a black hole but gravity does not change?" They will look at you like you are a mad man because that's a completely stupid question. If you get closer to a black hole, gravity changes and so there is no point in trying to figure out what would happen if it didn't.

On the other hand, if you go to an economist and ask: "What happens if output falls and the central bank does not ease policy?" the economist is expected to have an answer to that question even though most of the data that we have includes the central bank easing policy when output falls.

Macroeconomist are asked not only to describe the world as it is, but also to pretend that the world is very different (i.e. falls in output not generating a policy) and then predict what happens in that make-believe world. That is very hard if at all possible.

I thought that accounting was to macroeconomics what chemistry and physics are to geology. And yet apparently ideas such as loanable funds and the money multiplier still occupy macroeconomics and yet do not abide by the principles of accounting. Isn't that like geologists getting tangled up in theories that contradict core principles of chemistry?

I totally agree with you that thinking of deriving macroeconomics from accounting alone would be as inane as attempting a deriving geology from chemistry without looking at the earth. Equally though having macroeconomics theories that don't conform to the principles of accounting is as inane as it would be to have geology theories that don't conform to the laws of chemistry. Accounting doesn't just tabulate data - it provides a framework into which any description of monetary reality has to fit. Just as chemistry and physics provide a framework into which any description of physical (eg geology, cosmology) reality has to fit.

having macroeconomics theories that don't conform to the principles of accounting

Could you show me, in math rather than words, an example of a macro model which does not "conform to the principles of accounting"? Remember, math, not words.

Accounting doesn't just tabulate data - it provides a framework into which any description of monetary reality has to fit. Just as chemistry and physics provide a framework into which any description of physical (eg geology, cosmology) reality has to fit.

"Accounting doesn't just tabulate data - it provides a framework into which any description of monetary reality has to fit."

It's impossible for any monetary theory to violate accounting principles because accounting is entirely agnostic as to what happens in the world. For instance, if money is disappearing into thin air, I can just have a "Money that has disappeared into thin air" account and whenever that happens, I put a credit in the cash account and a debit in the "Money that has disappeared into thin air" account. And just like that, physical impossibility has neatly fit into my balance sheet without the slight difficulty.

Right. Accounting isn't to economics as chemistry is to geology (that is, a building block). Something that makes accounting sense does not have to make economic sense at all. Accounting is more like a language. It's more like French to geology. You could write a piece about geology that is perfect French but violates multiple principles of geology (and chemistry and physics). People just decided on certain accounting rules so that information could be conveyed in an understandable (and hopefully honest) manner.

If your macroeconomic theory requires a "money that has disappeared into thin air account" then you need to have that in the accounting and then it is up for observation whether operationally that is actually occurring. So for instance your theory might require a certain level of debt write downs so as to have that "money disappearing into thin air" BUT on the ground those debt write downs are not happening and yet the phenomenon you are intending to model is happening and so you know you have to go back and sort your theory out.I agree that something that makes accounting sense does not have to make economic sense at all (MMT fits that bill all too often) BUT although accounting is not sufficient it is NECESSARY. Homeopathy is biomedical science that has abandoned the constraint of conforming to chemistry. Much of macroeconomics looks to me (a lay person) like an analogous abandonment of the constraint of accounting.Do you dispute this stuff? :http://www.boeckler.de/pdf/p_imk_wp_100_2012.pdf

Stone: I don't see why macro-economics should be any harder than geology or astronony/cosmology.

Noah: I do. The reason is that the microfoundations of cosmology and geology - chemistry, physics, etc. - are on pretty solid ground, while microeconomics (the foundation of macroeconomics) is not well-understood.

This is an interesting answer, Noah, because it's very different from what you said in the original post. There you were saying that basic information is not available -- a claim I find pretty dubious, BTW, when computers can track a billion prices at a time -- and here you're saying that it's because basic theoretical machinery is not yet developed.

You may well be right in this second point. To take the analogy of understanding supernovae, if we imagine a line of scientific development close to the historical one, then there is simply no way to make sense out of the phenomenon of supernovae without a theory of nuclear physics. There was a time before people had any understanding of nuclear physics, or even the existence of nuclei; and they were not bad or stupid people by any means -- Maxwell! -- they just hadn't yet made the right investigations to lead there.

So, what is the economist's equivalent of nuclear physics in 1890? Something that you don't yet know you are missing, though there are hints from many directions, and will lead to huge breakthroughs once you map it out. Of course, I can't tell you! and you can't tell me either, though you may have interesting clues in heterodox writing like the Occupy Guide reviewed a few posts hence.

Sorry, I still the "lack of macro data" is a cop-out; and it would appear that I am in tune with Prof. Krugman on this score, whom you so much admire, see here

http://krugman.blogs.nytimes.com/2013/04/22/building-a-mystery/

Money (partial) quote, as it were: "What Samuelson does is to throw up his hands and declare that we just don’t understand what’s going on in the macroeconomy. ...But it’s no mystery..."

I'm not qualified to judge this debate per se, on what was or wasn't obvious or interpretable in terms of standard or exotic macro theory; so I'll have to leave that to you.

Instead, I will throw the Hail Mary and say that, in my not-formally-educated opinion, macroeconomics doesn't work well because it's basically fraudulent from the beginning. Y'all -- and here I certainly include Krugman -- are deep in the tank for rich people; and since the expiration of Marxism, essentially the entire enterprise of academic economics are just cogs in the machine of justifying a lopsided distribution of wealth.

When your agenda has a built-in protected conclusion, don't be surprised when you don't find the truth in the end.

essentially the entire enterprise of academic economics are just cogs in the machine of justifying a lopsided distribution of wealth.

But I think you're right from a demand perspective. People WANT economists to be this. Economists get hired to be this. The econ major has prestige and economic value because of this.

So I think you're right that this is where the money comes from to sustain the field. The true money-making business of most economists is to be the priests of the financial and consulting sectors.

But I think you're wrong when you say:

I still the "lack of macro data" is a cop-out

It's not a cop-out. If there were better data and we really could predict recessions, economists and their profession would have economic value beyond just being priests of Wall Street. And then you'd see things change.

I think a decent analogy is astrology/astronomy. Astronomers, for centuries, made all their money by predicting planetary movements and eclipses and stuff for rich aristocrats who believed in astrology. It was silly stuff. But once they stumbled on some physics rules that worked here on Earth, physics as a whole (validated by astronomy and astronomical observation) began to have infinitely more use than just for astrology.

With econ, I'm not sure we'll ever get there. Hopefully we will, maybe we won't. But my point is this:

Bad macro data is the reason why macroeconomists are forced to work as priests of Wall Street instead of as predictors of the economy.

Thanks for your very open-minded reply. Economists demonstrate supply and demand by participating in it themselves!

I still don't agree with your original post, but I may be confusing _necessary_ with _sufficient_, relative to you. When you say "If there were better data and we really could predict recessions", well, how much more data do you need? To go to the limit, if you had a record of every transaction ever made over a significant period, say a trillion numbers a day over the whole world for ten years, then would macro become qualitatively more possible? Yes, I'm sure you could constrain models much better! with all that data; but the availability of data alone will not help you think of new models to test.

What you folks really need, in my view, is a radical departure from the current orthodoxy, analogous to heliocentrism, maybe, if you want to stay in the astronomy metaphor. More data may help you prove and verify the radical departure, but the new idea has to come from some new thinking by people.

When you say "If there were better data and we really could predict recessions", well, how much more data do you need?

It's not really a question of "how much". It's a question of "what kind". Macro data is history. History only happens once. We can't test it, like we can text X-rays or chemical reactions or antibiotics. We can only watch it go by. So there's not a lot of hope for getting real scientific data. (Actually, I do have some hope that certain video games can be used for real macroeconomic experiments, but it's a long shot).

What you folks really need, in my view, is a radical departure from the current orthodoxy, analogous to heliocentrism, maybe, if you want to stay in the astronomy metaphor. More data may help you prove and verify the radical departure, but the new idea has to come from some new thinking by people.

I'd like that too...the problem is that with heliocentrism, it was easy to prove that the new idea was right. Just whip out a telescope and measure where things moved. Any new economic idea, even a correct one, will be more difficult to prove correct.

However, we might get lucky. We might discover what causes recessions to happen (which IMHO we still don't know), and therefore how to stop them. I agree that it'll take some very new thinking. I don't think people are doing enough deep original thinking in econ these days. I think most working economists have allowed themselves to be convinced that they know more than they do. What if free trade isn't always good? What if inflation usually isn't a monetary phenomenon? What if business cycles are not fluctuations around a trend, but "seizures" that we can eliminate? These ideas should be considered, but they're not being considered.

Now keep in mind that micro is different. In micro, most "heresies" are considered and tested, and some are even accepted. Behavioral econ smashed the purely rational "homo economicus" ideas, and now only third-string microeconomists believe in that old stuff. Experiments basically forced the field to change.

So when I'm talking about the failure of "economics" to think new thoughts, I'm talking about macro.

and economists looked around and saw that lots of other smart people - like historians, sociologists, geologists, paleontologists, evolutionary biologists and more - had many of the same issues. And decided to learn from them rather than digging further into a dead end. No?

"After the financial crisis, a bunch of people realized how little macroeconomists do know. I think people are now slowly realizing just how little macroeconomists can know. There is a difference."

Sorry Noah this has got to be your worst post! Basic Macro and highly analytical macro (not the cringe-inducing, notation-heavy, deluded, and ultimately weak mathematical type, but the deeply analytical kind of Keynes, Hicks) has been very successful. Paul Krugman among many others have spoken of this throughout.

There were quite a few that vied for top spot. My favourite was the one where you explained the effect of inflation for various players in the economy (homeowners/mortgagees, savers/retirees etc) and the meaning of Hyperinflation. I used that post to appropriately reset the minds of a bunch of young traders/goldbugs!

I see it differently. There are lot of factors which we can't see unless we are in their shoes. Scientists like Elbert Einstein were bullied and harassed and made fun of. People couldn't judge them. These scientists have unpredictable behavior, which is always in the favor of human kind.

Thank you for the article. I liked reading it though I don't agree much.Regards,Henry Jordan

This is a horrible description of the problem. Mark Thoma has been confused from the start, so if you're trying to base your ideas upon his, you're going down a dead end.

I caught Mark saying basically this same thing over a year go, and I tried to explain to him where he was wrong, but I don't think he understood. He's pretty much making stuff up too.

Mark doesn't understand the models. It is clear that he doesn't understand IS-LM.

But more importantly, we can't learn new macro lessons unless somebody in charge is willing to try something new. Both Obama and Bernanke have proven to be cowards, so we'll be learning nothing from them.

I question the premise that macro doesn't work very well. It works just fine from my perspective.

I think rather than trying to reconcile the real macro models with reality, many economists are trying to reconcile competing macro models. That won't work. I think Mark is trying to play middle man, the guy who tries to find peace and agreement where there isn't going to be any. I think he's more worried about his peers than the victims of the economy.

We can test the model against the data. If it matches pretty well, then we can act based upon those results and measure the results. Just like any science.

The problem with macro economics is that the systems are very complex, and anyone wanting to invalidate a model can make a claim that invalidates the model in their minds and other minds based upon assumptions. There can always be doubt. But if people don't want a solution or agree that we need a solution to unemployment, then we can't agree on a model to fix it either.

But that isn't a problem with the model, that is a problem with the people looking at the models.

The decision to create a Federal Reserve banking system required some technical knowledge. It was created upon a model. Congress decided to trust that model. It was given dual mandates assuming, based upon the models, that it could achieve those mandates.

It has clearly failed to achieve those mandates now. Does that not mean that we have to use the models we have, analyze them from problems, and try to fix what we believe went wrong?

Would it not be economic and banking malpractice to create a technical system binding everyone by law using assumptions that turn out to be invalid if we were not to revisit those assumptions and try to fix them?

We can test the model against the data. If it matches pretty well, then we can act based upon those results and measure the results. Just like any science.

OK, I'm totally on board with that. Now, you do realize that no model we've ever made has good out-of-sample forecasting ability, and all DSGE models will be rejected by statistical tests, right?

Does that not mean that we have to use the models we have, analyze them from problems, and try to fix what we believe went wrong?

Why only "the models that we [now] have"?

Would it not be economic and banking malpractice to create a technical system binding everyone by law using assumptions that turn out to be invalid if we were not to revisit those assumptions and try to fix them?

"Now, you do realize that no model we've ever made has good out-of-sample forecasting ability, and all DSGE models will be rejected by statistical tests, right?"

I disagree with both of these statements.

"Why only "the models that we [now] have?"

Because the models we have, and I'm speaking of the monetary models, are the basis of the federal reserve system. We better start with those models, because we are using them, and they have failed. So not let's fix them and take are bet shot at the next step in nature's scientific lesson.

"Now, you do realize that no model we've ever made has good out-of-sample forecasting ability, and all DSGE models will be rejected by statistical tests, right?"

I disagree with both of those statements.

"Why only "the models that we [now] have?"

I'm speaking of the monetary models upon which our federal reserve system is based. We had better start with the models the system is operating on now and fix them. Then take action so we can learn the next lesson scientific lesson of nature.

I like the post... but I think it misses two major points. (1) why is there one and only one form of macro that dominates the major journals and academic departments all over the world? such dominance is only befitting if this form of scientific endeavor has successfully provided answers to major questions in the field. yet, this is the very form of macro that has until now claimed that finance has a minor role to play (2) empirical micro has provided some definitive answers and strong validations for micro theory... however, not all of micro theory makes it into these models for reasons of tractability. why then does this Penn/Rochester/Minnesota school of tractibile Macro have such dominance over the field?

why is there one and only one form of macro that dominates the major journals and academic departments all over the world?

Because when no models of any kind work, the value of a model is decided by consensus rather than evidence?

See also: String theory in the 1990s and 2000s.

empirical micro has provided some definitive answers and strong validations for micro theory... however, not all of micro theory makes it into these models for reasons of tractability. why then does this Penn/Rochester/Minnesota school of tractibile Macro have such dominance over the field?

I agree pretty much... but how do you think this consensus was built? Did it come about by open debate in journals? Were other viewpoints given their due? I don't know the physics literature but did string theory dominate theoretical physics as the only paradigm in existence. I think you realize what I am hinting at here. The sheer dominance of the PRM school is not scary in the least because they "won the debate" but because of the means they have used and continue to use to "build their consensus." There is no debate... no opposing camp .. no journal ...that will publish any criticism of macro in its modern form. Why?

I agree pretty much... but how do you think this consensus was built? Did it come about by open debate in journals? Were other viewpoints given their due?

Generally yes, there was a lot of open debate in journals and at conferences etc. I don't know whether other viewpoints were given their due.

But you should know...that the "PRM" school has been slowly losing the academic macro battle since the mid-80s. New Keynesian models, and monetarism (which is what NK really is), are dominant now, and RBC (Minnesota-type) models have mostly been forced to try and go colonize other subfields like international, labor, and asset pricing, where their tractability makes them attractive and the researchers haven't yet heard about that paradigm's huge flaws.

The real question, in my mind, is not why RBC dominantes macro - it doesn't - but why DSGE is the only acceptable modeling framework...

The debate over the R&R conclusions, a part of the continuing methodological discussion in the post-bubble era, reminds me of some older economics literature that I enjoyed in graduate school.

The post by Noah drives it home. Creating a unified economic science would require either controlled experiments or very long stable time series. A point made bluntly by Robert Solow in 1985, and I would argue held dearly by those critical of the dominance of mainstream methods.

Economics & Economic History, by Robert M. Solow, The American Economic Review, Vol. 75, No. 2, Papers and Proceedings of the Ninety-Seventh Annual Meeting of the American Economic Association (May, 1985), pp. 328-331

I've long noted that reinventing the wheel is important to surviving as an academic. It is also very nice when you have an original thought only to find that, lo, someone else had it earlier. It is a shame for economics and the rest of society though, that we have to keep reinventing these wheels, rather than actually breaking out of the mindset that perpetuates wrong thinking.