There are basically two types of demand-side models, both of which are nearly useless:

1. Some general equilibrium models are used to find which stabilization policy regime is optimal from a welfare perspective. Most of these models assume some sort of wage/price stickiness...The problem is that...You can get pretty much whatever policy implication you want with the right set of assumptions. Unfortunately, macroeconomists aren’t able to prove which model is best...

2. A second type of model tries to show how to best implement a specific type of policy regime, like inflation targeting...Unfortunately, these “implementation models” conflict with the EMH—it’s not clear why the central bank wouldn’t just peg the price of a futures contract linked to the goal variable...

To summarize, despite all the advances in modern macro, there is no model that anyone can point to that “proves” any particular policy target is superior to NGDPLT. There might be a superior target (indeed I suspect a nominal wage target would be superior.) But it can’t be shown with a model. All we can do is construct a model that has that superiority built in by design...

Models are toys to show our students. When we face serious real world dilemmas it’s time to put away the toys and get real[.]

Sumner essentially restates my big critique of modern macro: How do we know which model to use at any given time? There are a ton of macro models but no commonly accepted scientific standard for validating/rejecting them. So what we (the profession, the Econ Hive Mind, whatever) end up doing is choosing a model based on how plausible we think its assumptions are. We go with our intuition, gut feeling, or politics instead of data. Models are "stories", and we choose the story that sort of sounds the best.

But what should we do instead? Here is where Scott and I part ways. Scott basically says: Why constrain your story with math when you can just tell the story? And I say: Sure, but you're still just telling stories.

There are advantages to telling your stories in English instead of in Math. One is that you can be more flexible - you can add stuff to your story, or be ecumenical in your view of the world, without being worried about pesky internal contradictions. The other advantage is that you can explain your ideas to a wider class of people, who either don't understand Math or can't be bothered to read through it.

Of course, there are disadvantages as well. One is that you can't give quantitative predictions: If we deviate from NGDP Level Targeting by X%, how much will our incomes go down? And so forth. This problem can be solved by using simple "ad-hoc" or aggregate-only models, which are fairly easy for most people to grasp and can be modified without much effort. Then there's the second disadvantage of English storytelling, which is internal consistency. Formal models rely on their assumptions, but once you choose your assumptions, your conclusion is forced by the Math. With English storytelling, you can state conclusions that don't follow from your own assumptions, and often no one will be the wiser.

But I feel like this list of advantages and disadvantages ignores the elephant in the room: How do we know if the story is right? (For you George Box-quoting jerks, what I mean is: How do we know how usefully this story can be applied in a given situation?) Whether you tell your story in complex Math, simple Math, or English, it is still desirable to have a way to evaluate the story.

How do I know that NGDP Level Targeting (Scott Sumner's cause celebre) is a good policy? It hasn't been tried before. So to evaluate how good of a policy it would be, we need to know a lot about how the economy works. Scott Sumner has some ideas (informal models) about how the economy works. How well do these ideas match reality? I'm not sure. And more importantly, I don't know how I'm supposed to be sure. If Sumner used formal models, I could at least go and do a statistical test of the model (for example, a Full Information Maximum Likelihood test) against the available macro data. Sure, no one would pay attention if the test rejected the model - the data are crappy and the econ profession lacks generally agreed standards for scientific disproof - but at least I could do it. But since Scott's stories about the economy's inner workings are not quantitative and thus not subject to data, I don't even know how to evaluate them.

I mean, sure, I can think of some extreme stylized facts that, if true, would put the lie to Scott's basic story - for example, if inflation tended always to rise in recessions - but to require such a huge, glaring discrepancy in order to invalidate a model is setting the bar for your theory pretty low (in the language of econometrics, it's "protecting the null hypothesis").

In other words, the basic, fundamental problem with macro, as things stand, is that it's not scientific. Making it less formalized, or less mathematical, doesn't get around that problem. And it seems pretty likely to me that if macro ever does come up with a way to tell if models are right, it's going to require that those models be of the formal mathematical type.

Update: Though, come to think of it, biology models are not mostly math. If you give a description of how the liver works, the description won't rely on equations, and in fact will be more useful than one that does rely on equations. So maybe macroeconomic science won't rely on math; maybe it'll rely on qualitative descriptions of the function of key components of the economy. I'm really not sure. But in any case, we need some standard of model evaluation...

Worse, I think that formal mathematical models are the thing that is discrediting the economics profession. And that's precisely for the reason that a lot of economists think models are good: their logic is impersonal, cold and rigid. That's fine in a system where we know all or most of the variables. But in economics we don't, and that's why fuzzy, personal subjective "models" (i.e. heuristics) do better than mathematical models in a lot of circumstances (I'd like to say the majority of the time, but I don't have the stats to back that up), but there are plenty of examples where formal neoclassical models have failed and heuristics have worked, e.g. 2008).

I beat up on John Maynard Keynes a lot, but from re-reading the General Theory he understood this point, in a way that Hicks and Hicks' descendants like Krugman don't.

here's a deeper question: Which fed policy performs better when we don't know what the model is? If we don't know what the model is, how do we even know there is an optimal policy, and it is unique? Maybe we should pick a rule that performs best across a range of likely assumptions.

There are a ton of macro models but no commonly accepted scientific standard for validating/rejecting them.Until you have such a standard you don't have a science. I'm reminded of Feynman's criticism of rat learning experiments in Surely You're Joking, Mr. Feynman. If you don't have a basis, even in theory, for rejecting a model you can't even begin.

So what we (the profession, the Econ Hive Mind, whatever) end up doing is choosing a model based on how plausible we think its assumptions are

He didn't. He believed that we should just select the model that best matches the stylized facts we're interested in explaining. But there are at least two big problems with that approach to model validation/rejection. The first problem is that if your model isn't structural, you're doing "uncomfortable science" (Google that term if you don't know it, it's a good one). The second problem is that if you have a different model for each phenomenon, your models will contradict each other, and you have to have some way to select between them.

I'm not sure uncomfortable science is the right term. That assumes that is the nature of the data which forces the conundrum. I see more of a fitting of facts to conclusions. For example, the Michigan-Wisconsin study on gasoline tax and the New Jersey study on minimum wage seem to be ignored because those facts contradicted the theories.

"the basic, fundamental problem with macro is that it's not scientific. Making it less formalized, or less mathematical, doesn't get around that problem."

But it might reduce the discipline's tendency to massively over-claim.

No, scratch that. For that to happen, economists would have to adopt a scientific mindset.

"if macro ever does come up with a way to tell if models are right"

Testing models is not the problem. We have that sorted. Macro is missing scientific models, models that show *how* the interactions of thousands to billions of constrained heterogeneous agents lead to the results we see.

2. In light of 1 feel free to call me Mike-evilsax is just my stage name.

3. This has been something of a Eureka! moment for me. I think this is illustrating why models are important. While Sumner's move-models can prove anything so what's the difference-seems to offer us some freedom this freedom is a fake.

4. All this has inspired me to write a post. I'll give a link to it later-after I've actually written it! I know you're busy but obviously I'd be honored if you can find the time to check it out.

You might be right about your comparison with biology. Veblen certainly thought so.

I'm not sure that making economics less formal would make it less scientific. At least in my field, development economics, subjective details matter a lot for conveying an accurate picture of the country or place concerned (in the sort of way that Aziz says above). These subjective details can only be expressed in language. Any 'scientific' view must include both the so-called hard, objective 'view form nowhere' and the local, subjective knowledge of the insider. It's no good just performing a load of mathematical tricks over in Washington or Geneva and preaching the answers to the inhabitants of less well-off countries. You've got to go and get your hands dirty and -- shock, horror -- actually ask people what they think.

See in this way, 'verifiability' becomes a bit irrelevant. I'm not even sure that there's a single unified economic reality in which answers can be deduced from a set of universal principles. This isn't to suggest that economic science isn't possible, just that models aren't the only way of getting at the truth and that methods associated with 'objectivity' should be combined with more 'subjective' approaches.

I like your line of thinking. Making things mathematical does not make them scientific, it only makes them internally coherent. But that internal coherence can be as useful in reality as Borges short stories: great entertainment for those with the predilection, but fiction.

I discussed something similar about a month ago in my post "Has Paul Krugman Become a Member of the Literati". At that time Stephen Williamson accused Krugman of basically abandoning mathematics-based economics. I generally disagreed. Here's the link:

My own thought is that any model put forward in words has, implicitly, a mathematical model behind it, which is not to say that the model is internally consistent. Where possible, it seems that it's always best to go back and forth between the two. My own economic logic is more verbal, but I have tried lately to make sure they check out with at least a basic mathematical model, whether microfounded or not.

The one issue with the biology comparison is that at least in biology we knew the basic building blocks. This is an organ, this is a gene, this is an individual. There are certain characteristic scales on which biological events occur. In economics it's much more dismal; we don't even know what these characteristic scales are, or even if they exist.

IIRC, these definitions get more untidy every year. Genes may be transcribed in different ways, they interact with each other quite strongly. Developmental biology is changing the picture of the simple gene to protein model.

Organs again are a bit fuzzier, since they interact with other things (i.e., the effect of A/a depends on B/b).

And in many 'species', there's massive gene flow among individuals, and individuals might have most of their survival ability depend on the others around them.

First of all, I do not understand the question "English or math". These are not mutually exclusive. In fact, good papers use both. In the introduction the authors tell the story in plain English for those who are not interested or capable of going through the math. They summarize this story again in the concluding section, and also list the possible limitations resulting from their simplifying assumptions. The technical part is read mostly by the referees who want to ensure internal consistency (that there are no logical mistakes or leaps), and those who want to borrow the model for their own research.

Second, you seem to be siding with the Popper's view of falsification that has been criticized by Thomas Kuhn as well as natural scientists. In the end a theory is judged not against the data, but against other theories and the alternative of no theory. Even at the time of Newton there were discrepancies between the predictions of his theory and the observed behavior of planets. Yet the theory was replaced only when a better theory, that of relativity, came along. Moreover, Newtonian physics are still used today in civil and mechanical engineering, relativity theory is used to design space missions, and quantum physics are used to study radiation. How about that for a multitude of models?

No, it makes plenty of sense without it. The key phrase is "in the end". You compare the predictions of a theory with data for the purpose of testing a theory against an alternative. As Kuhn pointed out, the scientific community hardly ever gives up a theory that fairs poorly against the data UNLESS they have a better alternative.

Re biology and liver analogy - is it aposite? The liver is a single organ in a complex system. But describing it is not math free. In order to know if it is functioning properly, we measure things. So there are two parts - a functional decomposition - what is it composed of, what goes into and what comes out - but also a mathematical description of how we measure the flows into and out of it and how we measure and describe its current status.

But the liver is a small part of a single organisms within an ecological system. Macro is looking at the ecological system. And that is very much described in mathematics. A pure textual description tells us very little about it. What is important is things like, births, deaths, predation, environmental influences - i.e. the dynamic interactions, and the relative volumes of these things are crucial to understanding. It's only economics historical hang up with equilibrium that stops it concentrating on such things instead. Macro-economists should be concerned with such things as innovation, firm start ups, bankrupcies etc - not just micro-economists. That they aren't is where I think it has lost the plot.

Also demographics - births, deaths, household formation, immigration, education etc. The volumes of these things are macro-economic variables. If you don't close your eyes to the dramatic demogrpahic changes in the 1970s, the change from the 1960s to the 1970s is not so extraordinary.

I think the real question is why everyone in economics seems to have forgotten about the power of a good analog simulation such as Phillips and his hydromechanical simulation of the British economy, dating from the 50's. There's a good description of the machine and its role in economics and education here...http://systemdynamics.org/conferences/2009/proceed/papers/P1038.pdf

Reiss has a very good paper on using simulations to nudge economics towards being a truly experimental science at: http://www.jreiss.org/papers/S%26G_42(2)_2011.pdf As he points out, simulations have a long history of circumventing the analytical intractability of equations describing non-linear phenomena.

Cambridge has a restored Phillips machine, and I would love to see it used as a means of having an economic debate where one's positions are demonstrated on the machine. In his video presentation http://www.sms.cam.ac.uk/media/1094078 McRobie at Cambridge, who restored their Phillips machine, speculates that it was the monetarists who caused the use of the simulator to be abandoned.

Which brings us to an important point: a sufficiently sophisticated analog model could be used to compare economic theories on a far finer scale than the Phillips machine. This approach has indeed been takein in the aerospace industry, where, for example, a generic twin engined airliner model has been extensively tested in the wind tunnel and is used as a basis for comparing the accuracy of Computational Flow Dynamics (CFD) models. However, one should be warned that fluid dynamics, both in the Phillips machine, and in its more modern iterations, has a decidedly Keynesian bent.

I've been playing around with a flight analog of the economy, which I write about in terms of risk management in complex systems in 'Hedging the Apocalypse' at: http://somewhatlogically.com/?p=598 (Modeling done in conjunction with Dominican University 'Green MBA' program.)

Wait.. After all this people still believe in EMH? That's sort of similar to Intelligent Design. It might be a nicely packaged theory that would be useful to believe for various reasons, but it doesn't make the theory true - or useful.

Replicatable profits in the financial sector are derived from the creation of inefficiencies, especially through information gaps. Morgan Stanley's most recent successful efforts in persuading everyone that Facebook was worth $38 a share is a classic example of EMH in action. "Everyone" knew that Facebook was worth at least that much. Now everyone knows that Facebook isn't worth that much. But the people that were informed that Facebook wasn't worth that much have made a lot of money shorting the stock in the interim. In other words, EMH is a finance marketing tool used to sell financial products to people that Goldman Sachs terms "muppets".

If economics doesn't develop an idea of disproving a theory, then it can't qualify as even a semi-scientific discipline. What honest economics should be doing is developing models and predictions based on those models, along the lines of "If policy A, then outcome B", and then evaluating those models based on the degree to which A in fact leads to B.

The trouble is, that much of the profession practices not honest economics but dishonest economics. In this scenario, instead of the goal being a dispassionate search for an accurate model of economics, the goal is to convince people that policy A is a good idea (whether or not it actually is). This leads to patently bogus arguments (Paul Krugman's "zombie lies") being brought up repeatedly no matter how often they're discredited. And as long as there are lavish rewards for policy A, some economists will be paid very good money to practice dishonest economics.

Dave, as I mentioned, and Kuhn has demonstrated, no science has a way of disproving a theory. They have ways of testing particular aspects of a theory, but failure of the data to comply generates further research into why this is so. It does not lead to discarding the entire theory. The fact that such unresolved issues are often kept secret by those who like the policy implications of a theory is truly troublesome. But so is the fact that others use these discrepancies to justify an "anything goes" attitude and discredit policies that are based on predictions of the theory that do comply with the data. For example, one prediction of the EMH is that managed funds should not consistently under-perform index funds. Several studies have shown that this is the case. Fund managers do not like this, after all their income depends on their clients belief that there advice does actually have value, so they often use the failure of the EMH in other aspects to also attack this particular prediction. But are they correct to do so?

My point is that if you have a model that is consistently making predictions that are very wrong, then an honest researcher is perfectly within his rights to question the model. Imagine 2 models describing a relationship between A and B: Model 1 says A=B*B, Model 2 says A=5*B*B*B. If your measurements say that A=2*B, then model 2 should be discarded, model 1 might be used until something better comes along, but neither one is really right. What would be wrong is to try to make the measurements look like model 1 is right, and even worse would be claiming that model 2 was in any way correct.

As far as the EMH is concerned, I can't buy strong-form EMH even conceptually, for the simple reason of information asymmetry. On a micro scale, of course, I have every reason to lie in order to sell a $5 item at $30. But my buyer, now seeing what I've sold him, also has every reason to convince another sucker to buy the same $5 for at least $30, leaving the market price at $30 even though the underlying product is still worth $5. There's no reason to think that with a large enough market and nobody looking to carefully, the price would remain $30 and a mythical $25 worth of value comes out of thin air.

Noah, I agree very much with your post. I do have one addition however about your exact biology analogy in the update. In the late 1980's I believe, the econometricians Soren Johansen and Katarina Juselius spent some time at John Hopkins to test competing models of the liver. They ultimately rejected the competing "bath tub" model for the "tube" model, which went on to be a Nobel prize-winning idea (aside: in the former you represent it as a tub which dilutes contents, in the latter as a tube which "eats" some of the contents passing through it to varying degrees depending on its functioning). In fact, biology does rely on a means of rigorously testing competing models (bio-stats), and the abstract models are also mathematical in nature.

I do however agree that useful scientific predictions in economics are in general qualitative (Friedman's essay has been the basis for the use of models with sharp predictions but his example was simply when demand goes up price goes up). In our mathematical models the parameters are generally symbols which we only constrain qualitatively (or on occasion assuming long-run homogeneity or something). We don't say in our math that beta equals 2.55, we estimate these parameters assuming our model is correct (and generally reject them if they return insignificant results or those with the wrong sign). We then pretend these parameters are time-invariant, often without testing for parameter constancy, and typically find that they, in reality, break down over time (results are sample dependent). Pretending any one model provides exact prediction (up to a random error of known distribution) forever into the future probably accounts for part of the Wall street blow-up (and relates directly to Hayek's Nobel speech about the pretense of exact knowledge, which he referred to as charlatanism).

Yes. There's a distinction between a theory being quantitative and the tests of the theory being quantitative. Tests will always have to be quantitative, even if the theories are qualitative "tub/tube" type of things.

I think in most cases, pure qualitative predictions are useless. They must come with some quantitative sense.

Suposse I say: if you print money, you will have a boom, and then a bust.

That is useless unless I have some Idea of how large the boom will be, how large the bust will be, when they will happen and how much money I have to print for that to happen.

If the FED prints $10 I'm pretty sure that the prediction mentioned above will not hold. But if the FED prints $1000000000 trilion then I'm pretty sure shit will happen without any boom.

So, the "qualitative" prediction you refer to may actually be quantitave in the sense that it referes at least to order of magnitudes, but they are just not very precise. Otherwise, I think qualitative predictions are not useful at all.

This Cosma Shalizi research proposal would seem to show one way forward,

" 5000+ words, and many equations, about a proposal to improve, not macroeconomic models, but how such models are tested against data. Given the actual level of debate about macroeconomic policy, isn't it Utopian to worry about whether the most advanced models are not being checked against data in the best conceivable way?"

The first thing one must decide is to what use the model will be put on.

(i) if it is for prediction in policy matters, a very compelling way to judge the model is to just set up a loss function, and evaluate its performance with a reasonable separation of the calibration and validation data.

The step above is likely to doom almost any model for most practical purposes. One could dispute the loss function. But, so what, that's the point. The model is not real and if it is going to be used for prediction you have to decide how big the discrepancy must be untill you decide that it is useful or not.

And this is not a validation of the "truth". It is relative, and open to the fallacy of affirming the consequent.

But it is very reasonable in most cases.

Now, I don't know what you really meant sugesting the statiscal test for the model. But, unless you also define how big the discrepancy must be to reject it, you will just conflate statistical significance with economic significance. And that is certainly a trivial (but unfortunately common) mistake.

(ii) if it is a "Hal Varian" model, to improve understanding of a mechanism, then it is intuition, comom sense and persuansion that will do the trick. The rules here are just: don´t commit logical fallacies.

And so on.

I don't know if you're familiar with the literature of modelling in earth sciences, but there's a paper in Science vol 263, 1994 "Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences" that is very useful for economists.

Noah, Interesting post, which I'll have to think about. Here's an initial reaction:

1. There are two issues here. Why did some famous macroeconomists apparently find my National Affairs article to be persuasive? And should it have been persuasive? (I.e., was it persuasive for the right reasons?)

2. I think at this late date most macroeconomists have basically made up their minds about the "do nominal shocks have real effects" debate, which has been going on for 100s of years. We've gone over the data exhaustively. Most have concluded nominal shocks do have real effect. I'm speaking to that group.

3. I also think that most famous macroeconomists have a "I know it when I see it" attitude toward demand shocks. Thus most look at cases like Britain today, or America in 2011, and think it fairly obviously that a bit more AD would be helpful. Indeed this consensus is so powerful that conservative opponents of fiscal stimulus in 2009 usually didn't argue that it will "work," and hence would be bad, but rather that it will fail to boost AD due to crowding out. That's very revealing. Tacit admission that more AD would be nice.4. What I've done is pointed out to the profession "Look at all those times where your gut instinct tells you more AD is needed. Do you notice that they don't correlate very well with inflation (which has been above 2% in Britain during the recession, and was above 2% in the US during 2011) and they do correlate very well with NGDP relative to trend?"

5. Of course that may be a bad argument, because the gut instincts of macroeconomists may be wrong. But I think that's why my proposal has been somewhat persuasive.

PS. Thanks for the Box link. I've been saying "all models are false, but some are useful" in my blog, thinking that I made it up.

What I've done is pointed out to the profession "Look at all those times where your gut instinct tells you more AD is needed. Do you notice that they...correlate very well with NGDP relative to trend?" Of course that may be a bad argument, because the gut instincts of macroeconomists may be wrong. But I think that's why my proposal has been somewhat persuasive.

Agree.

Now this may be going a bit far afield, but what about the Lucas Critique? If NGDP hasn't been targeted, how do we know that the gut-instinct relationship between NGDP and AD will hold if we try to use NGDP as the lever to control the economy?

Thanks for the Box link. I've been saying "all models are false, but some are useful" in my blog, thinking that I made it up.

Argh! My pet peeve. It's just a really bad quote. It basically deprives the word "right" of all useful meaning. I will write a post about this soon... ;)

Now this may be going a bit far afield, but what about the Lucas Critique? If NGDP hasn't been targeted, how do we know that the gut-instinct relationship between NGDP and AD will hold if we try to use NGDP as the lever to control the economy?

I've asked this many times, but I am seemingly not worthy enough of an answer (but I am worthy enough for contempt and scorn apparently).

Hopefully he'll answer it now that you have asked it.

In fact, I would suggest a response to Goodhart's Law (on which the Lucas Critique is derived) applying to NGDP targeting. I find it a clearer, more succinct "version" of the same intuition.

I think at this late date most macroeconomists have basically made up their minds about the "do nominal shocks have real effects" debate, which has been going on for 100s of years. We've gone over the data exhaustively. Most have concluded nominal shocks do have real effect. I'm speaking to that group.

If one were to conclude that money does affect the real economy, and no economist should deny this, it doesn't imply that central bankers can know what the "correct" supply of money or the "correct" volume of spending should be. The fact that money and spending is controlled by the state, isn't sufficient evidence that they can be expected to know such things through being "advised" by like-minded central planners.

You are ignoring, as usual, how central bank inflation, commercial bank credit expansion, and the resulting change in interest rates themselves affect the real economy, in such a way as to make spending and interest rates lose their information content, thus leading investors astray (see Goodhart's Law).

I also think that most famous macroeconomists have a "I know it when I see it" attitude toward demand shocks. Thus most look at cases like Britain today, or America in 2011, and think it fairly obviously that a bit more AD would be helpful.

These same "famous" macro-economists had no clue the housing market was about to collapse, and those same economists call for more inflation the same way children call for more food and water. It's never enough. Why? Because inflation is what is preventing the real economy from correcting, so it's macro-economists chasing their own tails. They call for more inflation, that prevents corrections of capital misallocation, which hampers economic recovery, which is responded to with "more inflation."