A paper used to justify austerity economics appears to contain an Excel error.

An economics paper claiming that high levels of national debt led to low or negative economic growth could turn out to be deeply flawed as a result of, among other things, an incorrect formula in an Excel spreadsheet. Microsoft's PowerPoint has been considered evil thanks to the proliferation of poorly presented data and dull slides that are created with it. Might Excel also deserve such hyperbolic censure?

The paper, Growth in a Time of Debt, was written by economists Carmen Reinhart and Kenneth Rogoff and published in 2010. Since publication, it has been cited abundantly by the world's press politicians, including one-time vice president nominee Paul Ryan (R-WI). The link it draws between high levels of debt and negative average economic growth has been used by right-leaning politicians to justify austerity budgets: slashing government expenditure and reducing budget deficits in a bid to curtail the growth of debt.

This link was always controversial, with many economists proposing that the correlation between high debt and low growth was just as likely to have a causal link in the other direction to that proposed by Reinhart and Rogoff: it's not that high debt causes low growth, but rather that low growth leads to high debt.

However, the underlying numbers and the existence of the correlation was broadly accepted, due in part to Reinhart and Rogoff's paper not including the source data they used to draw their inferences.

A new paper, however, suggests that the data itself is in error. Thomas Herndon, Michael Ash, and Robert Pollin of the University of Massachusetts, Amherst, tried to reproduce the Reinhart and Rogoff result with their own data, but they couldn't. So they asked for the original spreadsheets that Reinhart and Rogoff used to better understand what they were doing. Their results, published as "Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff," suggest that the pro-austerity paper was flawed. A comprehensive assessment of the new paper can be found at the Rortybomb economics blog.

It turns out that the Reinhart and Rogoff spreadsheet contained a simple coding error. The spreadsheet was supposed to calculate average values across twenty countries in rows 30 to 49, but in fact it only calculated values in 15 countries in rows 30 to 44. Instead of the correct formula AVERAGE(L30:L49), the incorrect AVERAGE(L30:L44) was used.

There was also a pair of important, but arguably more subjective, errors in the way the data was processed. Reinhart and Rogoff excluded data for some countries in the years immediately after World War II. There might be a reason for this; there might not. The original paper doesn't justify the exclusion.

The original paper also used an unusual scheme for weighting data. The UK's 19-year stretch of high debt and moderate growth (during the period between 1946 and 1964, the debt-to-GDP ratio was above 90 percent, and growth averaged 2.4 percent) is conflated into a single data point and treated as equivalent to New Zealand's single year of debt above 90 percent, during which it experienced growth of -7.6. Some kind of weighting system might be justified, with Herndon, Ash, and Pollin speculating that there is a serial correlation between years.

Recalculating the data to remove these three issues turns out to provide much weaker evidence for austerity. Although growth is higher in countries with a debt ratio of less than 30 percent (averaging 4.2 percent), there's no point at which it falls off a cliff and inevitably turns negative. For countries with a debt of between 30 and 60 percent, average growth was 3.1 percent, between 60 and 90 it was 3.2 percent, and above 90 percent it was 2.2 percent. Lower than the low debt growth, but far from the -0.1 percent growth the original paper claimed.

As such, the argument that high levels of debt should be avoided and the justification for austerity budgets substantially evaporates. Whether politicians actually used this paper to shape their beliefs or merely used its findings to give cover for their own pre-existing beliefs is hard to judge.

Excel, of course, isn't the only thing to blame here. But it played a role. Excel is used extensively in fields such as economics and finance, because it's an extremely useful tool that can be deceptively simple to use, making it apparently perfect for ad hoc calculations. However, spreadsheet formulae are notoriously fiddly to work with and debug, and Excel has long-standing deficiencies when it comes to certain kinds of statistical analysis.

It's unlikely that this is the only occasion on which improper use of Excel has produced a bad result with far-reaching consequences. Bruno Iksil, better known as the "London Whale," racked up billions of dollars of losses for bank JPMorgan. The post mortem of his trades revealed extensive use of Excel, including manual copying and pasting between workbooks and a number of formula errors that resulted in underestimation of risk.

The paper's problems also highlight the value of peer review. Although the paper was included in the American Economic Review, it was included as a "Papers and Proceedings" article, taken from a conference and published without being reviewed. A peer reviewer might have sought the Excel spreadsheet at review time and could have exposed these flaws sooner. In turn, this may have prevented an apparently flawed paper from receiving the widespread attention that it got. The response is similarly unreviewed, so it too may be flawed.

290 Reader Comments

taxation which centralizes power by taking from the many to be spent by the few.

Its a mistake to assume that taxation leads to a centralization of power. Countries like North Korea have very, very low taxes, while some of the most free have very high taxes. The correlation is actually not what you would expect because high taxes tend to only be tolerated by relatively just societies.

The tinpot dictators of the world get a bullet to the head if they try to implement the Nordic tax model.

The problem, or perhaps risk, with Excel isn't that it will calculate a given formula incorrectly. It's that the UI makes it entirely too easy to input the wrong formula when dealing with ranges of cells. Because Excel tries to convert mouse-based selections into input for a formula, an errant mouse click can very trivially result in additional unwanted inputs into your sum or average or regression or whatever. Or, as in this case, missing a few data points because someone didn't drag out enough cells to capture the complete set. Easy enough to fix if you notice of course, but that's only if you notice.

The problem with this analysis is that it assumes that growth is the only important metric. What about things like centralization of power and inflation? Higher public spending does both and they are both strong negatives.

Where is this purported inflation that you claim we are suffering in the US (we have high public spending, after all)? How are those gold investments working out for you?

Not that I'm saying go buy gold, but this is akin to a gold bug having said, "How is that stock market working for you?" on march 9, 2009 when the dow and S&P hit 12 year lows. If someone is investing long term, short term collapses are usually not a cause to panic but an opportunity to invest more.

The level of the gold market fluctuates, but the way gold has been marketed has not. The failure of the economy for 5 years and running to exhibit the inflation used rampantly as a fear-based marketing tool to sell gold WILL eventually cause a "reversion to reality," and we are seeing the start of that now.

Stop putting the blame on Excel, and stop claiming to use 'real' tools such as DBs, Matlab and similar stuff... having used Mathematica and Matlab for years, I had to write an application in Excel with Visual Basic some time ago. It just rocks.... the GUI of a spreadsheet, tightly combined with a simple programming language and many high-level functions is a very nice addition to my programming toolbox. Don't blame the tool if people don't use it correctly. Excel is actually a very nice thing.

On another point, I doubt that many reviewers might have found technical errors such as this. It's not that reviewers invest days and days to redo and check every assumption and every step the authors took - no reviewer has the time to do this (famous exceptions as the one with the Fermat's theorem don't count).

The blame should be placed on the fact that any spreadsheet software exists.

If someone is not smart enough to write "real" software in a proper programming language, then they probably also aren't smart enough to be trusted with the "formula" feature in excel.

If those 15 rows had been inserted in a proper relational database, and a proper query such as "select average() where age > 90" used to calculate the result, then you would not have errors like this.

Human errors can happen in any tool, but experienced software developers know how to structure their code in such a way as to make them extremely unlikely. And they know not to assume the code is working and will put it through extensive testing.

What, they are using a RDBMS? They aren't writing *real* software?!

I've seen plenty of bad queries and bugs in software code; mathematical errors in complex calculations are particularly prone to making it through testing. "Experienced" developers make mistakes, too. Blaming the tool here is foolish -- this is EXACTLY the kind of number crunching Excel was written for. The fact someone had a bug in their spreadsheet is the fault of the tool, any more than a bad SQL query is the fault of the RDBMS.

taxation which centralizes power by taking from the many to be spent by the few.

Its a mistake to assume that taxation leads to a centralization of power. Countries like North Korea have very, very low taxes, while some of the most free have very high taxes. The correlation is actually not what you would expect because high taxes tend to only be tolerated by relatively just societies.

The tinpot dictators of the world get a bullet to the head if they try to implement the Nordic tax model.

I'd argue that inflation is not low at all. It is under-reported and doesn't even factor in what would have been natural deflation.

There is no such thing as "natural deflation". Money is artificial. Its value fluctuates with supply and demand.

And no, prices have actually been quite stable for the past decade on most indicators, save perhaps commodities.

North Korea has total state control of all methods of production which is basically really, really high taxes and government spending.

No deflation is 100% natural. As technologies and productivity improve, prices fall. Look at flat screen televisions. The innovation is just so huge there that prices fall despite inflation. To say inflation is natural is to say that we are moving backward in our productivity.

You always see inflation most dramatically in commodities because they have the thinnest profit margins.

Their response has nothing to do about their excel "error". That they did not accept the data from 6 countries, since accepting the data in their spreadsheet would disprove the answer they were looking for...

The response they give back is one where they've crossed their arms and ask for anyone to look up 250 years of the world's economic data to disprove them since they claim they have used such a large broad range of economic figures themselves in cooking their results.

taxation which centralizes power by taking from the many to be spent by the few.

Its a mistake to assume that taxation leads to a centralization of power. Countries like North Korea have very, very low taxes, while some of the most free have very high taxes. The correlation is actually not what you would expect because high taxes tend to only be tolerated by relatively just societies.

The tinpot dictators of the world get a bullet to the head if they try to implement the Nordic tax model.

I'd argue that inflation is not low at all. It is under-reported and doesn't even factor in what would have been natural deflation.

There is no such thing as "natural deflation". Money is artificial. Its value fluctuates with supply and demand.

And no, prices have actually been quite stable for the past decade on most indicators, save perhaps commodities.

In free markets deflation is natural. I will explain.

Natural deflation happens in the market as people become more productive and goods and services become less expensive. For example, Andrew Carnegie drove the price of steel down aggressively because he was producing it more efficiently than ever before. The drop in steel prices reverberated in the market and led to many capital goods being more abundant and less expensive, which led to consumer goods being less expensive and more abundant as well. Look at electronics, do you not see the prices of the same technology dropping regularly? The only thing to note here is that this is real deflation opposed to nominal deflation, I'm not speaking to monetary deflation as in the contraction of the money supply.

Ya know, geeks shouldn't knock spreadsheets. People forget or maybe never knew that visicalc was THE killer app the drove the industry, while the geeks were busy controlling model trains with s100 homebrews and other nonsense.

Whenever I see stories like this I remember a team of nuclear engineers which I used to support. They did modeling in excel and insisted that each cell of their massive spreadsheet was a "polygon" so they obviously needed faster video cards to run their spreadsheets faster.

Circular saws: The ruiner of hands and fingers? I'd say so (having done it myself!). Certainly you can blame the user, but sometimes a tool is also to blame as well.

Excel as a tool is dangerous. It doesn't have easy to use assertions, commenting or the ability to have multiline formulas. There is no easy way to review a spreadsheet for coding errors (ever had a code review for one?).

It's a powerful tool that looks simple to use, but has no safety at all in the way people typically use it. The title could have been written - "Improper spreadsheet use:..." but if there is no well established/understood easy way to use it properly, I think some of the blame should be put on the tool.

The problem with this analysis is that it assumes that growth is the only important metric. What about things like centralization of power and inflation? Higher public spending does both and they are both strong negatives.

Both of these assertions are not necessarily true. They are also not necessarily negative.

First of all, higher public spending does not necessarily lead to centralization of power. What if city governments increased public spending -- does that qualify as "centralization of power"? I'm not so sure given that there are tens of thousands of cities in the US.

Secondly, higher public spending does NOT always lead to increased inflation. In fact, in the economic situation we are currently in at the zero lower bound (where interest rates want to go lower but can't because then they would be negative), increased public spending does not increase inflation. Just look at the fact that inflation has remained very low since the financial crisis even during periods of increased government spending. It's also hard to argue in the current economic situation that increased government spending would be a negative. The majority of economists state that increased government spending would be a big positive to the economy as it would put to use resources that currently lie fallow (such as unemployed people).

If you are using inflation to reduce wages in order for people to get jobs that probably means that there is a non-market barrier in the way of them working now. That is working now for the same real wage which they will be working for later once inflation has reduced their real wage to a point where they are employable(marginal productivity>marginal cost to employer)

So if you are unemployable at $9 an hour but at 8$ an hour you are employable then you can either accept a wage of $8, or inflate the currency so that $9 is as valuable as $8 was, then you are working for the same real wage as you can purchase the same amount of goods and services with your new inflated wage as you could with your pre-inflated lower wage.

First spreadsheets can be abused by incompetent users and this appears to be an incompetent user problem. Not looking at the entire spreadsheet but one simple solution is use named ranges so the formula is written as =avg(name) each time that range of data is needed. Many spreadsheet users do not use this so "the one off" error is very easy to do.

Second, economics is not as precise as some economists like to claim. Many years ago (about 30) , a very prominent economist wrote a book that showed economics had several problems that make economic predictions dubious or even outright wrong. A major problem that bedevils economics is the confusing correlation with causation. The problem is that people confuse economic modelling and its mathematics and graphs with scientific rigor. Physics, chemistry, and biology rely on deductions based on repeated observations and theories that can be falsified and tested to see if the theories are correct. Economics has a problem with testing because if they are wrong it will ruin innocent people lives as the economy implodes and there is no laboratory setting that can be used.

Economics has been nicknamed the "dismal science" for a very good reason.

I suggest everyone read this before jumping to any conclusions. In fact, it would probably be worthwhile letting this cook some time before serving up any economic conclusions one way or another. Claims of bias appear completely unjustified.

Nevertheless, this is a good article for Ars not because of any specific conclusions we can make about macroeconomic policies, but how power tools not only provide for potential greater productivity but also quicker and more devastating mistakes, as anyone who has lost an appendage on a table saw should be able to testify.

Having worked in the regulated medical device industry, here's clear evidence why spreadsheets critical to production of said devices need to be tested and validated.

Circular saws: The ruiner of hands and fingers? I'd say so (having done it myself!). Certainly you can blame the user, but sometimes a tool is also to blame as well.

Excel as a tool is dangerous. It doesn't have easy to use assertions, commenting or the ability to have multiline formulas. There is no easy way to review a spreadsheet for coding errors (ever had a code review for one?).

It's a powerful tool that looks simple to use, but has no safety at all in the way people typically use it. The title could have been written - "Improper spreadsheet use:..." but if there is no well established/understood easy way to use it properly, I think some of the blame should be put on the tool.

The popularity of spreadsheets is because they require minimal programming training to give results. Some trained in programming (almost any language) is aware of the issues raised about documentation, debugging, and coding practices. My experience is that most complex spreadsheets are virtually impossible to follow, worse than spaghetti code.

What? Inflation is the best for borrowers which poor people are. Deflation on the other hand is a killer to borrowers.

Deflation simply raises the cost of borrowing, if you have 5% interest and 2% annual deflation then your real interest rates is around 7%. Meanwhile, if you have 5% interest and 5% inflation you are essentially borrowing money for free. This is why if interest rates were forced higher by the market or if the fed chose raise them the banks would be in deep trouble as they would essentially have a negative rate of return on many of their loans.

Wasn't the London Whale debacle also caused by a poorly managed spreadsheet?

Whoops, prior to submitting it for editing I had overwritten an entire paragraph that raised this very issue. I've reinstated it.

But, yes.

This is the real problem with Excel. Obviously, Excel isn't to blame as such; it's the poor workman who blames their tools, etc. But Excel does make it very easy for projects to grow, shall we say, "organically". Something that starts simple can rapidly become non-simple, and it's very difficult, in Excel, to debug (especially spreadsheet formulae), apply standard version control practices, unit test, and so on. People cobble things together using macro recorder, copy-and-paste from samples they find on the Web, and all sorts of other techniques.

I totally understand why they want to do it, and in that light, Excel is enormously empowering, which is very much a good thing (I would say the same is true of Access, too). But sometimes, a little more structure and rigour would be a welcome improvement. The problem, I think, is how to produce a tool that's as approachable and versatile as Excel, but also more robust.

Excel is a serious problem because it hides your work. In order to debug an Excel spreadsheet, you basically have to poke at every single cell and make sure it is what you think it is.

Excel also does nice automatic things for you, including extending formulas when you add rows or columns, etc. Which is nice if it's what you want, but can also lead to problems like what happened in this article. Perhaps the four countries that were excluded were accidentally added by clicking the row below and inserting a row above, instead of the row above and inserting a row below -- makes a big difference as to whether a formula is copied or not.

The problem with this analysis is that it assumes that growth is the only important metric. What about things like centralization of power and inflation? Higher public spending does both and they are both strong negatives.

Both of these assertions are not necessarily true. They are also not necessarily negative.

First of all, higher public spending does not necessarily lead to centralization of power. What if city governments increased public spending -- does that qualify as "centralization of power"? I'm not so sure given that there are tens of thousands of cities in the US.

Secondly, higher public spending does NOT always lead to increased inflation. In fact, in the economic situation we are currently in at the zero lower bound (where interest rates want to go lower but can't because then they would be negative), increased public spending does not increase inflation. Just look at the fact that inflation has remained very low since the financial crisis even during periods of increased government spending. It's also hard to argue in the current economic situation that increased government spending would be a negative. The majority of economists state that increased government spending would be a big positive to the economy as it would put to use resources that currently lie fallow (such as unemployed people).

If you are using inflation to reduce wages in order for people to get jobs that probably means that there is a non-market barrier in the way of them working now. That is working now for the same real wage which they will be working for later once inflation has reduced their real wage to a point where they are employable(marginal productivity>marginal cost to employer)

So if you are unemployable at $9 an hour but at 8$ an hour you are employable then you can either accept a wage of $8, or inflate the currency so that $9 is as valuable as $8 was, then you are working for the same real wage as you can purchase the same amount of goods and services with your new inflated wage as you could with your pre-inflated lower wage.

No deflation is 100% natural. As technologies and productivity improve, prices fall. Look at flat screen televisions. The innovation is just so huge there that prices fall despite inflation. To say inflation is natural is to say that we are moving backward in our productivity.

You always see inflation most dramatically in commodities because they have the thinnest profit margins.

You're conflating "fluctuations in prices" with "fluctuations in the price level." When commodity prices swing, it can take a long time for the effects of those changes to be felt in consumer prices, if they ever are, because they have to filter through the supply chain, where they can be dampened somewhat. Consumers don't care about the price of silicon, they care about the price of smartphones, and the day-to-day changes in the former don't much affect the latter.

"Flat screen TVs are cheaper" is not deflation. Saying that inflation is natural is the same thing as saying that societies generally get richer over time. Stop reading ZeroHedge.

Excel is a serious problem because it hides your work. In order to debug an Excel spreadsheet, you basically have to poke at every single cell and make sure it is what you think it is.

Excel also does nice automatic things for you, including extending formulas when you add rows or columns, etc. Which is nice if it's what you want, but can also lead to problems like what happened in this article. Perhaps the four countries that were excluded were accidentally added by clicking the row below and inserting a row above, instead of the row above and inserting a row below -- makes a big difference as to whether a formula is copied or not.

Press F5, hit Special and make sure Formulas are all checked.

Highlight those cells (change their color).

Go through them one by one and check the formulas; highlight cells checked in a different color.

The problem with this analysis is that it assumes that growth is the only important metric. What about things like centralization of power and inflation? Higher public spending does both and they are both strong negatives.

Both of these assertions are not necessarily true. They are also not necessarily negative.

First of all, higher public spending does not necessarily lead to centralization of power. What if city governments increased public spending -- does that qualify as "centralization of power"? I'm not so sure given that there are tens of thousands of cities in the US.

Secondly, higher public spending does NOT always lead to increased inflation. In fact, in the economic situation we are currently in at the zero lower bound (where interest rates want to go lower but can't because then they would be negative), increased public spending does not increase inflation. Just look at the fact that inflation has remained very low since the financial crisis even during periods of increased government spending. It's also hard to argue in the current economic situation that increased government spending would be a negative. The majority of economists state that increased government spending would be a big positive to the economy as it would put to use resources that currently lie fallow (such as unemployed people).

If you are using inflation to reduce wages in order for people to get jobs that probably means that there is a non-market barrier in the way of them working now. That is working now for the same real wage which they will be working for later once inflation has reduced their real wage to a point where they are employable(marginal productivity>marginal cost to employer)

So if you are unemployable at $9 an hour but at 8$ an hour you are employable then you can either accept a wage of $8, or inflate the currency so that $9 is as valuable as $8 was, then you are working for the same real wage as you can purchase the same amount of goods and services with your new inflated wage as you could with your pre-inflated lower wage.

Maybe you should give an actual reply instead of posting a link to a article on a related subject. But for those who want a quick summary of his article rebuttal here it is- when there is deflation, wages tend to resist the deflation in nominal terms. (example with just monetary deflation), just say there was 5% deflation in a year, your inputs cost 5% less and you sell your outputs for 5% less, your margins may still be squeezed in real terms by your labor inputs because wage earners tend to resist nominal wage reductions even if the purchasing power of their wages remains the same in real terms.(i.e. monetary deflation gives wage earners a real pay raise because their nominal pay doesn't decrease as much as the real value of the wages they earn. The reason this is not very significant in my previous statements is because in my examples above I was focusing on explaining the kind of deflation we see when systems of labor and capital become more efficient and thus prices fall not because of monetary deflation but out of productivity increases.

To simplify things:Deflation from productivity increases = goodDeflation from contracting the money supply = could be good or bad

I think even the term "austerity" is a negative buzzword created by people with an agenda.

Having a balanced budget is smart.

If the economy is down and interest rates are low, it is generally better for the government to run a deficit: it raises the level of production, and the bill comes in a few years, when the economy is growing again and the relative cost will be lower. (Sometimes, the actual cost is lower, if inflation outpaces the interest rate, and sometimes the growth is enough to cover the cost without actually raising tax rates.)

On the other hand, if the economy is growing quickly, government spending should be lower to avoid crowding out private investment, and extra tax revenue should go toward paying down debt.