Why S.&P.’s Ratings Are Substandard and Porous

Five years ago, if you were an investor looking for guidance on which country’s debt was the safest to invest in, Standard & Poor’s ratings wouldn’t have done much to help you navigate the headwinds of the financial crisis.

Spain, which markets now posit has about a three-in-ten chance of default or restructuring, also had a AAA rating, which it maintained until January 2009. Today it still has a AA rating, one notch higher than Japan’s.

Iceland, the tiny country with the oversized banking sector that came perilously close to national bankruptcy, was in 2006 rated AA+, the same rating the United States now has.

Greece, which now appears more likely than not to endure at least a technical default, had debt rated A, lower than most European countries but a reasonably good grade by world standards. It too was not downgraded until January 2009, and its bonds were still rated as investment-grade until March 2010.

Although Standard & Poor’s assigns ratings based on a series of letter grades, they can easily be translated into a numerical scale — sort of like the way that letter grades in high school are translated into a grade-point average:

This allows to test the reliability of the ratings in various ways, as well as to reverse-engineer them and see how the sausage is made.

What factors is S.&P. looking at when it rates sovereign debt? A country’s debt-to-G.D.P. ratio? Its inflation rate? The size of its annual deficits?

S.&P. does look at each of these factors. But it also places very heavy emphasis on subjective views about a country’s political environment. In fact, these political factors are at least as important as economic variables in determining their ratings.

For instance, the S.&P. ratings have an extremely strong relationship with a measure of political risk known as the Corruption Perceptions Index, which is published annually by Transparency International. These ratings have been the subject of much criticism because they are highly subjective, relying on a composite of surveys conducted among “experts” at international organizations who may have spent little time in most of the countries and who may instead base their judgments on cultural stereotypes.

I don’t know whether or not S.&P. looks at these ratings. But the fact that the two sets of ratings are so closely related is troublesome. It suggests that S.&P. is making a lot of judgment calls about countries they have no particular knowledge about. Keep in mind that even when it comes to the United States, S.&P. made a $2 trillion error that reflects their lack of understanding of the way that bills are scored by the Congressional Budget Office. Are we to expect that they add value based on their perceptions of the political climate in Kazakhstan, or Cyprus, or Uganda?

Other factors that S.&P. looks at, which can be determined through regression analysis, include a country’s G.D.P., its inflation rate, its recent deficits and its long-term debt. But the subjective Corruption Perceptions Index is more closely related to the S.&P. ratings than any of these economic fundamentals.

In addition, in 2006, S.&P. tended to rate European countries higher than others, even after controlling for all these other factors — something which has been especially problematic since, with some exceptions like Venezuela and Lebanon, countries in the euro zone now dominate the list of those most likely to default.

None of this would be a problem if S.&P.’s ratings had performed well. But there is little evidence that they do. The next chart presents a comparison of S.&P. ratings as of June 30, 2006, to the risk of default five years later (on June 30, 2011) as measured by the prices of credit default swaps, financial instruments that pay an investor if there is a default on a bond obligation.

S.&P.’s bond ratings from five years ago would have told you almost nothing about the risk of a default today. They had no insight about the threats in European markets, nor about which countries in Europe were relatively more likely to default. (Norway, which remains among the most solvent countries in the world, had a AAA rating in 2006, but so did Ireland and Spain.)

By comparison, simply looking at a country’s ratio of net debt to G.D.P. would have been a better predictor of default. It wouldn’t have done well by any means: it only explains about 12 percent of default risk. Still, this simple statistical indicator does better than the S.&P. ratings. (Nor is it the case that some combination of debt-to-G.D.P. ratios and S.&P. ratings does better than either one taken alone. Once you’d accounted for a country’s debt-to-G.D.P. ratio, the S.&P. ratings would not have improved your projections of default risk by a statistically significant margin.)

Certainly, one might contemplate more sophisticated models than this (for instance, accounting for a country’s inflation rate in addition to its debt seems to be helpful). But when considerably more advanced studies have been published by academic economists like Carmen M. Reinhart, they have come to similar conclusions. Ms. Reinhart found that, although S.&P. rating changes have some value in predicting defaults, they are significantly outperformed by objective, statistical indicators.

One might reasonably protest that my study (although not Ms. Reinhart’s) is comparing apples to oranges. Whereas S.&P. is attempting to forecast actual defaults, I’m instead looking at the market’s perceived risk of default as of today. None of these countries have actually defaulted yet. There’s still the chance that the markets turn out to be wrong and S.&P. turns out to be right.

Here’s the problem with that: S.&P. ratings tend to lag, rather than lead, the market. That is, in cases where the market’s view of default risk is misaligned with S.&P.’s, S.&P. is a good bet to change their rating to catch up to market perception.

As I mentioned, for instance, investors had already determined that Irish debt and Greek debt had become quite risky long before S.&P. downgraded those countries. We can also study this in a slightly more formal way. Suppose that we’re trying to predict what S.&P.’s rating for a country would be today based on two factors: S.&P.’s rating on June 30, 2009, and the market’s perception of default risk (as determined through credit default swap prices) on the same date.

If you place these variables into a regression equation, the market price of credit default swaps is a statistically significant predictor of what S.&P.’s rating would be two years later. What that implies is that the markets pick up on salient information about the countries’ default risk before S.&P. does.

In fact, the evidence from the past five years suggests that it may be worthwhile to adopt a contrarian investing strategy that specifically bets against S.&P.’s ratings. If you were trying to predict a country’s default risk today, based on the market’s perception of its default risk two years ago as well as its S.&P. rating at that time, you would find that accounting for S.&P. ratings actually subtracted value from your model. That is, if the market had priced two countries as having a 20 percent default risk in 2009, but one of them had a AA rating from S.&P. and the other had a BB rating, the country with the worse S.&P. rating is likely to have proven to be the safer bet.

The reason for this is that S.&P. ratings probably have some influence on market perceptions about default risk — even though they aren’t very good. If markets evaluate a country as having a 20 percent chance of default, but S.&P. rates it as being quite safe, that price represents a compromise between daft investors who take S.&P.’s ratings to be gospel, and savvier ones who have conducted their own analysis and have concluded that the country is at significant risk of default. By betting against S.&P.’s ratings, you’re taking the side of the smart investors — and getting a subsidy from the suckers who think S.&P.’s price is right.

But there is another “tell” to indicate that S.&P.’s ratings are slow to incorporate new information. It’s something which they seem to think is a feature of their ratings, but which instead is evidence that they are fundamentally flawed.

The giveaway is that S.&P.’s rating changes are serially correlated — that is, downgrades tend to follow downgrades, and upgrades tend to follow upgrades. According to the company’s internal analysis, once a country is downgraded it has a 52 percent chance of being downgraded again in the next two years. By contrast, there is just a 9 percent chance that S.&P. will reverse course and upgrade the country.

What this implies is that S.&P.’s ratings are inefficient about how they incorporate new information. If a country is downgraded from AAA to AA, and that implies that the country is quite likely to be downgraded again in the near future, the question is why S.&P. didn’t apply a steeper downgrade in the first place.

Consider the case, for instance, where I had a model to determine the value of shares in Google. Initially, my estimate had been that a price of $600 per share is appropriate. But then there is some shock to the system — say, some fresh evidence that the country is on the verge of another recession and that this could adversely affect Google’s profits. I estimate that $500 is now a fair price.

So I tell you that I’m willing to sell you Google shares, today, for $500. But I also tell you that tomorrow, I’m likely to lower my estimate further, so you can probably buy Google stock from me for $400 per share.

No competent brokerage firm would ever convey that kind of information to investors. If I signal to you that I’m likely to accept a cheaper price tomorrow than I am today, nobody would buy at today’s price.

But this is essentially what S.&P. does. Rather than downgrade (or upgrade) a country by several notches, even when there is abundant to support it, they instead do so in stages. Greek debt, for instance, has been downgraded seven times since January 2009, as S.&P. has slowly caught up with the grim realities that investors had long ago perceived.

I suspect the reason that S.&P. behaves this way is because they know that their ratings can have reverberations on the market and are trying to avoid a sudden downgrade that might induce panic.

But in so doing, they are violating their mission of providing the most earnest and accurate assessment of a country’s default risk at any given time. A country that is downgraded from AAA to AA is riskier, in S.&P.’s view, than one that was just upgraded from A to AA — even though they now have the same rating — since the former country is likely to be downgraded again and the latter is likely to be upgraded again. S.&P. knows this, and smart investors know this. But they won’t tell you this because dumb investors might get spooked, which could rattle the markets.

A more cynical view is that S.&P. is playing the role of the schoolmarm, looking for excuses to reward or punish countries based on good behavior — and that this is getting in the way of their objectivity. Investors think, for instance, that France is 2 or 3 times more likely to default in the next five years than the United States based on France’s exposure to Greek debt. However, France maintains its AAA rating whereas the U.S. was just downgraded to AA+. Arguably, it is not France’s “fault” for being exposed to Greek debt — whereas the United States’ fiscal problems are largely of its own making. But France is probably the riskier bet all the same.

None of this is necessarily to disagree with the downgrade in the United States’ rating. A rating system based on objective factors, like debt-to-G.D.P. ratios, might plausibly have the U.S. rated even lower than AA+.

Then again, investors still perceive the United States to be extremely safe. Based on the very low interest rates on Treasury bonds, as well as the low prices for credit default swaps on U.S. debt, investors continue to view it as among most likely countries in the world to meet its obligations.

I’m not an efficient markets hypothesis guy. I think that markets can misprice commodities, and that canny investors can profit from them. But relying on the consensus of the market is almost certainly better than relying on Standard & Poor’s, whose advice has more often than not led investors toward the losing side of bets. The fact that billions of dollars in wealth are tied up in the judgments of a company with such a poor record is all the proof you should require that the global financial system is in need of reform.

Nate Silver is the founder and editor in chief of FiveThirtyEight. @natesilver538