A Few Doubts about the Stress Tests

[This is the talk on the Bank of England’s stress tests given by Kevin Dowd to the Adam Smith Institute, March 17 2016. The second edition of Kevin’s No Stress: The Flaws in the Bank of England’s Stress Testing Programme will be published soon by the Adam Smith Institute.]

[This draft: June 5 2016.]

Good afternoon everyone.

I’d like to begin by thanking you all for coming, and to thank my friends at ASI for the invitation to speak.

My topic today is the Bank of England’s stress tests. These remind me of Tweedledee’s definition of logic:

If it was so, it might be; and if it were so, it would be; but as it isn't, it AIN’T!

You might describe the stress tests as a perfect example of policymaking by fairy tale.

Not so much the appliance of science, but the appliance of make-believe to policymaking.

We base policy on a single scenario that we admit will almost certainly never occur!

Lewis Carroll would have had a field day with that one!

Its almost as bad as believing six impossible things before breakfast.

1. Why the stress tests matter

So why do the stress tests matter?

They matter because we are concerned about the financial health of the banking system.

The Bank’s narrative is that the stress tests show the resilience of the banking system

That the banking system has successfully recapitalized post-crisis.

Governor Carney put this nicely at the December 1st press conference:

the [UK banking system] has built capital steadily since the crisis … It is a resilient system, you can see it through the stress tests. … The long walk to higher capital is over.

My narrative is almost the exact opposite, but based on the Bank’s own data:

The stress tests do not reveal banks’ strength, because that strength isn’t there. For example, tests based on the ratio of CET1 capital to Risk Weighted Assets are useless because RWA is a pretend number, and the banks’ capital rebuild is based on these pretend numbers.

To assess bank strength we should focus on the leverage ratio instead. Traditionally, the leverage ratio is the ratio of core capital to total assets – not Risk-Weighted Assets.

In fact, banks’ leverage ratios are still low – by my best estimate, less than 3.7% on average.

This is VERY low.

The long walk to higher capital has barely begun.

2. The fatal errors in the stress tests

Turning to the tests, these are not unreliable. They are subject to a string of fatal errors.

Let me just run through a few of these.

The Bank of England’s stress tests miss banks’ weakness because they use extremely low pass standards.

They rely on a single adverse macro scenario, but the chances of that occurring are very low.

Think about it – no single scenario can possibly give you confidence that you are safe in the face of all the other scenarios you didn’t consider: it’s just not possible.

There are also other problems:

Stress testing creates new systemic risks that are invisible to everyone’s risk management systems: it creates icebergs that our ships’ radar systems cannot see, and you might even say that this is a design feature of the system.

There is a kind of Goodhart’s Law at work by which risk models break down when used for risk control purposes.

Central bank stress tests are also subject a number of credibility issues.

One purpose is to determine how strong the banking system is.

Another purpose is to promote confidence in the banking system.

But these objectives will clash if the banking system is weak.

It’s OK to promote confidence, but whether that confidence is justified or not is another matter.

So what happens if the central bank conducts a stress test and discovers that the banking system is really weak?

Can you imagine it admitting that?

Listen up everyone: better run and get your money out!

All that taxpayer money we threw at the banking system didn’t work!

No, I didn’t think so either.

Instead, the stress tests always give the same answer everywhere – the unrelenting message is always the same - the banking system is just fine … even when it later turns out it wasn’t.

Stress testing also has a positively dire track record.

This track record shows that the underlying methodology is totally unreliable.

Real rocket science put a man on the moon, but stress test ‘rocket science’ has produced a string of disasters.

Stress tests in the US told us that Fannie Mae and Freddie Mac had a failure probability of zero; they were unsinkable, like the Titanic.

They then sank spectacularly a few years later.

Regulatory stress tests told us that the Icelandic banking system was safe, then the Irish banking system, then the Cypriot one, and then the Greek one.

All these national banking systems failed shortly after the stress tests had certified them as safe.

Relying on these exercises is a form of Russian roulette: most of the time you will be OK, once in a while you won’t.

With one near exception, I am not aware of a single case where a stress test correctly identified a risk build-up to which the authorities responded in the right way, by taking suitable counter measures in advance to defuse the problem.

This near exception was of course Northern Rock.

In this case, the UK tripartite authorities produced a scenario that was eerily close to what subsequently transpired.

They then did nothing about it, and were caught completely off guard when their own scenario came to pass.

When news of that fiasco came out, there was an entertaining game of pass the parcel in which each of the parties concerned tried to blame each other.

The British authorities had managed to snatch defeat from the jaws of victory!

Gillian Tett got it right when she described the tests as a predictable piece of public theatre.

Their purpose is to promote confidence – but whether that confidence is warranted is another matter.

The stress tests remind me of those earlier pieces of predictable public theatre, the periodic suspensions of the Bank Charter Act of 1844: these worked as if by magic to calm the occasional panics of the period.

Some of you might recall a delightful speech to the House of Commons by Benjamin Disraeli on August 30th 1848:Just at the moment when unutterable gloom overspreads the population, when nothing but despair and consternation prevail, the Chancellor - I beg pardon, the Archbishop of Tarento - announces the liquefaction of the blood of St Januarius, … a wholesome state of the currency … return[s]: the people resume their gaiety and cheerfulness; … everybody returns to music and macaroni … and in both cases the remedy is equally efficacious, and equally a hoax.

However, the difference between the liquefaction miracle and the stress tests is that the liquefaction miracle is real.

3. The results of the 2015 stress tests

Lets start with the headline tests.

Chart 1: Headline Results Based on the CET1 Ratio

This chart shows the test results based on the CET1 ratio using the Bank’s 4.5% pass standard.

At first sight, everything looks well.

However, the minimum pass standard should be at least as high as the regulatory minimum requirement, and this is 7%, not 4.5%.

Even the Bank’s own guidance document suggests that the pass standard in the stress test should be at least the minimum capital requirement!

As far as I can tell, every expert outside the Bank agrees on this point.

But still, the Bank insists that the pass standard should be 4.5% instead of 7%.

To quote another of Lewis Carroll’s characters, when I use a word, it means just what I choose it to mean, neither more nor less.

I would suggest this is scrambled logic.

And his character was Humpty Dumpty – we all know what happened to him!

This is what we get with a 7% pass standard:

Chart 2: CET1 Ratio Stress Tests with a 7% Pass Standard

A very different picture!

Better still is to set the pass standard at the maximum possible regulatory standard under fully phased-in Basel III, taking account of the maximum possible countercyclical buffer of 2.5%.

The BIS has been warning us for some time about elevated risks: these suggest that a higher countercyclical buffer is called for.

Those concerned about elevated risks include the Chancellor.

Anyone who thinks its mission accomplished with the UK economy is making a grave mistake, he said in January.

He then warned about a dangerous cocktail of new risks, and I agree.

In any case, the pass standards need to be prudent to be convincing, and the more prudent, the more convincing.

Instead, the Bank gives the impression that it chose the lowest possible pass standard it thought it could get away with.

This higher pass standard gives us the results in this next chart:

Chart 3: CET1 Ratio Stress Tests with Maximum Basel III Pass Standard

We now get quite a different picture! 4 banks fail, and only one clearly passes.

However, these exercises are problematic because they depend on the RWA measure, and I think we can safely say that this measure is discredited – and not least by some brilliant work by the Bank’s chief economist, Andy Haldane.