Analysis: what is meant by 13,000 ‘excess’ NHS deaths?

When the dust settles on the Keogh report published last week one figure is likely to linger: the “13,000 excess deaths” in the 14 NHS hospitals. It deserves careful scrutiny – and some has been applied by Isabel Hardman here with more details about this curious notion of “Hospital Standardised Mortality Rates” in the Health Service Journal here. But these still leave the question unanswered as to why these “extra” people are dying, and what, if anything, we can and should do about it. Here’s my attempt. It’s fairly detailed, and it’s still a lovely day so those who don’t have an appetite for such things may not want to click on the link. But those who do want to get their heads around this may find it interesting. The figure of 13,000 excess deaths was important enough to put on front pages of newspapers and quoted on the news bulletins, so it’s worth looking a little more at what it actually means.

Simplifying somewhat, we start with data which includes information on patients’ death or survival, and correlates that, using so-called regression analysis, with some observed characteristics – mostly specific health conditions and demographic characteristics. So we can say, for example, that a 55 year old male lung cancer patient has on average an X% chance of dying. Express that on a hospital level, and you have “expected” deaths. “Excess” deaths (or “excess” survival) are just the difference between actual deaths and the number “predicted” by the regression model for each individual hospital. A hospital with no excess deaths is one that performs exactly as the model predicts.

But – and this is the key point – these differences are simply the variation between hospitals that the regression model doesn’t predict. By definition, we don’t know what explains them; if we did, we’d have put it in the model in the first place as one of our explanatory variables. The ‘excess’ figure – again by definition – comes from the things we’ve left out of the model – the so-called “omitted variables”, as well as from pure random variation.

Now one of those omitted variables is almost certainly hospital performance, or quality, unless you think such things don’t matter at all . But it is almost certain there are others as well (in addition, the model is almost certainly “misspecified” as well, which introduces additional complications and biases). But the bottom line is that hospital performance will only explain some of the “excess” deaths calculated from the model – and the model itself won’t tell us how much.

A few important and policy-relevant points follow from all this:

1. Differences between actual and “predicted” deaths are a useful diagnostic. They tell you that something is going on in that hospital that’s not in the model. That certainly justifies sending inspectors in to hospitals where those differences are large; but it definitely doesn’t tell you that the difference between predicted and actual deaths in any given hospital is down to performance; or that in aggregate what proportion of the differences are down to performance.

2. There is no sense in which the average (the regression line) given by the model is the right outcome. Suppose all hospitals had actual death rates that were at, or very close, to the “predicted” ones. Would that mean everything was fine? It might mean performance was uniformly good (whatever that means!). But it might equally mean performance was uniformly awful.

3.There’s no reason why we should take differences from the “expected” rate as the relevant metric. It would be just as legitimate to take the best 25% and look at differences between that and individual hospitals. Lots of businesses do take this approach – everyone should aim to get their performance to that of the top quartile.

4. What about the hospitals which have actual death rates below the “predicted” ones? Are they saving “extra” lives, and how? As with the extra deaths, the answer is possibly, although we don’t know how many. But certainly it would be just as justifiable to send in inspectors to them to find out what they’re doing right. Again, lots of businesses would do exactly that.

It would be far more satisfactory if we had an actual, quantifiable measure of quality or performance. We’d then know how much of the variation was being driven by quality/performance, and how much by chance or other omitted variables. In other words, we’d be explaining differences in death rates, rather than identifying differences that we can’t explain. That seems a lot more useful. Of course, we don’t have perfect measures. But you can think of some things that it would be interesting to look at – e.g. nurse-patient ratios, years of experience for doctors, management/clinical staff ratios, etc. I apologise in advance who to any health economist/statistician who knows the relevant literature and may be able to cite examples of such research.

Policy matters too. In particular, Carol Propper and her co-authors have looked at the impact of choice and competition within the NHS, and generally finds positive outcomes; that is, increased competition, under the right conditions, can reduce mortality (interestingly, the general finding is that competition over “quality of care”, rather than price competition, is what matters). Arguably, if you’re trying to make general health service policy, rather than to find hospitals where individual management failures may or may not exist, this is sort of research that’s really needed.

PS I’m not a health economist or statistician, so this is primarily about the number crunching – or regression methodology – in question. The same method is used a lot in various statistics you see in the newspapers: for example, the assessment of school performance using “value-added” measures.

Jonathan Portes is director of the National Institute of Economic and Social Research and former chief economist at the Cabinet Office

Kindle

More Spectator for less. Subscribe and receive 12 issues delivered for just £12, with full web and app access. Join us now.

Some useful analysis. Back in the day this is what the state broadcaster used to do

2trueblue

The BBC became “Blairs Broadcasting Corp’, then Browns and now it still belongs to Liebore. We simply pay the bills, and just like running the most corrupt parliament Liebore contaminated the BBC so we really have no control and certainly no unbiased, good journalistic output. We should not pay for it. It is lauded as the great …… but no longer. Shame they did not get rid of Patten and lots of others at the top when the scandal broke, and really sort it all out. They say fish stinks from the head.

JonBW

Actually, a good way of answering these questions would be to investigate the experience of patients and their relatives; it would, of course, generate the kind of subjective ‘anecdotal’ evidence disdained by economists and statisticians, but it might also provide a much ‘truer’ and more holistic picture of the state of the NHS.

And I suspect it would make pretty grim reading: my own experience, based on work in the NHS and personal discussions is that in the hospitals that are ‘failing’, standards of care are at best inconsistent and at worst downright bad.

telemachus

Sir Mike Richards has a plan to cleanse it
Let him get on with it

Colonel Mustard

But first we must cleanse the NHS of Burnham.

Hello

If you’re looking to measure the quality of care at a hospital then surely it’s better to game the system as a comparison of a hospital against it’s own past performance? Then the rate of improvement can be compared at a national level, or even an international level if the data exists. Would that not also be a better metric on which to base appointments to the executive of a hospital, in the absence of a price mechanism?

Rating hospitals on an absolute measure is useful for ranking, but not very useful for perpetuating positive competition. There’s very little reason to improve the operations of the number one hospital if none of the other hospitals are threatening it’s position.

OldLb

So Birmingham. 500% the death rate of UCH for the same risk of patient.

And you want to say, Birmingham is great if it gets to 400% of the death rate.

On top too, UCH will have lots of avoidable deaths.

Hello

Oh, how do you think it should be dealt with? Tell them they’re doing a rubbish job for 5 years as the situation improves, and then leave them to decline again once they reach your specified target? Or perhaps you think things will turn around overnight, and all of a sudden they’ll wonderfully – if only we disapprove of them enough? Or maybe you think the country will decisively decide to abolish the NHS altogether?

My, my – what wonderful systems of measurement your have, my dear boy. Perhaps in the future we could measure hospitals on the amount of rainbows they manage to produce?

What exactly do you think should be done to prevent failures in the future, or do you think it would be acceptable if all NHS hospitals were performing equally badly?

You’re just another statist idiot who thinks targets should be set by government statistical authorities as a superior method of accountability.

salieri

I read this piece, then I read it again in the hope that I had missed something significant and it wasn’t just stating and re-stating the bleedin’ obvious. Then I read it a third time, and gave up looking. Perhaps we could now have an article asking what is meant by 257 ‘excess’ Labour MPs…?

telemachus

There is actually no such thing as excess Labour MPs

Colonel Mustard

More anti-democratic, single party state aspiration.

Colonel Mustard

Evidence that Burnham is more interested in covering up his role in the NHS scandal than getting to the truth:-

Revanchists know well that Jarman is a radical and are fond of quoting him without understanding of statistics
Remember the mess Keogh got into in Leeds by mis-applying statistics and worrying the good folk of Yorkshire needlessly before backtrack
Much of the excess mortality of the eleven hospitals is happening now and most since 2010

Colonel Mustard

Well you would say that. All your comments are pure Labour party dogma. You could save yourself a great deal of time by simply writing in each comment:-

“Labour good, Tories bad”.

It would be more honest spamming too.

HookesLaw

The forthcoming report in the iraq war should be interesting, not least about what it says about MI6 and the dodgy dossier.

OldLb

Avoidable deaths – 40,000 – peer reviewed,. BMJ.

Excess deaths reported is just for a small number of hospitals. They are reporting the differences between the average and the excess.

However, even at good hospitals there will be plenty of avoidable deaths. So the excess over the average is not the same as the real figure of avoidable deaths.

It’s a lottery, and it kills

HookesLaw

No its not – its a study by someone else who gave a figure of between 800 and 40,000. And then qualified it.
I repeat…
“A report reproduced in the BMJ by ‘Helen Hogan, Frances Healey,
Graham Neale, Richard Thomson, Charles Vincent, Nick Black (to which I
presume you refer)
says
‘Introduction Monitoring hospital mortality rates is widely recommended. However, the number of preventable deaths remains uncertain with estimates in England ranging from 840 to 40 000 per year, these being derived from studies that identified adverse events but not whether events contributed to death orshortened life expectancy of those affected.’
I suggest to you that figures with such a wide discrepancy
are meaningless. The figures come from a study of record reviews of 1000
adults who died in 2009 in 10 acute hospitals in England.
So your figure could easily be less than 1000. Your misrepresentation
and cherry picking does you no credit. As ever on here we see figures
bandied about with no merit to them except that they feed a prejudice.”http://www.ncbi.nlm.nih.gov/pubmed/22927487

i’ve already pointed this out to you but you still peddle your lies

Indeed the conclusion of the study says …
‘The incidence of preventable hospital deaths is much lower than previous
estimates. The burden of harm from preventable problems in care is
still substantial. A focus on deaths may not be the most efficient
approach to identify opportunities for improvement given the low
proportion of deaths due to problems with healthcare.’

People get ill and die.
We simply want people in the NHS to do a basic professional job. its not a lost to ask. Some people will die. We should be able to study what happens to improve what happens.
Labour’s policy was a disgrace but hysterical lies do not help anyone.

OldLb

40K avoidable deaths. Peer reviewed. Good enough for climate science, good enough for the public.

HookesLaw

Climate science is bogus – so you do not help yourself there.
Thanks to me the report is now there for all to see and draw their own conclusions – no one now needs your misinformation.