Posts Tagged With: Performance measurement

I think we can all be guilty of letting the little things slip because the Titanic momentum of the NHS in action can make the problem seem insurmountable. But, like they say, how do you eat an elephant? One bite at a time.

This is going to get a bit ranty at points, & I’m finding it really hard to write for some reason, so I will try and remain coherent…

It’s a year since the Francis Inquiry published. If 2013 was the year of the big 3 reports into NHS failings (Francis/Keogh/Berwick), the details in the Public Inquiry into failings at Mid-Staffs (Francis) was probably the most harrowing as far as personal stories were concerned. Unsurprisingly, this has generated a lot of interest at the one year anniversary: how do we move on, take the lessons from Francis (and the others), and make the NHS better?

In the previous post I was worried about our tendency to measure structure and process indicators and assume that they relate to outcomes. How well we know that what we do affects outcomes could fill a post of its own (maybe later). But let’s look at outcomes. What should we measure?

Let’s start with life and death – mortality rates – I don’t think any of us would argue that whether they are dead or alive is a matter of fairly major concern to the majority of patients. The cardiothoracic surgeons were the first ones to get going on this in the wake of Bristol. However, there were concerns about this, particularly in elective surgery. If you compare below Mr Scrub 1 (left), who isn’t in all honesty a very good surgeon but only operates on relatively healthy patients with Mr Scrub 2 (right), who is an extremely skilled surgeon with an excellent team but is willing to operate on much sicker patients, their crude mortality rates may be exactly the same. That can’t be fair can it?

Might surgeons shy away from operating on frailer patients (or cherry pick the healthiest ones) in order to improve their figures? This is the point of casemix adjustment – crude mortality is adjusted for demographics like age and comorbidities in an attempt to level the playing field. And so the Hospital Standardised Mortality Ratio and the Dr Foster controversy was born. Unfortunately, like everything, the HSMR wasn’t perfect. It only looked at inpatient mortality so a Trust that discharged patients who then dropped dead at home wasn’t picked up, and only examined deaths within certain diagnostic groups. It also became possible to improve a trust’s performance on HSMR by making the patients “sicker”, ie with more comorbidities. So potentially a trust’s position in the mortality league tables owed more to the quality of its coders than its clinicians.

This, according to Francis, was part of the problem at mid-Staffs, where the reaction of the trust board to apparently poor mortality figures was to commission research into flaws in the stats, rather than to look at the quality of care. This is not to say that the research they commissioned didn’t raise some very valid points; it’s still worth a read.

Obviously this required a response so the Department of Health commissioned the excellent people at Sheffield Uni (I have to say that, I’m doing my doctorate with them!) to develop an alternative to HSMR, which they did – SHMI. Sadly all methods of casemix adjustment have the same limitations – we want to control for anything, apart from the quality of care, that might affect mortality. The first problem is working out what all these things are, never mind recording them reliably……. The second is assuming that the same thing confers the same risk in all population (the constant risk fallacy). Let’s assume that a diagnosis of asthma is a risk factor for respiratory mortality – does this mean that Mr Smythe in Affluentshire who has an inhaler because occasionally he gets a bit wheezy on the golf course when it’s cold is at the same increased risk as Mr Brown in Deprivedville who in addition to his reactive airways has a 30 pack year history, 5 courses of steroids a year and poor health literacy? Of course not!

One last thought – even if we can sort out all these issues and work out what an institution’s mortality rate “should” be, can we assume that deviation from this reflects quality of care (or lack of it)? Some seriously scary headlines have trumpeted tens of thousands of deaths that “could have been prevented“. Let’s think about that – in terms of avoidable mortality, patients occupy a Goldilocks zone; most will not die irrespective of suboptimal care (assuming the absence of particular acts of commission like inadvertent intravenous potassium administration); many have simply reached the end of their life and care should be aimed at comfort and dignity, not “life-saving”.

So how many patients are in this Goldilocks zone? The NCEPOD report “Time to Intervene” found that over 1/3 hospital cardiac arrest calls (codes for my American friends) were potentially preventable; however they felt 74 of these 156 calls were “preventable” because the patient should already have had a DNAR order. So a preventable crash call, but not preventable mortality.

So that’s why it’s not just as simple as life and death – and that’s before we even consider all the other morbidity-related outcomes that patients care about…….

KC

Update 6th Feb 2014

Just out in Annals of Emergency Medicine, a Canadian Delphi-type study aiming to identify diagnoses where mortality and morbidity may be sensitive to the quality and timeliness of emergency care:

I like the list they came up with (although I’m not sure how much difference we always make to ACS or diverticulitis).

Some time ago I got a call on a Friday evening from my footballing husband. Apparently a ball he considered to be a header was thought by someone else to be at kicking height and they had a head-boot interaction, resulting in a good Harry Potter imitation.

He said, “It’s bleeding a lot and the boys have called an ambulance…. they’ll make me go to X hospital”. So I got in the car and took him to Y ED, where it was kindly sutured by one of the consultants.

But later I wondered about where that left us regarding X ED. Why was he so keen to avoid it? What was wrong with it? How can we expect the public and policymakers to know which EDs they might and might not want to attend?

Well, this is what the “Friends and family test” is supposed to address. Since April 2013, patients discharged from the ED have (when we remember) been given a card on which they can comment on their care. But does a good F&F report guarantee a good ED? To quote the MPS

“Understandably patients experience difficulties in assessing the technical competency of a doctor, so will frequently judge the quality of clinical competence by their interactions with a particular doctor” (page 8).

In other words, patients may use the quality of your communication as a proxy for the quality of your care – often, but not always, true. Might this not be the same of EDs? Departments may have extensive local support and loyalty despite (for example) lack of supporting services or failure to be accredited for specialist training.

So what else might we look at? For many years the only metric applied to UK EDs was the “4 hour target”. The problem with this was the gaming it led to – for a summary of the distorting effect of targets even a 10-year-old can understand, see Inspector Guilfoyle. – Tom Locker and Sue Mason demonstrated nicely the peak in admissions just before 4 hours, with evidence that it was due to frank manipulation of performance data in some cases, and it persisted. Anecdotes exist of majors cubicles with doors being designated “Decision Units” so the clock could stop ticking, and we have to ask whether a patient is better treated spending 5 hours in the ED then being admitted to a bed on the appropriate ward or 3.5 hours in the ED and a further 8 hours in the waiting room on MAU (no stopwatch there).

Which of these graphs does your department resemble? And which would you want it to resemble? [Hint – the clue’s in the colours]

This was recognised and in 2010 there was a move, supported by CEM and the RCN, to develop a wider set of quality indicators. Suggested parameters included adherence to relevant NICE guidance, mortality rates for specific conditions, time to specific interventions (eg door to balloon for AMI), proportion of patients with ambulatory care sensitive conditions admitted to hospital, senior review of high risk patients and staff experience.

Some of these are now being published on the Health and Social Care Information Centre website: left without being seen rates, unplanned reattendance within 7 days, time to initial assessment of ambulance cases, time to treatment and total time.

So what should we measure?

Back in 1966 Avedis Donabedian addressed this problem in relation to the whole of healthcare; he identified three facets of care that could be quantified: structure, process and outcome.

In terms of the ED, structure might be number of resus cubicles or whole time consultants; process door to balloon time or number of patients receiving analgesia; outcomes tend to be measured in terms of mortality (not as straightforward as it sounds – more of that in another post).

Now this is where I start to have trouble with our current setup; it’s all very well having lots of space and having a lovely short time to triage if the patients don’t benefit – after all, the patients are who we’re there for. So all these process and structural measures are proxies because we think they’ll improve outcomes. But we only know that for a very few: reperfusion time in AMI, prompt defibrillation in VF, and care in a specialist stroke unit being some. Processes and structures are easy to measure, but that doesn’t mean they’re the right thing to measure. I’ll let you think about that.