The blog of Ashish Jha — physician, health policy researcher, and advocate for the notion that an ounce of data is worth a thousand pounds of opinion.

Changing my mind on SES Risk Adjustment

I’m sorry I haven’t had a chance to blog in a while – I took a new job as the Director of the Harvard Global Health Institute and it has completely consumed my life. I’ve decided it’s time to stop whining and start writing again, and I’m leading off with a piece about adjusting for socioeconomic status. It’s pretty controversial – and a topic where I have changed my mind. I used to be against it – but having spent some more time thinking about it, it’s the right thing to do under specific circumstances. This blog is about how I came to change my mind – and the data that got me there.

Changing my mind on SES Risk Adjustment

We recently had a readmission – a straightforward case, really. Mr. Jones, a 64 year-old homeless veteran, intermittently took his diabetes medications and would often run out. He had recently been discharged from our hospital (a VA hospital) after admission for hyperglycemia. The discharging team had been meticulous in their care. At the time of discharge, they had simplified his medication regimen, called him at his shelter to check in a few days later, and set up a primary care appointment. They had done basically everything, short of finding Mr. Jones an apartment.

Ten days later, Mr. Jones was back — readmitted with a blood glucose of 600, severely dehydrated and in kidney failure. His medications had been stolen at the shelter, he reported, and he’d never made it to his primary care appointment. And then it was too late, and he was back in the hospital.

The following afternoon, I spoke with one of the best statisticians at Harvard, Alan Zaslavsky, about the case. This is why we need to adjust quality measures for socioeconomic status (SES), he said. I’m worried, I said. Hospitals shouldn’t get credit for providing bad care to poor patients. Mr. Jones had a real readmission – and the hospital should own up to it. Adjusting for SES, I worried, might create a lower standard of care for poor patients and thus, create the “soft bigotry of low expectations” that perpetuates disparities. But Alan made me wonder: would it really?

To adjust or not to adjust?

Because of Alan’s prompting, I re-examined my assumptions about adjustment for SES. As he walked me through the data, I concluded that the issue of adjustment was far more nuanced than I had appreciated.

Here’s the key: effective socio-economic adjustment doesn’t reward providers for giving bad care to poor patients. It just ensures that they aren’t penalized for taking care of more of them. In my clinical example, if people like Mr. Jones had a higher readmission rate, adjusting for SES wouldn’t give hospitals credit for lower quality care to poor patients. Done right, it would give credit to hospitals for having more poor patients, and that’s an important difference. Consider three scenarios of hospital performance on a readmission rates (modified from our JAMA piece).

In scenario 1 and 2, let’s assume that patients are readmitted 20% of the time on average, whether or not they’re poor. In scenario 1, Hospital A (a safety-net hospital) has higher readmission rates for everyone. They may have more poor patients, but their readmission rate is high for both poor and non-poor patients. So, compared to Hospital B, they look worse in unadjusted and adjusted scores. Adjustment doesn’t help.

In scenario 2, Hospital A has higher readmission rates for its poor patients and therefore has an overall readmission rate of 25%. Hospital B doesn’t suffer from readmitting its poor patients too often – hence its readmission rate is 20%. In this case, safety-net hospitals look worse than Hospital B in both unadjusted and adjusted analyses. Again, adjustment doesn’t help.

In scenario 3, Hospital A and B both struggle with readmissions for their poor patients – as does the rest of the country. The only thing that differentiates Hospital A from Hospital B is the proportion of poor patients in the hospital. In this case, adjustment makes a big difference. By adjusting, we account for the different proportions of poor patients between Hospital A and B. Adjustment ensures that organizations are judged by how well they care for their patients, not by how many poor patients they have.

One Size Does Not Fit All

The debate about whether to adjust for socioeconomic status needs to be far more nuanced than it has been to date. Specifically, we must recognize that quality measurement has multiple purposes, and we need to think about each one when deciding whether to adjust or not. If the goal is transparency –letting patients know how they are likely to fare – then the best approach is stratified data. In scenario 3 (where adjustment makes a difference) a poor patient will do about as well at both hospitals – and unadjusted numbers are misleading, because they tell poor patients that hospital B is better. If Hospital B has a larger co-pay or is out-of-network, you have done real harm by pushing a patient to a more expensive place that doesn’t provide better care.

To push hospitals to improve quality, unadjusted numbers are best. In all three scenarios, Hospital A should be more motivated to get better than Hospital B because for its patients, it tends to have worse performance. But in each scenario, the hospitals need stratified data. Without it they will have no idea where to target their efforts.

For penalties, we should use adjusted data. It will make no difference in scenarios 1 and 2. But, in scenario 3, it makes little sense to penalize the safety net hospital compared to other hospitals just for taking care of more poor patients. That’s not a smart policy. Penalties for bad care for poor patients? Sure. Penalties just for caring for more poor patients? Not so sure.

A way forward

The bottom line is that the care of poor patients is not evenly distributed across all U.S. hospitals. Some hospitals have a lot more patients like Mr. Jones than others have. And caring for people like him, who are homeless and without a social network, is challenging. None of us are very good at it. Why penalize the safety-net hospitals just for taking care of more poor patients?

Given the concern that safety-net hospitals may be disproportionately penalized, a bi-partisan group of Senators (3 Democrats and 3 Republicans) has signed on to a bill that would require CMS to account for SES when it doles out penalties for the HRRP (Senate Bill 2501). It’s an excellent start.

Adjusting for SES is an acknowledgement that medicine is not the only factor – and indeed may be a relatively minor factor – in health outcomes. For Mr. Jones, homelessness and poverty clearly contributed to his readmission to the hospital. Bad medical care did not. We should have no qualms penalizing safety-net hospitals for providing sub-standard care. But we just shouldn’t penalize them simply because they have more poor patients.

6 thoughts on “Changing my mind on SES Risk Adjustment”

Ashish. great piece. i agree with all your sentiments against such risk adjustment (not wanting to obscure poor quality care delivered to poor patients) and also with your arguments for actually going and making such adjustments. The only caveat is that, when hospitals care for quite different proportions of poorer patients, risk adjustment can still give misleading results – sometimes completely distorting the true relative performance of the hospitals being compared. I am referring to Simpson’s paradox – oft discussed in public health and epidemiology circles, but less so in health services research. A fellow editor at BMJ Quality & Safety and I recently wrote an editorial/mini-review on the topic to accompany a study empirically demonstrating the effect among a cohort of NICUs in UK for which there was detailed performance data and data for risk adjustment. The editorial includes an easy to understand healthcare example as well as a baseball example involving batting averages. [Simpson’s paradox: how performance measurement can fail even with perfect risk adjustment. BMJ Qual Saf 2014;23:701-705 http://qualitysafety.bmj.com/content/23/9/701.full?sid=df1d90e4-4f20-4986-bd28-56398a63324f ]

The upshot of the editorial is that, even with perfect risk adjustment (whether for purely clinical variables or SES), one must avoid comparing hospitals with disparate proportions of high risk patients. Suppose Hospital A and Hospital B do equally well for “non-poor patients” and equally badly on poor patients. But, Hospital A only has 2% such patients and Hospital B has 10%, it won’t matter that you have adjusted for SES. Hospital B will look like it has a much worse overall risk-adjusted readmission rate than does Hospital A because it has more opportunities to manifest its poor quality care delivered to poor patients. In fact, there can even be cases in which Hospital A is a little worse than average for non-poor patients and much worse than average with poor patients, but it will still come out looking better than Hospital B.

So, while I see your arguments, I think that in practice, one has to flag hospitals that take care of more high risk patients (poor patients in your example)and not attempt to lump them in with other hospitals. Or, we have to analyze high risk patients separately. I don’t think there is a way for even perfect case-mix adjustment to deal with the problem created by substantial differences in the proportions of high risk patients cared for by some hospitals versus others.

Still liked the blog piece a lot and, again, share all your sentiments. I only recently became aware of the degree to which case-mix differences can bedevil performance measurement even when we have perfect risk adjustment (never mind the many cases in which risk adjustment is far from perfect).

You have made great points. This is a particularly difficult issue. While we can all agree that hospitals should not be unduly penalized, there must be incentive for providers to reach beyond their ‘four walls’ and help those people most in need.

Excellent piece — nicely illustrates the impact of not adjusting for SES. A key issue is how to appropriately adjust for it. In your view, should SES adjustments take into account patient-level attributes (e.g. dual eligibility), community-level attributes (e.g. indicators of disparity in the patient’s ZIP code of residence), hospital-level attributes (e.g. safety-net status), or all of the above?

I’ve always been torn on this issue — great discussion. From a policy standpoint, we should all be against worse care for poorer patients. On the wards, however, it seems unreasonable for CMS to consider my patients at Columbia (Washington Heights) as similar to, say, Lenox Hill’s (Upper East Side). I don’t know how significant the difference actually is — probably not as enormous as is claimed — but it’s enough for working docs to grumble that the policy is ridiculous and unfair.

A potentially under-appreciated benefit of adjusting for SES (if such a thing can even be done) might be broader acceptance by critics, who feel that the program is set up for them to fail at the outset. Maybe you’re rewarding grumblers here, but sometimes you have to. Realistically, the amount of money at risk is large but not huge. If hospitals think they have no chance of preventing readmissions, especially if they feel the rules are unfair, some will eat the penalties ($ 10 million or less even for huge hospitals), and make up the difference on the superbill, the IRS 990 form, and the cath table.

I enjoy reading pieces like this, especially the step by step examples for each case to break it down for the “outsiders” like myself. Two things: 1) are there other quality measures that can also further justify the case for SES adjustment – e.g. utilization of resources (high utilizers vs. low for “poor” and “nonpoor”) or ER visits or medication adherence? 2) My physician/public health idol is Rishi Manchanda and he’s doing some great work at a Veterans Affairs clinic that serves homeless vets – I hope you can check him out.
Thanks for sharing your thoughts and writing again despite the tremendous responsibilities you have at your position.

Enjoyed your piece and the new perspective that you have taken away from it. I think seeing it from the patient perspective and understanding the impact other examples like this would have on other safety net hospitals is critical.
I think if more policy experts would spend a couple of days in any inner city, or financially challenged neighborhood where a hospital is delivering care they would see many more examples like this, several times a day. All hospitals and the professionals that work in them want to deliver the best care that they can, and they should not be penalized by a system that does not take the very important variables into consideration that you discussed, as they are real and critical factors that have to be taken into consideration in the care of these vulnerable populations.