US News hospital rankings too subjective and encourage self-promotion

Every year US News & World Report publishes its rankings of the nation’s top 50 hospitals. Hospital administrators await this top 50 report with a tension and fervor that rivals the NFL first-draft pick.

As soon as the report is released, snippets rocket their way into donor appeals, local hospital signage, and highway billboards. “Ranked X by US News & World Report” becomes a descriptor that is supposed to immediately invoke quality.

Just what is behind the US News & World Report rankings, and do they actually mean anything? Dr. Ashwini Sehgal, writing in the Annals of Internal Medicine, examined the structure of the famed ratings.

US News & World Report utilizes three components, equally weighted, to evaluate hospitals—structure, outcome, and reputation.

Reputation is a much more vague metric. Two hundred and fifty physicians from each of the medical specialties were randomly selected from the AMA Physician Masterfile. These doctors—about half of whom responded to the survey—were asked to rank the top 5 hospitals in their specialty. From these personal opinions, a reputation score was developed.

Of these three measures—according to the statistics in the Annals article—reputation correlated most strongly to the final ranking.

This means that this supposedly objective ranking of the nation’s hospitals depends mostly on the subjective reputation score. Not exactly the scientific, objective scale is purports to be.

On the home page for the rankings, there is a side-bar ad. “Promote Your Hospital,” it says. “Display your ranked hospital’s Best Hospitals badge on your website.” Clicking onto that link offers numerous opportunities for hospitals to “enhance reputation and visibility” via US News & World Report. One can “purchase an enhanced hospital listing” or “become a featured hospital.”

These rankings, therefore, seem mainly to be a marketing tool for US News & World Report. This shouldn’t come as a surprise to anyone. US News & World Report is a commercial entity whose goal is, and justifiably needs to be, earning revenue. It is not a scientific non-profit such as the Institute of Medicine or the NIH, from whom we expect high-quality, unbiased evaluation.

However, once the US News & World Report becomes a household name for hospital rankings, it does have some obligation to be as objective as possible. If it wants to be an arbiter of quality in any sort of quantitative sense, it has to demote the input of subjective measures.

The reliance of the subjective value of reputation suggests that we will be in an endless loop of self-promotion. Newer, smaller hospitals will never have a chance to be ranked at the top, even if they deliver superior care, simply because 125 doctors don’t know of them. But maybe they should

Comments are moderated before they are published. Please read the comment policy.

Elizabeth

“Of these three measures—according to the statistics in the Annals article—reputation correlated most strongly to the final ranking.

This means that this supposedly objective ranking of the nation’s hospitals depends mostly on the subjective reputation score. Not exactly the scientific, objective scale is purports to be.”

Sounds to me like confusion of correlation with causation. It isn’t just for medical studies.

DrQ

Or it could mean that the other 2/3’s of the equation, structure and outcome, are nearly uniform with not much differences in quality. Reputation, on the other hand, would have much more significant deviations, again supporting the idea of the rankings relying mostly on subjective criteria.

Christopher Staeheli

Having worked at a “top 50 Hospital” I always wondered how it got on the list. I discovered there was really no central QA process for compiling problems and addressing problems unless it mounted to a “sentinel event” so major errors in patient care that did not involve a death simply went undocumented and not addressed. The algorithm for an incident report stopped at the department manager unless it was a sentinel event. I certainly was not aware that any incident report I filed never went beyond the manager until i called the QA department to find out why some major lapses in care continued to go unaddressed. US News and World Report needs to add problem reporting to its list. The banks looked pretty good too when they did not report their problems, and they nearly killed us all.

ninguem

Every hospital in the country is a a top 50 hospital.

http://www.sherwoodcommunications.com Sarah Sherwood

Reputation should matter a bit because we are really talking about patient experience–which should matter the most. I am aware, though, that it is very hard to quantify a patient’s experience.