What’s wrong with hospital rankings?

Why ‘Rate my Hospital’ needs a second look

A version of this commentary appeared in the Winnipeg Free Press, the Medical Post and iPolitics.ca

The CBC’s Fifth Estate recently produced an investigation on the quality of hospitals in Canada — “Rate My Hospital” — which has been enormously popular and set off discussions across the country about the need to improve our hospital services. Clearly getting a better picture about how our hospitals perform is of interest to Canadians, and the wish to exceed status quo health service delivery resonates with patients, policy makers and healthcare providers alike.

So what’s the problem?

Rate My Hospital is based on various pieces of evidence that were collected from patients, hospital workers and hospitals themselves (where they cooperated), along with data from the Canadian Institute for Health Information (CIHI).The general goal of the initiative is to make all of the information relevant to patients at the individual hospital level: anyone can go on the program’s web page, pick a hospital, and learn about how well it performs.

Each hospital receives a letter-grade built on five standard indicators of hospital performance: mortality after surgery; re-admission after surgery and medical treatment; and adverse events after surgery and medical treatment. CBC used indicators taken from CIHI annual hospital data, “standardized them” within four peer-groups (teaching hospitals, large, medium and small community) and then assigned each hospital a grade.

So, instead of saying that St. Joseph’s Healthcare (Hamilton) has a 5.34/1,000 mortality rate compared to 8 at the national level, Rate My Hospital determines that it is one standard deviation below the average in its peer-group, thus, it receives a letter grade A.

The goal of making data accessible and in a formula most of us can understand is laudable.Unfortunately, it is also misleading.

For starters, the formula assumes that each indicator contributes equally to the overall ranking. However, how can we decide that re-admission matters as much as mortality?

But the main concern is that the ranking system is relative, not absolute.In any system, excellent or terrible, some hospitals will be one standard deviation away from the mean, by the very definition of the standard deviation.A standard deviation is the average distance to the mean. It necessarily takes some units to be distant by more than the average to get that average distance.

Does this mean that the information in Rate My Hospital is valueless?

Certainly not, but it should be of more interest to hospital CEOs than to patients or relatives. Patients do not really need an overall score for each hospital, because individual patients are admitted for a specific diagnostic: as a result, they should be much more interested to know how a given hospital performs on a specific treatment.

And what are the untold consequences of publishing evidence on adverse events at the hospital level and having patients use that evidence to decide where to be admitted?

It really depends on the origin of adverse events: if they result from overwhelmed providers in facilities used beyond capacity, publication of adverse events may have a welcomed balancing effect. That is, patients may move to facilities with lower occupancy rates and lower adverse events rates — a good thing.

But, if they result from caseload characteristics, such as more frail patients at one hospital than another, and if better informed, potentially less frail, patients are the most likely to use that information to select their hospital, some hospitals will end up with ever more complex and frail patients whereas others will end up cherry picking the easy ones.In other words, it could lead to imbalance, and ever greater disparity in outcomes between facilities.

Other data used by Rate My Hospital are similarly problematic, such as the survey in which hospital nurses were asked whether they would recommend their hospital to relatives. The results made headlines because, worryingly, 25% of nurses would not recommend their own institutions. However, this was not really a representative survey and it is quite likely that dissatisfied nurses were more willing to answer the survey (in the negative).

There is no doubt that having more information available on hospitals is good thing – and patients should make healthcare decisions based on good evidence. But patients do not need rankings based on assumptions and standardization that paint only a general portrait. Patients need more refined measurements of how a given hospital performs on a menu of relevant items specific to them.

We can’t rely on the media to provide this.Perhaps it’s time governments — and hospitals themselves — stepped forward and worked together with the media to make this happen.

Michel Grignon is an expert advisor with EvidenceNetwork.ca, an associate professor with the departments of Economics and Health, Aging & Society at McMaster University and Director of the Centre for Health Economics and Policy Analysis (CHEPA).

License to Republish: Our commentaries, Infographics and videos are provided under the terms of a CreativeCommons Attribution No-Derivatives license. This license allows for free redistribution, commercial and non-commercial, as long as it is passed along unchanged and in whole, with credit to the author and EvidenceNetwork.ca
EvidenceNetwork.ca supports the use of evidence when reporting on health and health policy in the mainstream media. Specific points of view represented here are the author’s and not those of EvidenceNetwork.ca. Let us know how we’re doing: evidencenetwork@gmail.com

Find an Expert

We have experts (listed by topic and by geographic region) who are ready and able to answer media questions and connect you with the evidence on issues in health policy.

Why We Need More Canadian Health Policy in the Media

This free ebook is a compilation of the EvidenceNetwork.ca commentaries published in major newspapers in 2015, written by experts in the health policy field.