Hot Topics:

Critics: Rankings unfair to doctors, misleading to patients

By Todd Feathers, tfeathers@lowellsun.com

Updated:
06/26/2016 07:45:38 AM EDT

It was an ambitious experiment aimed at saving taxpayer money and ensuring that Massachusetts' public employees saw the very best doctors, but a $7 million, state-run doctor ranking system may be unintentionally misleading patients about their doctor's quality.

The rankings, known as tiers, affect the more than 430,000 state and municipal workers, retirees and their family members whose health insurance is paid by the Group Insurance Commission.

Through complex medical claims analysis, the GIC places specialist physicians in one of three tiers, where Tier 1 doctors are supposedly the best and most cost-effective. But experts say the rankings place cost above quality and that the cost assessments draw from sample sizes that are too small to be statistically reliable.

A single prescription sent one Tufts Medical Center cancer specialist from Tier 1 in 2014 to Tier 3, the lowest ranking, in 2015. And so many doctors switch tiers from year to year, without any apparent changes in the way they practice medicine.

"(The rankings) are totally unstable from year to year," said Dr. Mark Friedberg, who researches health-care delivery and payment models for the RAND Corporation, a nonprofit that studies public policy.

"It's basically lying to people," he added.

The officials who oversee the GIC call the rankings imperfect but necessary.

The Sun interviewed dozens of doctors and health-policy experts, and analyzed data on nearly 6,000 doctors tiered by the GIC.

Advertisement

Among the findings:

* About 27 percent of doctors changed tiers in at least one of the GIC's partner insurance plans from 2014 to 2015. Some switched all the way from Tier 1 (excellent) to Tier 3 (standard), or vice versa.

* A doctor's tier is supposed to be based first on a quality-of-care score and second on a cost-efficiency score. Yet only a third of doctors were given quality scores in 2015. In nine out of the 17 specialties the GIC tiers, such as general surgery and oncology, no doctors received quality scores.

* The rankings determine how much GIC patients pay to see a doctor. The co-pay for a Tier 1 doctor is $30, but if that doctor's rank falls to Tier 2 patients must pay $60 for a visit. If the doctor falls all the way to Tier 3, patients must pay $90.

"Is it a perfect program? No," said Ray Campbell, interim executive director of the GIC. "But I think it's a very good program focused very narrowly at a specific problem, which is the quality of care provided by specialists. We actually feel quite good about it."

A small portion of the fluctuations in doctors' tiers were due to their quality scores blinking into and out of existence, he said. The GIC only assigns a quality score if it can calculate with 90 percent certainty that it is accurate.

Most changes, however, involved doctors who did not have quality scores. Without examining each doctor individually, the GIC could not explain why such a high percentage of doctors switched tiers.

Dr. Ateev Mehrotra, a professor of healthcare policy and medicine at Harvard Medical School who has studied doctor tiering in Massachusetts, said his research shows that the GIC ranks are subject to a large amount of chance because they are based on insufficient sample sizes and can be thrown off by a particularly sick patient.

After interviews for this article, Campbell said the GIC will dig into its data to determine why so many doctors fluctuate in the rankings.

"We are sensitized to this issue of tier switching now and we will look at it more closely," he said.

"We think the patterns that emerge are defensible, but we're not going to leave it at that."

Criticized from the start

The GIC's rankings, which are part of its Clinical Performance Improvement initiative, were created by the agency's longtime executive director, Dolores Mitchell.

Mitchell, who retired in March after 28 years with the GIC, was not the stereotypical bureaucrat: She decorated her office with a bullwhip. Her agency's annual reports, usually dull, technical documents, were written around themes such as Alice in Wonderland and Star Trek.

At the time of its launch in 2006, CPI was possibly the only program in the country that ranked individual doctors, rather than the physician groups and practices that employed them, on cost and quality, according to health-care experts.

The state has spent $7 million on the CPI since work on it began in 2004, and it consistently costs more than $500,000 a year to maintain. The annual cost is slowly rising as the program collects more data.

CPI has been a lightning rod for criticism since its inception.

In 2008, the Massachusetts Medical Society, which represents the state's doctors, sued the GIC and two insurance plans. The MMS alleged that CPI defamed doctors ranked in the lower tiers and defrauded patients who used it to select physicians.

"There has been a fair amount of concern raised about tiering by our members for several years and the GIC method of tiering was particularly worrying based on the methodology they use," said Dr. Dennis Dimitri, a Worcester physician and past president of the MMS. "We shouldn't apply flawed methodology and inadequate methodology and then have both physicians and patients suffer as a result of these inadequacies."

The MMS dropped the lawsuit after a Suffolk Superior Court judge ruled that, because its members were paid by the insurance companies and not directly by the GIC, the society did not have grounds to sue the state agency.

Many doctors continue to be perplexed and annoyed by their rankings, though, or ignore them entirely.

"I don't think anybody uses it," said Dr. Alan Geller, a Weymouth neurologist. "From the standpoint of doctors, I think it has no meaning."

Geller, Tier 2 in 2014, jumped to Tier 1 in 2015 in all four of the GIC-contracted insurance plans whose networks he is part of. That was good news for any current or retired public employees who visited him: Their co-pays were cut in half.

Geller, though, was surprised. He said he hadn't made changes to his practice over the past year that would have affected his cost efficiency, and he was among the majority of doctors whose tier was not based on quality of care.

He had no idea his tier had changed until contacted for this story.

How much stock does Geller put in the tiering system? When the letter notifying him of his ranking arrives, he said, "it just goes into the wastebasket."

Flaws seen in methodology

Critics say CPI's methodology contains several dangerous flaws. To begin with, the quality of most doctors' care is not taken into account.

Under the GIC's system, a doctor who receives a high enough quality score is supposed to be placed in Tier 1 regardless of cost score.

But the GIC does not measure quality of oncologists, immunologists, gastroenterologists, general surgeons, orthopedic surgeons, dermatologists, ophthalmologists, podiatrists, or urologists.

Those specialists can't be judged reliably on their quality because the private organizations that certify them have not defined adequate quality metrics, Campbell said.

Doctors also complain that they are sometimes placed in different tiers in different insurance plans. Tiers are assigned based on percentages -- the top 15 percent in a given specialty are Tier 1, the middle 65 percent are Tier 2, and the bottom 20 percent are Tier 3 -- and insurance plans have different numbers of doctors in their networks. A doctor ranked in Tier 2 in the smaller pool of Health New England insurance might be ranked Tier 3 in the much larger Tufts Health Plan.

The biggest criticism of the GIC's tiering, however, is that the cost scores the rankings depend so heavily upon are derived using a too-small sample size of 30 cases, known as treatment episodes.

In a 2010 study published in the New England Journal of Medicine, Mehrotra, the Harvard researcher, and his colleagues found that even when doctors were measured based on hundreds of treatment episodes "the majority of physicians did not have cost profiles that met common thresholds of reliability."

"The problem, at a very high conceptual level, is gosh darn it, patients aren't the same," Mehrotra said. "For an individual physician, one or two patients who are unusual, who have a particularly sick presentation, that could be a driver for why that physician's costs are more than expected."

Doctors' cost-efficiency scores can vary based on which episodes the GIC happens to pick for its analysis.

In 2014, Dr. Kenneth Miller was ranked as a Tier 1 hematologist. He was associate chief of Tufts Medical Center's hematology and oncology division and had regularly been selected by his peers to appear on Boston-based Best Doctors Inc.'s list of the top doctors in America.

The next year, he fell to Tier 3. The precipitous drop was due to a single medication that Miller prescribed, said Campbell, the GIC's interim director.

"If you treat patients that aren't very sick, then the resources you expend aren't very much," Miller said.

He couldn't say with certainty which drug the GIC had focused in on, but he suspected it was either Soliris or Ibrutinib.

Soliris is considered the only treatment for PNH, a rare blood disease. It is one of the most expensive drugs in the world, with an annual cost that can be well over $500,000. Ibrutinib is a cancer drug that can cost more than $100,000 a year.

"I see patients who fail the usual treatments and these new drugs are expensive but of great proven benefit," Miller added in an email.

Campbell said the GIC was looking into why the drug that singlehandedly changed Miller's tier wasn't excluded as an outlier and whether Miller's ranking should be adjusted.

The GIC occasionally finds mistakes in its rankings and does its best to fix them, he said.

"Now is that fair to that one doctor that they were a Tier 3 because they're a nephrologist who specializes in sleep studies and their data looks a little different?" Campbell said, referring to a hypothetical doctor. "I guess it isn't fair to that doctor. But is it fair to our 430,000 members and the taxpayers who are spending $2.7 billion to say 'We may misclassify a doctor so we will not measure?' That doesn't seem to be the right answer."

Doctors share frustration

From the outset, the GIC has presented its rankings as a motivational tool that would push doctors to cut back on practices, like ordering expensive tests that may not be necessary, that are driving the nation's exploding health-care costs.

But on that point, the rankings seem to be failing. None of the doctors interviewed for this story knew which tier they were in.

"It's arcane and, really, a physician like me can't understand how insurers determine these things," said Dr. Rachel Nardin, a neurologist at Cambridge Health Alliance Hospital who fell from Tier 1 to Tier 3 between 2014 and 2015.

Dr. Joseph Bouchard, a cardiologist at UMass Memorial Medical Center in Worcester, went from Tier 3 in 2014 to Tier 2 in 2015.

"I didn't even know that my tier changed, so that shows you how much it plays into my everyday practice," he said.

One doctor interviewed does aim for a Tier 1 ranking.

"I don't think there's something wrong with it. This is America, everybody gets a grade," said Dr. Wassim Mazraany, a surgeon at Riverside Surgical Associates in Lowell who dropped from Tier 1 to Tier 2 in 2015.

"It's very difficult to be in Tier 1 and stay in Tier 1 because very little changes can drop you off," he said.

The GIC has never audited the CPI initiative or studied its effects on insurance costs, Campbell said. But research shows that the program's other stated goal, driving patients towards better-ranked doctors, may be somewhat more successful than its effort to reach doctors.

In a 2014 study, Harvard researchers Anna Sinaiko and Meredith Rosenthal found that patients seeing a specialist for the first time were slightly more likely to choose a Tier 1 or Tier 2 doctor than a Tier 3 doctor. There was no indication that patients who were already seeing a specialist were leaving for one in a better tier.

"It's a personal decision, whether they want to leave a doctor they've been going to for years because the co-pay is now $90," said Frank Valeri, president of the Retired State, County, and Municipal Employees Association of Massachusetts. "It's a difficult decision, it's a life decision."

It's also a decision that the GIC hoped would help Massachusetts cope with rising health-care costs. The state and residents spent $54 billion in 2014, according to the Health Policy Commission, more than $600 million over the state's goal.

"There's a sense of frustration, so you have to do something and this is the thing," said Friedberg, the RAND researcher, referring to the GIC tiering system. "This is taxpayer dollars and at the end of the day it's doing nothing."

Welcome to your discussion forum: Sign in with a Disqus account or your social networking account for your comment to be posted immediately, provided it meets the guidelines. (READ HOW.)
Comments made here are the sole responsibility of the person posting them; these comments do not reflect the opinion of The Sun. So keep it civil.