It’s very difficult to assess a doctor’s true skill or performance

How do you really know if your doctor, surgeon or hospital is good, bad or somewhere in between?

I frequently speak with large audiences, ranging from CEOs to Stanford business school students.

I often start by asking participants to raise their hands if they receive excellent health care. Each time I ask, about 90% of the hands in the room shoot up. But all the hands come down when I ask, “How do you know?”

When put in that context, people realize how little information they really have about their physician or hospital’s performance.

It’s easy to be fooled by a doctor’s reputation or the elegance of a waiting room. It’s very difficult to assess a doctor’s true skill or performance.

Of course, there are some indicators of a primary care physician or specialist’s basic competency. Board certification is the most reliable measure.

However, few of us are searching for a doctor who simply meets basic levels of competency.

We can gain insight into a doctor’s bedside manner and style of practice from online sites like HealthGrades, RateMDs.com orYelp.

But if we want to understand the quality of their outcomes, level of clinical expertise and diagnostic acumen, these subjective and anecdotal surveys don’t do the job.

Of course, the doctor-patient relationship is important. But what most patients care about is whether their doctor’s clinical results are better or worse than others.

The answer remains elusive for multiple reasons, some legitimate and some not.

Take an operation like removing your gallbladder through a laparoscope. Over 95% of the time, a surgeon will solve the patient’s problem correctly. The incidence of complication is rather low.

But let’s assume the difference between great and average is 98% complication free versus 95%.

From a statistical perspective, the difference between a great surgeon and an average one is huge (more than double the complication rate). However, nearly all of their patients will experience an uncomplicated outcome. That’s why asking a friend about their experience isn’t very helpful in determining quality.

So, if a friend can’t provide objective, comparative information on the expertise of physicians, what about asking your personal physician for a recommendation?

Although it’s better than asking a neighbor, few physicians have objective knowledge on the results of their colleagues.

Asking a physician might work if all doctors observed each other in the office and the operating room. But they don’t.

Your primary care physician never directly observes the work of a specialist. And sometimes, the doctor’s referral depends more on how nice the specialist is to the referring physician – not the superiority of their results.

How about finding out where your doctor or surgeon went to medical school or whether they have tenure at a local university?

It sounds great in theory but communicates little in reality. What it takes to be accepted to a prestigious university or hired onto a university faculty is different than what it takes to produce the best clinical outcomes.

What gains entry into an Ivy League school or onto a university faculty is the ability to do well on written tests, perform laboratory research and publish papers. It has nothing to do with manual dexterity or an ability to read an x-ray. The two are not opposites, but the correlation to clinical excellence is minimal at best.

So, where can we turn?

Data, not word-of-mouth, informs us that not all hospitals are equal. The Leapfrog Group has done admirable work in separating the best from the rest. Similarly, many states report outcome data on heart surgery and other complex procedures in U.S. hospitals.

In New York state, for example, the results after cardiac surgery are risk-adjusted (that is, corrected for the complexity of patients, severity of illness and other complicating conditions) before they’re published. Hospitals are then ranked by outcomes to help patients select the best from the rest.

But even where quality data exists, patients often don’t use it consistently or at all.

Case in point: When a former U.S. president needed surgery, he chose a hospital that was ranked near the bottom. He suffered a major complication that theoretically might have been avoided had he made a different choice.

Within larger health organizations like university hospitals and integrated delivery systems, we can predict better results from providers that use an electronic medical record (EMR), which provides comprehensive information on each patient 24/7.

Without a comprehensive EMR, doctors experience delays in acquiring a patient’s diagnoses, medications and lab results. This delay compromises quality and, as a result, patients can fall through the cracks.

Whenever possible, patients should ask whether their doctor or hospital uses a health IT system. While more than 80% of hospitals use EMRs, only about half of all doctors are taking advantage.

Patients may also want to be sure the physician they are referred to for treatment has no conflicts of interest.

Today, some physicians continue their close ties to the pharmaceutical and medical device world, which clouds independent judgment.

That’s why the Federal Government passed the Sunshine Act, which requires public reporting of all payments to health care professionals and teaching hospitals. The information will be available in a public database beginning this September.

In the meantime, ProPublica has already compiled payment information made public by pharmaceutical and medical device companies into a single database.

The reality of this myth is that there is no simple way to assess a doctor’s skill or clinical outcomes.

That’s why more and more patients are gravitating to large medical groups, which are filled with physicians who work closely with each other on a daily basis, observe each other’s work directly and evaluate their colleagues frequently.

When we look at published results from organizations like the National Committee for Quality Assurance (NCQA), Consumer Reports and the Medicare Advantage Star program, we see the same health care organizations at the top of the list more often than not.

Over the next few years, this type of information will become increasingly available. Wise patients will pay greater attention to the data to increase their odds of obtaining the best possible care.

There is a tendency today for us to believe we have the best doctors and health care available. After all, we chose them. But once again, what seems logical only serves to shine a light on another medical myth.

It’s very difficult to assess a doctor’s true skill or performance 16 comments

Comments are moderated before they are published. Please read the comment policy.

rtpinfla

This feels like someone is trying to plant the seeds of doubt in the minds of our patients. A lot of people in the example at the beginning of the article feel like they received good quality health care. They almost certainly did receive just that.
Then comes this guy extolling the virtues of “data” to make those same patients think, maybe I am getting terrible care from my doctor. This is laughable if it weren’t scary for so many reasons.
First, “quality data” if at best hard to interpret and at worst risks lives. A great example is those “ICU quality” measures regarding tight glucose control in the ICU. Any patient that chose a hospital based on that data would have actually increased their risk of dying in the ICU.
Another example- A doctor that is concerned about a diabetic patient’s risk of falls and lets that patient’s glucose drift up to ensure no falls from hypoglycemia would should up as an “inferior doctor”. And the other doctor that wants his data to show that he is “superior” will have to add more medicine, putting that same patient at risk of a fall. So the doctor that causes a patient to fall and break a hip trying to keep that glucose level low will show up as the better doctor, even though the “inferior” doctor did the right thing.

And IT improves outcomes? Really? Go ask anyone-patient or provider- in the VA system how much the love the quality they are getting from their EMR system. It seems that all EMR’s really do is make it easier for bean counters to collect this supposed “quality” data.
If, as a patient, you feel like your physician listens to your concerns and is taking good care of you, he or she almost certainly is providing high quality care.
Don’t let some bean counter try to tell you otherwise with his pile of…. data.

LeoHolmMD

“Of course, there are some indicators of a primary care physician or specialist’s basic competency. Board certification is the most reliable measure.”
Since when?

I agree that board certification has no bearing on competency. But I also think patient satisfaction is target that is moved by different patient expectations.

Its like choosing a fast food restaurant when you are on the go. Given the choice, I’m going to choose Chic-fil-A everytime over Burger King, McDonalds, etc. Why?

Because you can see the heart in people who work there. They aren’t just being cliche when they serve you and say “My Pleasure.” Its something that the leadership of the company has engrained in them.

In my office, I constantly get comments from patients of how peaceful and inviting the Pediatric environment is. Our staff carries that same ‘my pleasure’ attitude. That is because it was a priority for me and my practice manager when I went back to solo practice in 2010.

I really do want to serve my patients. But I’ll be the first to admit that I also ought to know the ropes after 30+ years of Pediatrics. But those mothers and families look just like my daughter and her family. The children remind me of my grandchildren.

There is something intangible and beyond metrics about a physician’s heart to serve. That’s what patients feel when they say they like their doctor. Good physician beside manner can only fake a true heart for patients for so long.

Warmest regards,

Ron Smith, MD
www (adot) ronsmithmd (adot) com

James O’Brien, M.D.

Maybe we should start a competing patient satisfaction survey company. With a twist.

The difference will be if a doctor says no to a patient demanding an unnecessary antibiotic, opioid or stimulant and that patient rates you as a bad doctor….

you’re rating goes up….and down if you give in…

rbthe4th2

ROFL. How about if the doctor gives you an antibiotic and you don’t want it but you have to take it any way?

And do?

LOL.

azmd

Actually, I recently analyzed some data that indicated prescriber errors tripled in the 18 months after our organization implemented a CPOE system.

LeoHolmMD

No surprise. There are brand new types of errors that no one is measuring.

guest

Exactly. Frustratingly, they frequently don’t end up getting reported to MIDAS so we don’t completely capture all such events. But it’s also well-documented that CPOE increases “provider cognitive workload.”

Since we all only have only so much cognitive energy, it stands to reason that there are other cognitive tasks in medicine getting shortchanged in favor of our navigating the shoals of the CPOE.

If “data” is a good indicator, let’s get the data.
First, where is the link to a study or two that support the assertion that physicians using IT are better?
Second, where is the data to allowing patients to see all the conflicts of all the interests as experienced by physicians? Sure, we have a massive database documenting receipt of bagels and brownies from Panera, but zero information on the contractual arrangements of employed physicians, who are operating under a perpetual conflict of interest, particularly when their employer is also a risk bearing entity, such as a payer. So let’s publish all that data.
Third, where is the supporting documentation for the assertion that people “gravitate” to large systems? There is plenty of “data” showing how large systems are buying doctors and other systems, thus forcing patients to obtain care at much higher prices from those large systems, sometimes unbeknownst to the patient until they get hit with the bill.
Fourth, Consumer Reports explicitly excludes small practices (three and under) in its “unbiased” ratings done in collaboration with insurers, who have zero conflict of interest, or so we are told.
Fifth, if people are incapable of assessing the quality of care they receive, what is the purpose of patient experience surveys, on which much of the “data” they are supposed to look at is based?

rbthe4th2

I think one of the best indicators I’ve seen so far, and would add to your excellent list, is a description by the patients of their personal practice style: collaborative?, open to research?, reasoning ability?, keeping up with research?, how do they deal with conflict? how do they deal with problems?, how do they deal with mistakes?

I had one doctor who disclosed to everyone his worst case where he got sued. I checked and verified and every word he said was true. I have never left his group and will wait for care from his group because he was honest.

I dont have to worry about honest docs. Mess up – Fess up – Fix it. Apologize. Learn from it.

If they do that, I get quality care. If not, I don’t.

SarahJ89

“How do you really know if your doctor, surgeon or hospital is good, bad or somewhere in between?”

You don’t. That’s why I refuse to fill out those stupid patient surveys.

rbthe4th2

I also refuse to do that. I receive good care, I go to their ADMIN. I had one tell me can you put something in writing for review/raise time? I gave them something within 24 hours. Forget that aggregate data crap, I want to make an impact and that’s how you do it.

Btw, you ask questions. The right ones will help you every time.

Karen Ronk

The sad reality is that you can check on a doctor’s credentials, board certification, online reviews, etc. and still end up with a bad outcome. In my opinion, it comes down to good old fashioned word of mouth. That, and physicians willing to be honest about bad doctors.

rtpinfla

Which wild conjectures are you referring to? I assume you mean my comment about the VA so I will discuss that a bit. But I’m happy to expound on any of the other conjectures.

I am in fact volunteer faculty once a month at a VA clinic so have first hand experience with the “system”. The EMR is actually not bad in some ways. The access to ancillary data and pharmacy profiles is invaluable. But that EMR does not exist in a vacuum. I won’t bore you with all the examples of the issues that our patients have to deal with but suffice it to say I would never put up with the hassles both physicians and patients have to deal with in my private practice. Thank God the workers in the trenches care enough about their patients that they can sometimes get what they need. The administrators only are interested in buffing their data and making sure they don’t look bad to the system. Vets get good care DESPITE the VA system and EMR, not BECAUSE of the VA system and their EMR.

As a military physician I was regularly admonished for not using CHCS-2 properly. I had coders calling me regularly telling me that my diagnosis was “not allowed because it didn’t support their E&M level.” This policy was supported by the command.

So GS-5 ‘coders” (actually they were auditors, looking for inconsistencies in the codes EMR generated)) was able to tell me, an O-5 sub specialist, what diagnosis I could use for a patient. For examples, they didn’t want me to ever use “anaphylaxis” as a diagnosis. I was supposed to diagnose “rash” and then I could say that it was a really bad rash somewhere in the free test. Each quarter my “quality measures” showed sub standard performance because I was not diagnosing patients according to the wishes of the computer and the GS-5 auditors. I am not exaggerating.

Finally, I was involved when the military started monitoring HEDIS data and measuring “quality” at a the overseas hospital that I was stationed at.. All I can say is “garbage in-garbage out”. Initially, many of the docs’ numbers looked terrible. The problem was, a significant number of the patients in the data base were never seen by the physicians that they were assigned. Many didn’t even live on that base anymore. Had you used this data you would have thought every doc in that hospital was an incompetent death machine. None the less, in 6 months the command showed significant “improvement” in all quality measures. Our asthma and diabetes adherence went up like 50% and we showed well above the national average. Now all those same doctors were the best doctors in the country. But it was all pretty much a shell game and was actually pretty easy to do.- the doctors were providing the same level of care as before (I think damn good care, by the way) I am sure this “improvement” was cited in our CO’s end of tour award.

EMR per se does NOT equate quality care, as the author suggests. And data can be manipulated to show anything you want and does not reflect accurately what goes on in the exam room.

SteveCaley

Americans prefer branded mediocrity to actual quality, which requires thought, preference and judgment. Fifty years ago, we did not have the calm reassurance that brand-names offered for restaurants, hotels, and many other accommodations, promised. You had to figure out if a motel was a good place to stay, based on your own judgment and not an iApp. Now, you don’t need to know where to eat, beyond the Applebees Locator App.
The “Locum Tenens” companies are swiftly turning into wholesale distributors of physician-months, and a tremendous percentage of our new physicians are commoditized and transacted as “locum futures.”
As soon as the brands become a bit more imprinted in the national identity of physicians, we will see doctors imprinted with logos, that no longer mean anything that they once did. Hire a Mayo Physician, a Dartmouth Physician, a Harvard Physician. These will not mean that they are trained at these institutions – they are rather physicians branded by them, in a combined board/employer sort of way. He’s a Kellogg’s doctor; she’s a Honda, this one is a PepsiDoctor.
It is inevitable, and will not be interrupted. We will have WalMedicine before the end of the next presidential administration, literally as well as figuratively.
Consumer quality perception is anathema to marketing. I’m loving it.

W. X. Wall

I understand where you’re coming from but IMHO you’re being naive about markets. You say that in a market-driven system people will are incentivized to get better value. That’s what we wish happens. But we all know that in a market-driven system, the major players strive to gain market distortions such as monopoly power that would allow for profit without the hard work of providing quality care. And that’s what’s happening. You say that prices are static. I disagree. Reimbursement rates are declining while insurance premiums are increasing. On either end, prices are hardly static and they are being driven by who has power in the market, not by who is doing a better job providing quality care.

What does all this have to do with data? We clinicians think of data in terms of its ability to allow us to improve care. We’re naive. Insurance companies and hospitals think of data as marketing. Indeed, the best work currently being done on Big Data is by business schools in their marketing departments.

Market-driven entities like hospitals and insurance companies are spending billions to acquire Big Data because it allows them to mine for better customers, identify costs that can be cut, and be sliced / diced to show why they are better than the hospital down the street. Whether quality improves or declines is immaterial.