When physicians were asked to rate confidence in their diagnoses, their ratings did not vary much between easy and difficult cases, yet only a few were accurate in their diagnoses of difficult cases, according to a new study by researchers at Baylor College of Medicine in a report that appears today in JAMA Internal Medicine.

"Experts have previously thought that overconfidence among physicians can lead them to miss the correct diagnosis, but there was no clear evidence to suggest that. To support such a statement, confidence needs to be measured accurately," said Dr. Hardeep Singh, associate professor of medicine at BCM, and Chief, Health Policy, Quality and Informatics Program at the Michael E. DeBakey Veterans Affairs Health Services Research and Development Center of Excellence and senior author of the study.

Diagnostic process

Researchers sought to simulate and capture the evolving diagnostic process that physicians go through and evaluated levels of physician confidence throughout the process. They conducted the study through an online educational community for practicing physicians and 118 internal medicine physicians from across the country participated.

Physicians in the study were asked to solve five cases online, including a practice case and then two easier cases and two more difficult cases.

For each case, they were first given the patient’s history and asked to list up to three potential diagnoses as well as their confidence for each diagnosis on a scale of zero to 10. Next, they were given physical examination findings and a similar process of recording potential diagnoses and confidence followed. Physicians then were provided the patient’s lab findings and basic imaging test results and asked to answer the same questions. At this point, they also were asked what other resources they would like to use to solve the case, such as more specific tests or physician consultations. Subsequently, physicians were given the results of a final diagnostic test or procedure which could help narrow down the diagnosis. Lastly, they were asked to make their final diagnosis for the case and asked to rate their confidence level on this final diagnosis.

Researchers found that the physicians were more accurate with their diagnoses on the easier cases with 55 percent getting the diagnosis correct. Only 5 percent got the diagnosis correct on the more difficult cases. However, their confidence levels barely differed between easier and more difficult cases. The average confidence level on the easier cases was 7.2 and the average confidence level on the more difficult cases was 6.4.

"With such a big difference in accuracy, you would expect confidence levels to change dramatically, but they barely moved, suggesting the presence of overconfidence in the midst of difficult cases," said Dr. Ashley N.D. Meyer, a postdoctoral health services research fellow at BCM and first author of the paper.

They also found that as their confidence got higher, the physicians were more reluctant to ask for more diagnostic tests and other resources to facilitate diagnosis.

Find a better way

"This gives us good evidence that doctors’ accuracy and confidence are not aligned, so in order to improve diagnosis, we need to find a better way to calibrate or align these concepts," said Meyer.

"This is early evidence from a case vignette study, and other studies are needed in more clinically applicable settings. If additional studies confirm our findings, these concepts should warrant closer attention during medical school and residency. We will need novel educational strategies for improving this alignment," said Singh.

Singh also noted that currently, physicians don’t get feedback on the accuracy of their diagnoses, and it is unclear who will provide them this feedback. This type of information could help them in future diagnoses, especially with those that are inherently more difficult and involve a lot of uncertainty.

Recalibrate

"If we can develop better strategies to provide them feedback, it’s a way for physicians to recalibrate themselves," said Singh.

Singh, Meyer and colleagues would like to now study these relationships in real clinical settings and in real time and explore the use of feedback.

"Right now, we have just begun to define the basic science of this problem," said Singh. "Our study is another foundational step in defining the basic science of diagnostic errors."

Others who took part in the study include Drs. Velma L. Payne, Derek W. Meeks and Radha Rao of the DeBakey VA and BCM.

Investigators were partially supported by the Houston VA Health Services Research and Development Center of Excellence and the VA Office of Academic Affiliations.