The Paradox of Evidence-based Medicine

While many doctors remain enamored with the promise of Big Data or hold their breath in anticipation of the next mega clinical trial, Koka skillfully puts the vagaries of medical progress in their right perspective. More often than not, Koka notes, big changes come from astute observations by little guys with small data sets.

In times past, an alert clinician would make advances using her powers of observation, her five senses (as well as the common one) and, most importantly, her clinical judgment. He would produce a case series of his experiences, and others could try to replicate the findings and judge for themselves.

Today, this is no longer the case. We live in the era of “evidence-based medicine,” or EBM, which began about fifty years ago. Reflecting on the scientific standards that the medical field has progressively imposed on itself over the last few decades, I can make out that demands for better scientific methodology have ratcheted up four levels:

Beginning in the late 1960’s, and then throughout the 1970’s, some began to call attention to the need for better statistical science in research publications. Chief among those emphasizing this problem was Alvan Feinstein. Another important figure was Stanton Glantz, and there were others as well. See, for example, this editorial in Circulation in 1980.

Then, in the late 1980’s and early 1990’s the movement that coined the term EBM emerged, notably from McMaster University in Hamilton, Ontario. Names associated with that movement include David Sackett, Gordon Guyatt, Salim Yusuf, and others. Beyond the proper use of statistics, they demanded that clinical studies be designed properly, and they anointed the randomized double-blind clinical trial as the supreme purveyor of medical evidence. Others, like Ian Chalmers and the folks at theCochrane Collaboration developed methods of “meta-analysis” whereby different trials and studies on a particular question could be analyzed in aggregate.

The mid-to-late 1990’s were the heydays of EBM. I remember the excitement in 1992 when the Evidence-Based Medicine Working Group published its teaching series in JAMAon how to properly interpret the medical literature. I was a senior in medical school at that time, and that series of papers quickly became the go-to reference for self-respecting doctors who wanted to demonstrate that they weren’t taking the conclusions of medical publications at face value. We would be able to judge for ourselves the validity of any new claim! And we began to systematically doubt old claims that were not backed by sufficient evidence. (The autobiography of EBM was recently sketched in JAMA here).

By the late 1990’s, however, it became clear that EBM was having two major effects, and the promotion of clinical judgment was not one of them. On the one hand, EBM gave rise to “guideline medicine” and cookbook recommendations that would soon provide insurers and government agencies a method to gauge “quality of care” on a large scale and to tie performance to that quality. On the other hand, the methodology also played into the hands of large pharmaceutical companies who, through the implementation of large clinical trials, could now identify small effects that physicians would feel compelled to apply to large populations of patients.

The former effect was not so problematic for the promoters of EBM who, by and large, were made up of “systems experts” and outcomes researchers for whom the “rational” provision of care by way of clinical guidelines and “appropriate use criteria” seemed like a great boon.

But the latter effect, i.e., the benefits the pharmaceutical and medical device industry reaped through the use of the mega-trial, was a tough pill to swallow. Many in the EBM movement were not particularly thrilled to see their pet methodology be at the service of major corporations, but they were hard-pressed to be able to criticize them on that basis.

An opportunity to correct that trend came when the Vioxx scandal brought to light the fact that too cozy a relationship between scientists and corporations could lead to tampering of the precious evidence. It would not be long before the next level of demand on clinical science would be made: in addition to being proficient in the use of statistics and in the designs of clinical studies, scientists would now also have to disclose any potential conflicts of interest, i.e., any financial relationship with a sponsoring corporation. Soon the sun would shine on the greed of doctors and scientists and, it was hoped, the truth would finally emerge victorious.

But even that has failed to satisfy the skeptics. After all, conflict of interest is undetectable and unavoidable. And the emphasis on conflicts of interest seems to have poisoned the atmosphere. Witness any debate between scientists on Twitter, and you will experience a new version of Godwin’s law:

The latest remedy concocted to deal with the vexing problem of scientific truth is data sharing, which I mentioned in a recent post. Soon, scientists may need to make all data forms, correspondence, and even their inner reflections, available to third parties on demand, so that, you know, we can finally be sure that the results are “true.”

I predict that statistics, optimal study designs, conflict of interest disclosures, and data sharing will never satisfy our yearnings for clinical truth. In recent days, I have been particularly struck by the epistemiological crisis in which we seem to find ourselves, despite the fact that we have access to huge data sets of information. There is no longer any medical fact or truth that can be considered reliable anymore. Skepticism is rampant and everything is in doubt, which is strange since there has never been more attention paid to being meticulous about science and methods.

It’s like a paradox: the more we insist on scientific reliability, the less certain our knowledge seems to become. As I mentioned before, I believe this paradox arises in part because we are bent on applying to medicine quantitative methods that are more suited for the study of falling stones and quarks than for the proper understanding of human beings. Randomized trials provide us with general rules, but the business of medicine is to particularize. When the time comes to apply our knowledge at the bedside, the evolution of EBM seems like a devolution against sound clinical judgment.

Also, quantitative methods have taken prominence because medicine has ceased to be a private affair between doctors and patients, but has turned instead into a public health endeavor. Yet patients still expect us to treat them as individuals, and most doctors hope to practice on a human scale in the context of small communities. The focus on big data and big methods is a major distraction, and it has yet to convince me that we are better doctors as a result of it.

In his magnificent essay, Koka notes

I realize it has become dangerous to use one’s clinical experience to inform one’s views….Forcing all progress through the funnel of a million strong clinical trial is a bar too high, a bar that protects us from medicine itself. No one benefits from that.

I agree with my colleague Koka. It’s time to return to the small data set. It’s time to reclaim our ability to think, judge and reason for ourselves. It’s time to put an end to the EBM madness.

Michel Accad practices cardiology in San Francisco and blogs at alertandoriented.comwhere this post originally appeared. He can be followed on Twitter @michelaccad

Leave a Reply

We have a lot of problem with the use of quasi-experimental design models used for EBM. While we can’t get away from quasi-experimental designs, we can do a much better job articulating their bias and weakness. Both business interest and human ego bias us to make an EBM argument when chance may be the reality.

- Share

Member

LeoHolmMD

Mar 15, 2016

EBM was designed to support clinical judgement, not replace it. There are large differences between the clinical study environment and reality. Trials study cohorts, clinicians study the person in front of them. Great discussion.

“A culture of denial subverts the health care system from its foundation. The foundation—the basis for deciding what care each patient individually needs—is connecting patient data to medical knowledge. That foundation, and the processes of care resting upon it, are built by the fallible minds of physicians. A new,secure foundation requires two elements external to the mind: electronic information tools and standards of care for managing clinical information.

Electronic information tools are now widely discussed, but the tools depend on standards of care that are still widely ignored. The necessary standards for managing clinical information are analogous to accounting standards for managing financial information. If businesses were permitted to operate without accounting standards, the entire economy would be crippled. That is the condition in which the $2½ trillion U.S. health care system finds itself—crippled by lack of standards of care for managing clinical information. The system persists in a state of denial about the disorder that our own minds create, and that the missing standards of care would expose.

This pervasive disorder begins at the system’s foundation. Contrary to what the public is asked to believe, physicians are not educated to connect patient data with medical knowledge safely and effectively. Rather than building that secure foundation for decisions, physicians are educated to do the opposite—to rely on personal knowledge and judgment—in denial of the need for external standards and tools. Medical decision making thus lacks the order, transparency and power that enforcing external standards and tools would bring about…”

– Lawrence and Lincoln Weed, MD and JD, “Medicine in Denial,” pp. 1-2

Larry Weed is regarded as the “father of the SOAP note” and the “POMR” (Problem Oriented Medical Record). The 2011 book comprises 267 pages of exquisitely argued butt-whup.

“I have mostly found mainstream health care professionals to be a dedicated, unpretentious, and self-deprecating lot quite aware of the limits of their knowledge and the risks of presumption. Once, during a series of health care quality improvement seminars I attended at Intermountain Health Care in Salt Lake City during my Peer Review tenure, a speaker– himself a noted pediatric surgeon– wryly observed that “the best place to hide a hundred dollar bill from a doctor is inside a book.” The Director of the seminar series, Dr. Brent James of IHC (and a Fellow of the Harvard School of Public Health), noted in our opening session that physicians would probably admit– off the record, of course– that perhaps only 10% of their clinical decisions made during daily practice could be traced to the peer-reviewed scientific literature. Dr. James also made the droll observation that, were you to walk into the typical medical adminstrator’s office, “you’d be much more likely to see copies of the Wall Street Journal rather than the New England Journal strewn about.”

What can one take away from such remarks? First, the many physicians I have come to know in the past few years are in the main acutely sensitive to the problems of clinical conceit and “paradigm blinders.” Indeed, the Utah pediatrician’s”$100 bill” wisecrack was offered to an audience of doctors and their allied health personnel during quality improvement training. Second, the body of peer-reviewed medical literature does not constitute a clinical cookbook; even “proven” therapies– particularly those employed against cancers– are generally incremental in effect and sometimes maddeningly transitory in nature. The sheer numbers of often fleeting causal variables to be accounted for in bioscience make the applied Newtonian physics that safely lifts and lands the 747 and the space shuttle seem child’s play by comparison. Astute clinical intuition is a necessary component of a medical art that must, after all, act and act quickly– so often in the face of indeterminate, inapplicable, or contradictory research findings…”

Accounting standards utilize numbers and mathematics that are clear and reproducible. 2-1=1 The second they move from pure numbers to a matter of interpretation or weighting different variables we can be left with an ENRON.

I am not arguing against any position rather I am looking at how one makes medicine into the number system we see in accounting while avoiding the ENRON’s.

“ perhaps only 10% of their clinical decisions made during daily practice could be traced to the peer-reviewed scientific literature. ”

Though I have no idea of the % I will freely admit to the problem you suggest above. I think I could spend my entire set of working hours reading the peer-reviewed scientific literature for those clinical decisions without ever finishing and still not have time to actually closely read the literature you talk about.

- Share

Member

Perry

Mar 16, 2016

” Astute clinical intuition is a necessary component of a medical art that must, after all, act and act quickly– so often in the face of indeterminate, inapplicable, or contradictory research findings…”

This is the crux of being a physician to me. Just like a seasoned homicide detective, physicians have to be able to rely on a 6th sense and astute observation of the patient to help make the right diagnosis or treatment plan. Today’s EMR, tech heavy world takes much of that away.

We frequently accept crossposts from smaller blogs and major U.S. and International publications. You'll need syndication rights. Email a link to your submission.

WHAT WE'RE LOOKING FOR

Op-eds. Crossposts. Columns. Great ideas for improving the health care system. Pitches for healthcare-focused startups and business.Write ups of original research. Reviews of new healthcare products and startups. Data-driven analysis of health care trends. Policy proposals. E-mail us a copy of your piece in the body of your email or as a Google Doc. No phone calls please!

THCB PRESS

Healthcare focused e-books and videos for distribution via THCB and other channels like Amazon and Smashwords. Want to get involved? Send us a note telling us what you have in mind.
Proposals should be no more than one page in length.

HEALTH SYSTEM $#@!!!
If you've healthcare professional or consumer and have had a recent experience with the U.S. health care system, either for good or bad, that you want the world to know about, tell us about it. Have a good health care story you think we should know about? Send story ideas and tips to [email protected].