What can consumers do when they are confronted with confusing or contradictory evidence?

PublishedMonday 18 February 2019

Simply put, in the realm of SCAM, we seem to have two types of people:

those who don’t care a hoot about evidence;

those who try their best to follow the evidence.

The first group is replete with SCAM enthusiasts who make their decisions based purely on habit, emotion, intuition etc. They are beyond my reach, I fear. It is almost exclusively the second group for whom I write this blog.

And that could be relatively easy, if the evidence were always accessible, understandable, straight forward, conclusive and convincing. But sadly, in SCAM (as in most other areas of healthcare), the evidence is full of apparent and real contradictions. In this situation, it is often difficult even for experts to understand what is going on; for lay people this must be immeasurably more confusing. Yet, it is the lay consumers who often will take the decision to use or not use this or that SCAM. They therefore need our help.

What can consumers do when they are confronted with contradictory evidence?

How can they distinguish right from wrong?

Some articles claim that homeopathy works – others say it is just a placebo therapy.

Some experts claim that chiropractic is safe – others say it can do serious harm.

Some articles claim that SCAM-practitioners are competent – others say this is not true.

Some experts claim that SCAM is the future – others stress that it is obsolete.

What can a lay person with no or very little understanding of science do to see through this fog of contradictions?

Let me try to provide consumers with a step by step approach to get closer to the truth by asking a few incisive questions:

WHERE DID YOU READ THE CLAIM? If it was in a newspaper, magazine, website, etc. take it with a pinch of salt (double the dose of salt, if it’s from the Daily Mail).

CAN YOU RETRACE THE CLAIM TO A SCIENTIFIC PAPER? This might challenge you skills as a detective, but it is always well-worth finding the original source of a therapeutic claim in order to judge its credibility. If no good source can be found, I advise caution.

IN WHICH MEDICAL JOURNAL WAS THE CLAIM PUBLISHED? Be aware of the fact that there are dozens of SCAM-journals that would publish virtually any rubbish.

WHO ARE THE AUTHORS OF THE SCIENTIFIC PAPER? It might be difficult for a lay person to evaluate their credibility. But there might be certain pointers; for instance, authors affiliated to a university tend to be more credible than SCAM-practitioners who have no such affiliations or authors working for a lobby-group.

WHAT SORT OF ARTICLE IS THE ORIGINAL SOURCE OF THE CLAIM? Is it a proper experimental study or a mere opinion piece? If possible, try to find a good-quality (perhaps even a Cochrane) review on the subject.

ARE THERE OTHER RESEARCHERS WHO HAVE ARRIVED AT SIMILAR CONCLUSIONS? If the claim is based on just one solitary piece of research or opinion, it clearly weighs less than a consensus of experts.

DO PUBLICATIONS EXIST THAT DISAGREE WITH THE CLAIM? Even if there are several scientific papers from different teams of researchers supporting the claim, it is important to find out whether the claim is shared by all experts in the field.

Eventually, you might get a good impression about the veracity of the claim. But sometimes you also might end up with a bunch of systematic reviews of which several support, while others reject the claim. And all of them could look similarly credible to your untrained eyes. Does that mean your attempt to find the truth of the matter has been frustrated?

Not necessarily!

In this case, you would probably consider the following options:

You could do a simple ‘pea count’; this would tell you whether the majority of reviews is pro or contra the claim. However, this might be your worst bet for arriving at a sound conclusion. The quantity of the evidence usually is far less important than its quality.

If you have no training to judge the quality of a review, you might just go with the most recent and up-to-date review. This, however, would also be fraught with problems, as you can, of course, not be sure that the most recent one is also the least biased assessment.

Perhaps you can somehow get an impression about the respectability of the source. If, for instance, there is a recent Cochrane review, I advise to go with that one.

Look up the profession of the authors of the review. The pope is unlikely to condemn Catholicism; likewise, you will find very few homeopaths who are critical of homeopathy, or chiropractors who are critical of chiropractic, etc. I know this is a very crude ‘last resort’ for replacing an authorative evaluation of the claim. But, if that’s all you have, it is better than nothing. Ask yourself who can normally be trusted more, the SCAM-practitioner or lobbyist who makes a living from the claim or an independent academic who has no such conflict of interest?

If all of this does not help you to decide whether a therapeutic claim is trustworthy or not, my advice has always been to reflect on this: IF IT SOUNDS TOO GOOD TO BE TRUE, IT PROBABLY IS.

15 Responses to What can consumers do when they are confronted with confusing or contradictory evidence?

Surgery for shoulder pain that isn’t caused by an injury has no better results than nonsurgical options or placebo surgery, and comes with potential harms, an expert panel has concluded.

The international panel of clinicians, researchers and patients focused only on rotator cuff disease, as opposed to rotator cuff injuries that result in a tendon tear in the joint, which can require surgery to repair.

Their recommendations are based on two systematic evidence reviews that included seven randomized controlled trials with 1,014 patients, one focusing on the benefits and harms of decompression surgery for rotator cuff disease and the other looking at whether the surgery produced meaningful improvements in pain, movement and quality of life.

So, here, I am talking about those you put in Group 1, “those who don’t care a hot about evidence.”

I don’t agree this is the major group. While there are certainly some who fit in this category—and they can be and are quite vocal because it pays the rent—I think most SCAM users simply wouldn’t know real evidence if it walked up and slapped them in the head.

They are not confused. They are simply in need of treatment and are open to just about anything. They see a SCAM therapy that looks plausible to their (seriously) untrained eye and they jump in. They don’t even know enough to know that there isn’t any evidence to support it. Or, that there is evidence but it disproves it. They simply don’t know any better. Garden-variety ignorance.

Thankfully, while the majority of them may be uneducated about these things, they are not stupid. With enough hard work by skeptics, they might—and I stress might—see the light.

In any event, I would encourage other skeptics to not give up. Everyone we convince, may well be a life we’ve saved.

I agree with DC suggesting Edzard is a ‘cherry-picker’. Many self-proclaimed academics use single words or phrases out of context to suit their own argument. The political equivalent is called a ‘spin-doctor’. Such a juvenile habit is very unhelpful to lay people visiting this blog and adds nothing useful to any topic being debated at the time. Get real Edzard, you are better than that.