The course of research into so-called alternative medicine (CAM) over the last 20 years has largely followed the same pattern. There was little research into many of the popular CAM modalities, but proponents supported them anyway. We don’t need science, they argued, because we have anecdotes, history, and intuition.

When media attention, which drove public attention, was increasingly paid to CAM then serious scientific research increased. A specific manifestation of this was the National Center for Complementary and Alternative Medicine (NCCAM). CAM proponents then argued that their modalities were legitimate because they were being studied (as if that’s enough). Just you wait until all the positive evidence comes rolling in showing how right we were all along.

But then the evidence started coming in negative. A review of the research funded by NCCAM, for example, found that 10 years and 2.5 billion dollars of research had found no proof for any CAM modality. They must be doing something wrong, Senator Harkin (the NCCAM’s major backer) complained. They engaged in a bit of the kettle defense – they argue that the evidence is positive (by cherry picking, usually preliminary evidence), but when it is pointed out to them that evidence is actually negative they argue that the studies were not done fairly. But then when they are allowed to have studies done their way, but still well-controlled, and they are still negative, they argue that “Western science cannot test my CAM modalities.”

But at the same time, they cannot shake the need for scientific evidence to continue to push their modalities into the mainstream. The “your science can’t test my woo” defense only goes so far. So they have put themselves into a pickle – they demanded funding for CAM research, but now have to deal with the fact that the research is largely negative. CAM proponents are mostly not interested in finding what really works, and abandoning what does not work (I have an open challenge to anyone who can point to a CAM modality that was largely abandoned, rejected, or condemned by CAM proponents because of evidence of lack of efficacy). They are interested in using science to support and promote what they already believe works (a cardinal feature of pseudoscience).

And so they have entered the next phase of CAM research – the post-RCT (randomized controlled trial) phase. They have discovered the “pragmatic” study. You have to give it to them for their cleverness. A pragmatic study is meant to be a comparison of treatment options in real-world conditions. It studies treatments that already have proven efficacy from RCTs to see how they work, and how they compare, when applied in the less controlled environment of real-world practice. Pragmatic studies are useful for addressing the weaknesses of RCTs, mainly their somewhat contrived conditions (having strict inclusion criteria, for example).

But pragmatic studies are not efficacy trials themselves. They cannot be used to determine if a treatment works, because they do not control for variables and they are not blinded, so they are susceptible to placebo and non-specific effects. It is an abuse of the pragmatic study design to test a treatment that is not proven or to make efficacy claims based upon them. That has not stopped CAM proponents from doing just that.

In the news recently is just the latest example. Let me tell you the results of the study before I tell you what the study is of – just look at the data (I replaced the name of the treatment in the tables with just the word “treatment”):

This is what you need to know about the trial. There were two groups, treatment and control. Subjects were those with frequent doctor visits for symptoms that have not been diagnosed. While they were randomized, they were completely unblinded – everyone involved with the subjects knew who was getting treated and who wasn’t. At 26 weeks the control group was crossed over to receive treatment – so the difference up to 26 weeks is most important.

On two of the measures, quality of life and general practitioner consultation rates, there was no difference. In the other two measures there was a slight improvement in the treatment group, barely statistically significant. If this were an efficacy trial, this data would be unconvincing. What can fairly be concluded from this trial is that the treatment has no to minimal effect, and the tiny and inconsistent effect seen cannot be separated from placebo or non-specific effects. Further, it shows how anemic even placebo effects are for this treatment in this patient population. As a pragmatic trial, it’s essentially negative. As an efficacy trial, it’s worthless.

The study author’s, however, concluded that their treatment was effective and recommended it be incorporated into general practice.

The treatment in question is five-element acupuncture – acupuncture designed to balance the five elements of fire, water, metal, earth, and wood. This is the equivalent of balancing the four humors – in other words, it is pre-scientific superstitious nonsense. The study authors are all proponents. What is most amazing is how they got their paper, complete with unsupported conclusions, past peer-review.

The study authors are using the “part of this nutritious breakfast” con. Even as a child I understood that the sugary cereal, or whatever the commercial was trying to sell to me, was only “part of this nutritious breakfast” because the rest of the breakfast was nutritious all by itself. A doughnut would be “part of this nutritious breakfast.” So the authors of this study include five-point acupuncture with an hour of kind attention from a practitioner along with lifestyle advice and encouragement. They admit that they cannot separate out the variables here. Then they conclude that acupuncture is “part of this healthy regimen.” There is every reason to believe that it is an irrelevant part – as irrelevant as the doughnut is to the nutrition of a complete breakfast.

Conclusion

The pragmatic study bait and switch is here – it is now a firm part of the CAM strategy for promoting implausible therapies that don’t work. Editors and peer-reviewers need to be more aware of this scam so they don’t fall for it and inadvertently promote it, as they did in this case. Further, the results of this trial are not impressive even at face value – it’s basically negative, so it’s a double swindle.

All of this has not stopped the headlines from declaring that “acupuncture works” – which appears to be the only goal of such research.

“What is most amazing is how they got their paper, complete with unsupported conclusions, past peer-review”…

It appears that the peer reviewed process has failed in this one instance. A true test of a research paper and it’s finding is whether it can be reproduced by another group of researchers. If it can not, then it’s findings and conclusions are usually not referenced and falls into obscurity.

A true test of a research paper and it’s finding is whether it can be reproduced by another group of researchers.

Weak to non-existent results from an unblinded pragmatic study of accupuncture? Oh yes, I’m confident others can reproduce those results. They may even exceed them, in being even less rigorous in design and even less statistically significant.

Further, it shows how anemic even placebo effects are for this treatment in this patient population. As a pragmatic trial, it’s essentially negative. As an efficacy trial, it’s worthless.

The study’s value may lie in demonstrating to those who might think otherwise that the additional treatment is not, overall, a very cost-effective intervention in this clinical context.

The study does not however preclude significant benefits for some included individuals, who may for some reason be more liable to the non-specific responses to this kind of medical interaction.

It is also an outlier of a study. Most non-blinded controlled trials of acupuncture are more strongly suggestive of benefits from the added medical interaction. So the interesting question is “why not so much with this one?” Is it because this group of patients (undiagnosed conditions) are resistant to ANY kind of medical intervention? Or, is there something in the informed consenting of the subjects that is dampening non-specific responses?

The pragmatic study bait and switch is here – it is now a firm part of the CAM strategy for promoting implausible therapies that don’t work

The proponents of these methods believe they work. Their patients are telling them so all the time. We don’t need to imply sinister intent.

I believe Peter Fisher was genuinely puzzled when he said, after yet another placebo-controlled trial of homeopathy showed no benefit: “Over these years we have come to believe that conventional RCTs [randomised controlled trials] are unlikely to capture the possible benefits of homeopathy . . . . It seems more important to define if homeopathists can genuinely control patients’ symptoms and less relevant to have concerns about whether this is due to a ‘genuine’ effect or to influencing the placebo response”.

Our argument against the use of homeopathy and acupuncture is not based upon any certainty that they cannot contribute to patient well-being.

It is a largely ethical stance that we have chosen to impose on ourselves, for mostly sound reasons, now that we have eaten of the tree of knowledge in a way that others have yet to do, and may never do.

Also, if, as I strongly believe these methods are in practice simply recruiting all the non-specific elements of medical practice, then CAM practitioners have a point — the RCT as usually designed and applied will tend to inhibit “treatment” effects.

pmoran: “Our argument against the use of homeopathy and acupuncture is not based upon any certainty that they cannot contribute to patient well-being. It is a largely ethical stance that we have chosen to impose on ourselves, for mostly sound reasons …”

I assume by this you mean the ethical stance concerning traetment with a known placebo. The problem with your point is that as you note: “… proponents of these methods believe they work.”. In fact these proponents have a religious-like belief in their treatments and exaggerate their effectiveness beyond the placebo effect, to the point that they promote them over truly efficacious treatment regimens.

While you are correct that many proponents may not have “sinister intent” (some certainly do and are just snake oil salesmen) the results of their behavior is the same as if they did have “sinister intent”(encouraging patients away from effective therapies). If I kill a jogger with my car does it matter to the jogger whether I did it intentionally or I was distracted on my call phone? Only one of those is “sinister”, but the jogger is dead either way.

Whenever I get into a discussion of this nature with a woo-person, I often hear that he or she does not care if it is “only a placebo effect” as long as he or she “feels better” after the treatment.

At this point my response is, “perhaps, but why should you hand over $50/hr (or whatever) for a placebo to someone who is claiming that it is the woo, when it is just the placebo effect, unless you have more money than sense?”

This sometimes puts them of a mind to actually listen long enough for me to refer them to this blog, and plants just enough of a seed of doubt that they actually visit. I’ve had two successes recently after a long period of despair that I could reach these people in any way. But maybe it’s just a coincidence.

– “10 years and 2.5 billion dollars of research had found no proof for any CAM modality”
– “What can fairly be concluded from this trial is that the treatment has no to minimal effect”
– “the results of this trial are not impressive even at face value”?

In a word, they are too “polite.” They are filled with weaselly language that almost satarizes the nonsense it is intended to critique.

I think it’s long past the time where the pro-science/evidence community became as rabid and vicious as the clowns who attack Dr Offit and others like him. The thing is, you don’t even need to stoop to the kinds of hyperbole and outright lies they use. You simply need to use plain-talking, straight-shooting, everyday language, that everyone can understand. Language that puts sCAM under the light it deserves.

I think all it takes is more frequent use of a single word: “failure.” It needs to be made clear just how frequently and how reliably sCAM nonsense fails. A simple rephrasing is all that is required.

For example, here are my “translations” of the phrases I noted above:

“10 years and 2.5 billion dollars of research had found no proof for any CAM modality”

becomes

“10 years and 2.5 billion dollars of research failed to find any proof for any CAM modality”

“What can fairly be concluded from this trial is that the treatment has no to minimal effect”

becomes

“What can fairly be concluded from this trial is that the treatment has failed completely or failed almost completely”

“the results of this trial are not impressive even at face value”

becomes

“the results of this trial are essentially complete failure even at face value”

Put simply, every time the word CAM or accupuncture or anything else along that line is used, then the word “failure” should also be used in conjunction with it, strongly and repeatedly, everytime. People need to understand that it’s not a matter of the proof hasn’t yet been found, but rather one failure after another, as certain as the rising sun.

It’s long since time that sCAM became understood as the colossal failure it is.

“A review of the research funded by NCCAM, for example, found that 10 years and 2.5 billion dollars of research had found no proof for any CAM modality.”

…in juxtaposition with:

“But then when they are allowed to have studies done their way, but still well-controlled, and they are still negative, they argue that ‘Western science cannot test my CAM modalities.’ “

…I had a bit of an epiphany. If “Western science” can’t test “CAM”, then why are the US taxpayers spending 2.5 billion dollars over ten years to fund scientific (presumably “Western” science) studies of “CAM”?

Isn’t that rather like saying “You can’t photograph ghosts with a camera.” and then spending millions of dollars setting up cameras to get ghost photos?

Not really; your rational holds if we actually listened to the CAMsters. However, there is some small merit in actual scientific study of CAM modalities for possible (but unlikely) health benefits, and if nothing else then to expand our understanding of nonspecific effects like demand characteristics, bedside manner and different ways of enhancing the placebo effect. So long as we agree to use science and listen to what it says, there’s use in the research.

Of course, one can argue (as probably most people who blog here have already) that the NCCAM is unnecessary because it throws good money after bad, it funds sloppy research that could be better conducted by other agencies, does research on modalities that are highly unlikely and would be better tested in bench research, etc. The real problem is seeking the prestige of science without the rigor, or the willingness to acknowledge that your modality has been disproven and should be abandoned.

The real problem is seeking the prestige of science without the rigor, or the willingness to acknowledge that your modality has been disproven and should be abandoned.

The latter, I think, is the deeper issue. Initially, they’re all for scientific testing of their pet woo, and the more rigorous the better. After all, they already KNOW what the answer is going to be, so the more compelling that result is the better.

Where they get into trouble is when the science doesn’t give the answer that they KNOW is correct. But they KNOW that it’s correct, so the only explanation is that the science was wrong. Then they have to rationalize away the negative result by saying “well, I guess we learned more about the limitations of science.”

Each position is consistent with what they were feeling at the time. But the ultimate result is that CAM is claimed to be scientifically testable precisely to the extent that those tests give the results that the wooists KNOW are right. Results-driven acceptance.

Let’s do a study on how you rational right-brainers actually feel insecure in your Science, resorting to derogatory terms such as “woo” and “they” to refer to those who are more open-minded to centuries-old methods of treatment. What are you afraid of? Look deep inside ~ I know that might be hard to do…

I seems that the only purpose of the NCCAM remaining is to find out if any of the “CAM” modalities actually work so that they can be subsumed into “real” medicine. After all, the “CAM” proponents have pretty much said that they will not accept any scientific results that fail to vindicate their a priori belief that “CAM” is safe and effective.

Given that, it seems wasteful for the NCCAM to research “CAM” modalities that have a low prior probability of efficacy – e.g. homeopathy, acupuncture and any of the “energy field” therapies. Since we’ll never convince the “true believers” that their dogma is wrong, why waste time and money trying?

There may be some “nuggets” of real medicine in the manure pile of “CAM”, but we won’t find them by repetitively demonstrating that homeopathy and acupuncture don’t work.

Bloodletting was literally bleeding you of a litre or so (sometime more!) of blood to try and cure pretty much all that ails you. It was thought to “work” for hundreds (if not thousands) of years. Then that pesky science showed it really didn’t do anything except harm.

So once again, why don’t you advocate for bloodletting as one of the most ancient, long standing, and widely revered methods of healing on record?

Sue, in science the only criteria that matters is evidence. The germ theory of disease could be discarded given sufficient good-quality evidence, it’s just really unlikely. If your criteria is anything but “evidence” (with an asterisk next to it specifying “good evidence” rather than anecdote) then you’re not doing science, you don’t understand science, and you’re missing the point of this blog. Scientists and skeptics don’t hate CAM, they just understand enough about the body to realize it’s extremely unlikely that most of it is effective.

I’m sure that sue did not mean to be derogatory in referring to posters as a “pack of wolves”. Everyone knows that wolves are an integral part of an ecosystem. They are also loving parents and exist peacefully with their family unit in the wild.