The director and deputy director of NCCAM pontificate about “scientific plausibility”

One of the overarching issues, if not the overarching issue that makes so-called “complementary and alternative medicine” (CAM)—or, as it’s now more commonly called, “integrative medicine”—so problematic is prior plausibility. It’s also one of the most difficult to explain to the lay public, because to someone not trained in science it can sound like not being open-minded. I like to joke about this whole concept by saying that it’s good to be open-minded but not so open-minded that your brains fall out. In other words, the main difference between science-based medicine (SBM) and evidence-based medicine (EBM) is how the two deal with this very issue. In EBM, clinical trial evidence, particularly from randomized clinical trials, are the be-all and end-all. I exaggerate, but only a little. Sure, other forms of evidence are present in the EBM hierarchy of evidence, but basic science considerations are relegated to the very lowest rung on the ladder of evidence. In contrast, in SBM, considerations of prior plausibility are much more important, and what determines prior plausibility is the evidence base supporting a therapy, in particular basic science considerations.

One common misconception about SBM is that it requires that we know the mechanism by which a drug or treatment works before we can consider it. In other words, when we reject something like homeopathy as being about as close to completely implausible as it is possible to be, homeopaths will claim that we are saying that, because we don’t know how homeopathy works, we’re dismissing it prematurely. They might have a point if homeopathy actually did, in fact, work, but it doesn’t. Even so, it’s a straw man. When supporters of SBM say that a treatment is implausible on considerations of prior plausibility, we mean something much more specific. Basically, we mean that basic science considerations alone are enough to tell us that the treatment is physically impossible. Take the example of homeopathy again. It’s my favorite example because it’s so easy to use to illustrate my point. Basically, for homeopathy to work, science would have to not just wrong, but spectacularly wrong, about several well-established physical laws and theories in chemistry, physics, and biology. This is far different than saying that something can’t work because we don’t know its mechanism. The same is true of reiki. Or pretty much any form of “energy healing.” Such therapies are so implausible on basic science considerations alone that, absent compelling evidence that they actually work better than placebo (evidence so compelling that it makes us doubt our understandings of physics), it is a waste of resources to do clinical trials.
This brings us to the problem of the National Center for Complementary and Alternative Medicine (NCCAM). It is a highly problematic institution because it is dedicated to studying, in essence, a large number of treatments whose scientific plausibility is highly questionable, while at the same time it also studies therapies that have scientific plausibility, such as diet and exercise, but that have been “rebranded” as somehow “alternative” when in fact they are nothing more than SBM—or should be. The reason I’ve started with this rambling introduction designed for newbie readers of this blog who might not be familiar with these concepts derives from two recent posts on the NCCAM research blog. I tell ya. I ought to send Dr. Josephine Briggs, the director of NCCAM, a thank you note for the amount of blogging material she’s been providing me lately through that blog. This time around, though, it’s both Dr. Briggs and her deputy director, John Killen, Jr., MD providing the blog fodder, Dr. Briggs through a post entitled Our Framework for Research Priorities, which is echoed by her loyal deputy in a post from yesterday entitled On Scientific Plausibility. Together, these two posts suggest that maybe, just maybe, the criticisms that those of us supporting SBM have leveled against CAM in general and NCCAM in particular might be having an effect.

Dr. Briggs cites the latest NCCAM five year strategic plan, a document that I covered extensively when it was published, sarcastically referring to it as saying, “Let’s do some real science for a change!” You think I joke, but I wasn’t. That really is basically what the strategic plan said, admitting that NCCAM funded some crap science in the past (such as studies of homeopathy and distance healing) and, basically promising to do better. The problem is that, after all these years and millions of dollars, NCCAM still wants to have it both ways. It wants to claim to do rigorous science without fully embracing what, exactly, doing rigorous science entails. For example, Dr. Briggs states her post that these are the sorts of questions that NCCAM asks about research grant applications it considers for funding:

These factors, and some examples of questions NCCAM might ask related to each factor, are as follows:

Scientific Promise.
How strong is the body of evidence supporting the concept?

Potential to Change Health Practices.
Is it reasonably likely that the results will make a difference to consumers, providers, or policymakers?

Relationship to Use and Practice.
Do the methods and approaches actually address the most important questions about use or practice in the real world?

It’s very hard not to note that, in terms of criterion #1, very few therapies that fall under the rubric of CAM fare very well. When it comes to CAM like homeopathy and “energy healing,” not only is the clinical evidence incredibly unimpressive, but such therapies are completely implausible from basic science considerations alone. Even more “plausible” CAM therapies, such as herbal remedies and acupuncture rarely have an evidence basis that any reasonable scientist could term “robust”; that is, unless he’s either drunk or messing with you. Indeed, the very term CAM is the name for a double standard. If CAM therapies met criterion #1, they wouldn’t need to be treated as a separate category. As the old cliche goes: What do you call “alternative medicine” that has been shown to work by science? Medicine. I would put a twist on that question and ask: What do you call “alternative medicine” that’s plausible enough from a basic science standpoint to warrant a clinical trial? Experimental medicine. See what I mean? When CAM goes up against real science, it’s like golf being played between a pro and me. Without a massive handicap, I would have no prayer. CAM is basically the duffer of scientific medicine. And what’s adding to CAM’s handicap? Big money institutes like the Samueli Institute and the Bravewell Collaborative and, of course, NCCAM and its $125 million yearly budget.

Criterion #2 might explain why NCCAM has during Briggs’ tenure moving away from the more “out there” CAM therapies, like homeopathy and distance healing. Not only are these sorts of interventions not easily amenable to rigorous scientific inquiry by the sub-criteria listed by Briggs, but in fact, as I’ve written about time and time again on this very blog, CAM practitioners frequently say that they want “rigorous” scientific investigations of their magic nostrums, but when the rubber hits the road they are not nearly as eager as they let on to have their treatments subjected to truly rigorous scientific investigation and randomized clinical trials; hence the preference for “pragmatic” trials among, for example, acupuncturists. These trials are listed as being “real world” and studying “real world efficacy,” but in reality they are putting the cart before the horse. Prove efficacy first in well-designed randomized clinical trials. Then—and only then—look at “real world” effectiveness.

Criteria #3 and #4 are where the true uselessness of Dr. Briggs’ criteria reveal themselves. They are perfectly reasonable criteria for SBM. However, when applied to CAM, not so much. For example, CAM practitioners almost never change their treatments or methods, no matter how much evidence accumulates against them. I mean, really. There are still homeopaths out there! There are still reiki practitioners, who believe that they can channel healing energy from the “universal source” (whatever you do, don’t call it god) by holding their hands in special positions over their clients (whatever you do, don’t call it the laying on of hands). Does Briggs really think that acupuncturists, reiki practitioners, herbalists, practitioners of traditional Chinese medicine, and the like will change their practice if research funded by NCCAM shows that some of their nostrums don’t work? Has CAM ever abandoned a treatment that’s been shown not to work? Heck, there are still quacks out there hawking Laetrile, high dose vitamin C for cancer, and the Gonzalez therapy for cancer, even though these have all been shown quite conclusively to be ineffective.

Dr. Briggs’ post was short, sweet, and to the point, even if it was misguided. Dr. Killen’s post is a bit longer, which means that it gives him ore time to get himself into trouble. It doesn’t take very long at all to admit that a lot of CAM is nonsense but nonetheless argue that was should study it anyway:

Scientific plausibility permeates discussions and debates about research on complementary, alternative, or integrative health approaches. This is no surprise; many interventions that fall under this rubric are ensconced in belief systems about illness and health—some ancient and some modern—that lack foundations in modern science. In addition, those who support research on these approaches often fail to articulate a scientifically grounded rationale or approach to research. Thus, it is common to see criticism based on scientific plausibility in the scientific literature, news stories, and blogs.

No doubt. Count this as one of the blogs and myself as one of the bloggers who have repeatedly leveled this very charge against NCCAM, CAM research, and the very concept of “integrative” medicine (or, as I like to refer to it, “integrating” quackery with SBM). Of course, Dr. Killen appears to be admitting that, yes, a lot of CAM is unscientific and a lot of CAM practitioners and advocates of CAM research can’t defend their woo based on science. I can’t argue with that. I’ve said the very same thing many times. The difference is, of course, that I wasn’t making such statements as a prelude to arguing that, even though CAM is utter pseudoscientific nonsense, we should spend millions of dollars to study it anyway. Dr. Killen is constrained by no such qualms:

This criticism often suggests that the existence of research implies either belief in scientifically implausible explanations or ignorance of basic scientific principles and concepts. So how do we justify investment of public resources in research on complementary interventions that are associated with pre-scientific or unscientific explanations? Simply, it is both possible and necessary to disconnect scientific interest from unscientific “trappings.” For example, an objective look at the body of accumulated evidence (from patient reports, clinical observations by many good clinicians, and clinical studies) suggests that some people with chronic low-back pain are deriving meaningful benefit from acupuncture, yoga, or procedures involving spinal manipulation. It is entirely possible to be scientifically curious about that body of evidence and investigate it further, while not in any way embracing scientifically unfounded explanations for those practices. For instance, it is not necessary to believe in meridians or qi to study the effects of the procedure of acupuncture on pain, or to explore the hypothesis that acupuncture mediates pain by conditioning or expectancy effects produced by a convincing ritual combined with a counter-irritant.

This is a bit of a straw man. We don’t imply by such criticisms (at least I don’t imply by such criticisms) that the researchers studying, for instance, acupuncture believe in qi flowing down meridians, but it’s certain that many, if not most, practitioners “on the ground” do. I’ve encountered them. Let’s put it this way, using Dr. Killen’s own example. If you’re going to study “counter-irritation,” which is not an entirely implausible mechanism by which sticking needles into the skin might lead to a perceived reduction in pain (although, from my perspective, it’s not particularly plausible, either, from a physiologic standpoint), then study counter-irritation, rather than acupuncture. The very structure of acupuncture trials buys into the mystical concepts that underlie it: “controls” that are defined by not sticking the needles in the “right places” along meridians, for instance. If a study of acupuncture were truly studying counter-irritation, it wouldn’t need to be dressed up in the mystical, magical, religious trappings of acupuncture. No, Dr. Killen, these studies are of acupuncture, not counter-irritation, and, worse, rather than asking if acupuncture works, they tend to proceed from the premise that acupuncture does work!

NCCAM can’t do that, of course, because it is those very trappings, the qi, the meridians, and various other prescientific, mystical, and religious concepts that underlie so much CAM that distinguish CAM from SBM. Without them, NCCAM has no reason to exist. There is nothing about “counter-irritation” that requires a special Center in the NIH to oversee studies of it. Without qi and meridians, there’s nothing in acupuncture that can’t be studied by the National Institute of Neurological Disorders and Stroke, which funds a lot of the non-woo chronic pain research supported by the NIH. Similarly, strip away the whole mystical aspect of herbalism, and all that’s left is pharmacognosy, the study of medicines derived from plants and other natural sources. If you take the “innate intelligence” and “subluxations” out of chiropractic, all you’re left with is manipulation of the sort that is better performed by physical therapists. Woo is what makes NCCAM. If you strip the woo from all the CAM therapies whose study NCCAM funds, there’s no reason at all that NCCAM couldn’t be dismantled and its components distributed to the appropriate centers and institutes of the NIH. It’s not for nothing that I half-jokingly have invoked the Internet meme that we should take off and nuke NCCAM from orbit. It’s the only way to be sure.

What amazed me more than anything, though, is an admission that Dr. Killen makes at the end. First, he repeats rather emphatically his straw man that “it is a mistake to assume that scientific inquiry is equivalent to acceptance of unscientific mechanistic thinking,” using this as a prelude to admit:

NCCAM’s first decade entailed a relatively broad and investigator-initiated approach to funding. This was appropriate to the time and the state of the available scientific evidence. The four factors we now consider evolved out of lessons learned during those years. So with the benefit of hindsight, it is pertinent to note that a number of studies funded during that timeframe would probably not be funded today because they could not pass our current hurdles regarding plausibility. In fact, the portfolio of research NCCAM has actually funded over the past several years demonstrates clearly that both the peer review process and NCCAM are now using these factors to shape our investments in research.

I can’t believe that Dr. Killen actually wrote that! Did you see what he just admitted? Basically, he just admitted that many of the studies funded by NCCAM back in the day were, from a scientific perpective, utter crap, just as I’ve been saying all along. He even goes so far as to point out that the studies funded a few years ago, during NCCAM’s first decade of existence as a center, wouldn’t pass muster now. Actually, on the basis of the horrifically tight pay lines all over the NIH alone, I’d bet that a lot of those studies wouldn’t have been funded now even if NCCAM hadn’t made a single change to its strategic focus, just as the first NIH grant I ever got back in 2005 would not have been funded today because the paylines have shrunk so much since then that it wouldn’t have passed muster if I had submitted it today. Be that as it may, Dr. Killen has just confirmed more than I would ever have dreamed possible that I was right on when I described NCCAM’s new strategic plan as “Let’s do some real science for a change!”

Unfortunately for Dr. Killen, his pleas to CAM researchers “to carefully parse rationale from “trappings” and give due recognition to the validity of concerns about scientific plausibility” notwithstanding, he’s in a no-win situation. These trappings are inherent to the mixture of various quackeries that make up CAM. They cannot be so easily “extracted,” to leave plausible scientific hypotheses behind. Moreover, the few CAM modalities that have scientific plausibility, such as nutrition and exercise, could just as easily and more appropriately be studied by other institutes and centers at the NIH.

20 Comments

The NY Times had an article on the new recommendations against testing for ovarian cancer that echo the recent ruckus about routine mammograms in younger women. The blood test seems to produce way too many false positives to be helpful and there is no data to show this test (and the subsequent surgeries) save lives. The article was careful to say this applies to asymptomatic women only.

The comments were filled with anecdotes from hysterical people who obviously didn’t read the whole article (not with any level of comprehension anyway). Moreover, and to the association with this post, they were demanding more research for an effective test to detect ovarian cancer. I’m not sure the budget of NCCAM would do the trick, but it might help. Realistically, I guess its Tom Harkin we have to convince, not the docs at NCCAM, that this money could be better spent elsewhere.

Incidentally, a terrific poster showed up amidst my frantic posting to correct some of the more egregious misconceptions–a cancer researcher using the screen name “metastasi”! He/she echoed a lot of what goes on here, but seemed very youngish by some of his phrasing.

“Moreover, the few CAM modalities that have scientific plausibility, such as nutrition and exercise, could just as easily and more appropriately be studied by other institutes and centers at the NIH.”
This is true, but we must excel in the art of the possible too. If the congresscritters give money to NCCAM, the NCCAM folk could at least funnel it into research that is not so crazy. Serious researchers in nutrition could submit grants to NCCAM, and have some of that money be better spent at least. Perhaps they are too embarrassed to try it. It’s an alternative to nuking, or lobbying politicians who have, er, certain biases shall we say.

I am somewhat curious about NCCAM’s budget as well, and checking Wiki it appears they are aiming for around $130 million 2012. Given that it has existed in current form since 1998, and assuming an averaged layout of $100 million per year that’s $1.4 billion for producing zilch.

Given the high cost associated with the Afforadable Health Care act in the US, surely that cash going to NCCAM could be put towards providing affordable health care that actually works, as opposed to ‘affordable health care’ that does sweet f**k all.

Or barring that, put them under the same rules as other researchers – publish papers and produce results or get lost.

OT, but any plans to talk about the latest meta analysis study on acupuncture that was apparently released yesterday? I saw something about on the network news (Hey, guess what? Acupuncture works!), but the abstract seemed to be making a far more limited claim.

Rork, it’s the congress critters that insist on the crazy, and get upset when the money they shovel to NCCAM doesn’t get the desired results. The NCCAM director is between the rock and the hard place, they can fund reasonable stuff and being ripped for not being “alternative”, or they fund loony stuff and get ripped for not finding it works.

National surveys conducted by the National Center for Health Statistics (NCHS) at the Centers for Disease Control and Prevention show that nearly 40 percent of Americans report using one or more practices such as acupuncture, massage, yoga, meditation, spinal manipulation, dietary supplements or herbal medicines to help manage their health and wellness. Similarly, data show that health care systems and providers are incorporating such interventions. For example, an American Hospital Association survey conducted in 2007 showed that 37 percent of hospitals offered complementary modalities; and a national study reported last year by the NCHS reported widespread availability of complementary approaches in hospice settings. Other data from the Departments of Defense (DOD) and Veterans Affairs (VA) show increasing use of complementary modalities in their populations. According to the VA, 89 percent of their facilities offered complementary therapies in 2011. Both the DOD and VA have integrated complementary modalities into the care of patients with posttraumatic stress and sleep disorders, and to improve treatment of pain.”

Tripe posted by supporters of woo-meisters and scammers (Bolen) is NEVER truly OT @ RI.
And I thank you for that.
-btw- I see he is using the same old, tired epithet when referring to our esteemed and gracious host.

Yep. He is even posting the actual portions of the legal docs….makes me wonder if this reprobate is on their legal team.

I fail to see how any sort of contact with Orac has anythign to do with this other than to try to get another SLAPP handed their way.

What I don’t get is why they’re asking for every donor that he’s got. That list has to be massive. Hell, I’ve communicate with Dr. Barrett a few times over the years so I’m probably going to be in there somewhere.

Orac – I still think #3 and #4 are useful despite CAM practitioners ability to weasel away from negative findings, for two reasons:
1. Insurance claims. It will be harder to “integrate” CAM therapies when there are strong studies showing that they are useless.

2. Studies will change practice if they clearly show harm is caused by the practice. On the other hand, if the harm is already readily apparent no IRB worthy of the name would approve the study, so the window of opportunity is a bit narrow.

What I don’t get is why they’re asking for every donor that he’s got. That list has to be massive. Hell, I’ve communicate with Dr. Barrett a few times over the years so I’m probably going to be in there somewhere.

Indeed. Timmy seems to be even more manic and frothing than usual. He’s claiming that I write my blog from work and implies that my cancer center is paying me to do it. (I don’t and it doesn’t; the most I’ll do is to check my personal e-mail during lunchtime and occasionally moderate a some comments direct from e-mail on my iPad.) He implies that I’m paid off by big pharma. (As I’ve said before, II wish! Show me all this filthy lucre that I can supposedly can get by blogging about quackery.) He claims I’m in some sort of conspiracy with Dr. Barrett. (I’ve exchanged the occasional e-mail with him, but we can hardly be viewed as friends, co-workers, collaborators, or even anything more than acquaintances; I’ve never met him face-to-face.)

Bolen is obviously a Big Pharma plant, because the more that people follow his nonsense, the more they’ll need real medicine later on… 😉

BTW, one argument I find helps when chatting with those tempted by $CAM is the ‘wisdom of crowds’. Notice how the woo-brigade LOVES it when a real doctor (not to be confused with the Toytown College of Mail-Order Joke Certificates (McKeith et al)) crosses into fairyland; they give them prime speaking spots, adulation, etc. because of the genuine – albeit now disgraced – credentials.

“They’re a real doctor, they must know what they’re talking about!”

“Really? Well in that case, here’s another 1000 doctors who say that that one is utterly full of shit. Who’s more likely to be correct?”

I guess another way of looking at it is that the studies showing questionable scientific plausibility should at best be reduced to small pilot studies. Not long ago I encountered* a NCCAM-funded project to the tune of $2M. I’m not familiar with the size of US-based grants but that’s a lot of money for a project that to me seemed to be wielding a nice hammer (fancy MRI equipment and skills) to hit on something that has, at best & kindest, weak support (worse none – i.e. what they’re trying to hit it isn’t likely to be a nail).

(* Not one of my better efforts, I think, it was intentionally understated but I’m not sure that readers would get that.)

Guy Gordon, that is not the main objection to chiropractic interventions. The main objection is thet they do not work better than a placebo. A subsidury objection is that they cannot work in the way they are described, because they are supposed to act on something that does not exist.

Your standard R1 Grant doesn’t have a ‘fixed’ amount but going over $500,000 requires special permission, whilst a R3 pilot study is $50 grand a year with two years max.

Orac can probably give better numbers as he seems to spend a lot of time and effort trying to get the NIH to pony up. And after having to beg and scrape for months to get a grant, and then seeing some nutter get a huge bag of money for researching “bleach and Qi” would make even a advanced computer blow a T-cell.

[…] making a bit of fun of its recent attempts to justify its existence by arguing that it’s now taking scientific plausibility into account. It’s all been enough for me to half-jokingly suggest that we should take off and nuke NCCAM […]