Subscribe

Listed By …

False positives?

George Lewith, Michael Dixon and Peter Fisher had a puzzling letter in The Guardian yesterday, defending homeopathy. One of the odd things in the letter was the contention that, “out of six reviews of the scientific evidence carried out by the independent and respected Cochrane Collaboration, two are cautiously positive and four inconclusive.”

This description bears little resemblance to what they actually say. As you read these excerpts from their plain language summaries just remember that these gents are not appealing for more research, but for ongoing funding to treat patients at the UK taxpayer’s expense.

Here is what the six reviews actually say. Try and see if you can spot the two that are, “cautiously positive” (emphasis mine).

“Overall the results of this review found no evidence of effectiveness for homeopathy for the global symptoms, core symptoms or related outcomes of attention deficit/hyperactivity disorder.” (Heirs and Dean, 2007)

“The review of trials found […] that no strong evidence existed that usual forms of homeopathy for asthma are effective […] Until stronger evidence exists for the use of homeopathy in the treatment of asthma, we are unable to make recommendations about homeopathic treatment.” (McCarney et al, 2004)

“The researchers did not find any good quality trials and so cannot say whether it is or is not effective for treating this condition. As no information is available on how much homeopathy is used for dementia, it is difficult to say whether it is important to conduct more trials.” (McCarney et al, 2003)*

“[…] trials demonstrated no differences in any primary or secondary outcome between the treatment and control group […] there was not enough evidence to show the effect of homoeopathy as a method of induction […]” (Smith, 2003)

“[…] Trials do not show that homoeopathic Oscillococcinum can prevent influenza. However, taking homoeopathic Oscillococcinum once you have influenza might shorten the illness, but more research is needed […]” (Vickers and Smith, 2006)

The only one that I think could be called “cautiously positive” is Vickers and Smith (2006)**. All this says is that a specific over-the-counter homeopathic remedy might shorten a bout of the ‘flu by a few hours. That’s it.

I cannot see how Heirs and Dean (2007) can be viewed as anything other than negative; but there again my livelihood and professional reputation are not bound up with the fate of homeopathy.

McCarney et al (2004) says that they cannot justify treating asthma with homeopathy. Smith (2003) shows that there is insufficient evidence to support the use of homeopathy for the induction of labour.

If you are trying to assess whether to actually treat patients with homeopathy – all these reviews say that this would not be supported by the available evidence.

These members of the pro-CAM brigade*** appear to be justifying spending £4M p.a. from the public purse with nothing stronger than: taking a non-individualised homeopathy pill bought from a high-street chemist might shorten a dose of the ‘flu by a few hours. And other things might work if only we could find the evidence! ****

It really is depressing to see a familiar pattern emerge when dealing with literature on Complementary/Alterbative Medicine (CAM) again (and again and again…): you just can’t trust the references, but have to do your own literature search in order to find out what they actually say.
Perhaps it is a sure sign of a pathological science, and one that I think some good books on the subject missed*: an incredibly “optimistic” use of citations and references. (I’m using really nice language here. My cynical self thinks it just shows how much respect the authors have for the scientific method.)
Another part of my cynical self thinks that in the present context “inconclusive” is double-speak for “negative”. Any test that compares an idea (for example: homeopathy) to the null hypothesis (for homeopathy: placebo, regression to the mean,…) has, by definition, only two possible results: according to a pre-defined criterion (the famous p=0.05), the null hypothesis is rejected (the “positive” result) or not. This is not a symmetrical yes/no-test – apart from the fact that an outcome significantly worse than the null hypothesis could be interpreted as posivite, any negative findings can be labelled as “inconclusive”… which is technically correct, since there could be a very small effect that cannot be detected due to the inevitable limitations of studies…
But then, any theory that makes grand claims, but needs to discuss the subtle differences between “negative” and “inconclusive”, or that is looking for “statistical” instead of “clinical” significance is – or rather: should be – in deep trouble.

* (**) just finished reading “Snake Oil Science” from R. Barker Bausell. Very interesting and sometimes thought-provoking book. Thanks for the comments on this site, which made me order the book.
** I really should stop using bogus footnotes in comments.

pleick: Welcome back. I agree, it’s a depressingly regular trend. In the era of the internet it puzzles me that people are so careless – probably hoping that people will not have the inclination to fact-check.

Glad you enjoyed Bausell’s book. I’m also finding the footnote habit is hard to break; I’ve just given in!

The negative vs. inconclusive debate is interesting. There is a perfectly respectable general technical argument which says that failing to reject the null hypothesis is not the same as a ‘negative’ result. However, in the case of interventions with low plausibility I think that there’s a good argument to say if there was no reason to expect that it works (or even good reasons to expect that it doesn’t) and it doesn’t then both of those things taken together are telling you something stronger than ‘inconclusive’. I quite like a comment by made by Fisher in 1935:

“Every experiment may be said to exist only to give the facts the chance of disproving the null hypothesis.”

If it doesn’t then I’m happy to call it negative. I know that it runs the risk of accepting ‘false negatives’, but the biases in CAM trials (as Bausell explains) tend to be in favour of the intervention, so I don’t think that this is a practical issue. I think that is strengthened when the question people like Lewith, Fisher and Dixon are really addressing is not “is this perthaps statistically significantly different from a placebo”, but “is this worth using to treat patients.” This, I’d say, sets the bar higher.

To use a classic example: not finding a unicorn on searching a forest is not at all the same result as not finding a deer!

I never left – I just felt that I didn’t have anything of value to add to your posts on chiropractic. Not that they are an exception to the trend mentioned earlier, of course.
In Germany, chiropractic is not well known – to be honest, I barely knew what it was until the BCA had the brilliant idea to sue Simon Singh for libel. The mother of all CAM practices here is homeopathy. It is not even really alternative, because (along with anthroposophic medicine and phytotherapy) it has a special status in Germanys health care system. I’m not familiar with the details, but homeopathic medicines only need to be recognized by the homeopathic associations to be licensed, scientific evidence is not needed.

I totally agree with your comment on the distinction between inconclusive and negative results. When a clinical study detects no significant difference between placebo and a given therapy, labeling the result as “inconclusive” is technically correct, but misleading in everyday language. Therefore, the term is absolutely ideal to make results that should be called negative in everyday language look much less bad than they are.

gabslagunasaid

Well.. having read this blog..I would like to share my views. I am an internist myself yet I believe in homeopathy as an adjunct to medical management. You see, there are instances wherein a patient’s symptoms cannot be relieved by another pain reliever, even if the ceiling dose is already reached, making dangerous side effects possible. This where homeopathy enters.
I think further studies are needed in this area.
But for now, I am comfortable working on medical managment combined with homeopathic medicine. I know a lot of people love that.
Im also an advocate of PSC Plant Stem Cells Therapy. If you want to know about it, please log on to:

Thank you for sharing your views. I note that you couch them in terms of belief. When you say you believe in homeopathy, do you actually believe that it has specific effects beyond placebo? If so, what convinced you?

The example you give, showing when you would use homeopathy, is essentially the use of a placebo. If, having gone as far as they would wish to with interventions that have specific effects, I have no problems with people trying a placebo therapy of their choice.

However, as a patient I would always want my medical advisors to behave ethically and tell me the truth about any therapy. Personally, I would not want my doctor to really believe that a placebo therapy had specific effects. This would not, in my view, show good judgement. Personally, I prefer my medical practitioners to work on the basis of an honest appraisal of the best evidence.

I wonder, what do you tell your patients about the homeopathic remedies the you advocate? Do you let them know what the evidence for their efficacy really is? Do you tell them what is in the remedy, as opposed to what has been diluted out of it?

Now, what I object to on my blog is the promotion of placebo therapies like homeopathy as if they were effective. As I show in this post, it leads otherwise intelligent and honourable people to misrepresent the evidence. This is something that I have found happens quite alot when people want to believe in particular therapies. And there is plenty of evidence to show that homeopathy is no more than a placebo.

Also, I do get annoyed with the spurious explanations that are often handed out to ‘explain’ therapies like homeopathy. Things like ‘water memory’, ‘vital force’ and ‘like cures like’. How do you feel about these myths?

In the end, I am not very interested in what is popular – some very silly and even dangerous things have been popular – but what actually works. In the end, as I am sure you know, medicine has its limits and at times, even though it pains us, we must face up to this harsh reality.

I do feel that some people (and I’m not saying that you are one of these) try to avoid this unpalatable truth by turning to CAM therapies like homeopathy, convinced that they will work where medicine has failed.

As for your PSC Plant Stem Cells Therapy, this is certainly new to me. I’ll have a look and maybe post my views.