"I am but mad north-north-west: when the wind is southerly I know a hawk from a handsaw." --Hamlet, Act II, scene ii.

Friday, 13 June 2008

Homeopathy works! At least in Mailworld...

There's a terrible, terrible article on homeopathy in the Daily Mail online (cheers to Dr. T on the Bad Science forum for spotting it). The title is "Homeopathy works!", and it really is just about the least critical article I can imagine.

For a start, the fact that homeopathic remedies are generally so diluted that they contain no active ingredient is glossed over with the phrase "Critics argue that the active substance is so diluted that homeopathic remedies have no more effect than placebo or dummy treatment".

The article reports on a study by Dr. David Reilly at, God help us, the Glasgow Homoeopathic Hospital. What do we know about the study? Virtually nothing. According to the article, the study looked at 50 patients suffering from nasal allergies. Some were given homeopathy and some a placebo. I tried to find the study, so I could understand what they did. A search on the pubmed database doesn't find it. There is no mention of the status of the study in the Mail article. Has it been submitted to a journal? Presented at a conference? Written up on the back of a fag packet? We don't know. So there is no way the claims made in the article can be evaluated in any sensible way. Was the study blinded? Did it have adequate randomisation? What was the placebo used? We don't know. The only thing I can say about it is that the study size is small, and so more likely to produce a false positive result.

So, the reporting of the particular study that the article discusses is lamentable. But perhaps even worse is the total failure to place the study in the context of the rest of the scientific evidence. Dr Reilly is allowed to get away with saying that the study replicates some of his previous studies, and that "there were positive findings in 70 per cent of a further 180 clinical trials" of homeopathy.

Perhaps 70% of trials of homeopathy did apparently show a positive result. But this could be a result of poor study designs or insufficient sample sizes. Luckily, this can be tested using a technique called a meta-analysis. Essentially, you get the data from all the published trials, pool it together, and apply statistical techniques to the pooled data to evaluate bias. Such an analysis was conducted by Aijin Shang and co-workers, and published in the Lancet in 2005. This is a study much misunderstood by homeopaths, but what it shows is that large and well-designed trials show no effect for homeopathy, and that positive results of some trials can be explained by poor experimental design, bias and small sample sizes. A previous meta-analysis by Linde et al., published in the Journal of Clinical Epidemiology, came to similar conclusions. Dr. Reillys' previous work on homeopathic treatment for allergies was included in the Shang study, and was not deemed to be of high quality.

So, the current state of the scientific evidence is that there is no plausible mechanism for homeopathy to work, and that the results of large, well-conducted studies are consistent with homeopathic remedies being no more than placeboes. Going by the Mail article, you would get the impression that homeopathy was dismissed by blinkered scientists, but that evidence was gradually stacking up in favour of it. That is simply not the case.

EDIT: It has come to my attention that the Mail story is actually reporting the results of a trial that was published in the BMJ...in 2000. So why it has appeared 8 years later is anyone's guess. This trial was also included in the Shang meta-analysis, and not deemed to be of high quality. This is perhaps because the trial was not double-blind, even though it was described as such in the BMJ [Edit: my mistake. The wash-in period of the trial was single blind. The intervention was double-blind]. The authors also calculated that they needed to recruit 60 subjects in each group (a total of 120) for the study to have the required statistical power. There was a total of 50 subjects. So the only sensible conclusion would be that there was insufficient evidence either way.

Edit 2:Here's the author of the piece, Jenny Hope, giving evidence to the House of Commons Select Committee on Health regarding the influence of drug companies on research and research reporting. Note this bit:

When a story is in the public domain—or being placed with the specific intention of generating publicity—someone will write about it. If it's me than I know where the information is coming from, the background, and carried out the interviews—there is some quality control.

The quality control on this particular story seems to be negligible. Perhaps Ms. Hope was having a bad day?

29 comments:

The Mail have surpassed themselves this week, they really have. Did you see the piece on 'an area of Afghanistan turning cannabis into heroin' on the Bad Science blog?

The Mail piece is terrible, but it does provide a handy illustration of why it (the Mail) is in no way credible as a news source - as well as pointing up some more general problems with the press.

General: The media tends to publish (or re-publish) old stories about old studies, they write about 'a study' showing something when in fact they've used two or three studies and mixed all the findings in order to stitch a story together, and they don't give you either a reference to the paper or sufficient information in order for you to find it yourself.

In this story, the Mail:Claim that 'critics argue' a particular point, but it's a straw man of the Mail's making; misrepresent the study ('study attempts to settle the controversy over homeopathic treatment'? I really don't think that an 8-yr-old study of 50 people somehow supersedes the various meta analyses that have been conducted on homeopathic remedies); point out the tiny level of active substance used in most remedies but neglect to mention the remedies that contain *no* active substance; imply that homeopathy works and may have a plausible mechanism ('it works on the principle that a substance which in large doses will cause the symptoms of an illness can be used in minute doses to relieve the same symptoms'); refer to 70% of 180 trials being positive - without mentioning that the *best quality* trials and reviews are not positive, thereby misleading readers; quote only supporters of homeopathy from the homeopathic hospital and the Faculty of Homs - why no quote from, say, Edzard Ernst?

Basically, the Mail uses logical fallacies, omissions of evidence and spin in order to mislead their readers (or pander to the prejudices of their readers).

Actually, the meta analysis conducted by Linde et al 1997, did not support the hypothesis that reaction to homoepathic remedies were placebo effects. He did, however conduct a later examination of the same material and after, eliminating some trials previously included (new criteria not cited) and conducting a little more number-crunching he was able to be more damning of homeopathy. The meta analysis conducted by Shang was little better - any researcher worth his salt would know that the study was also flawed in its number crunching and exclusion criteria.

As for the criticism that the article was an advertisement for homeopathy - Ben Goldacre of Bad Science fame has been exposed as a stooge for a while now but continues to sound his pro-pharma, anti-CAM trumpet unfettered by journalistic ethics. In the face of this, a somewhat lame pro-homoepathy article really is a drop in the ocean

I have no idea why such inconsequential research was reported eight years on by The Mail. There are hundreds of trials supporting the efficacy of homeopathic treatment dating back to the 1920s. Currently, lots of exciting research is emerging in the field of ultra high dilutions and the ability of these to bring about cell changes which can reverse disease states.

Some examples of positive research: An examination of the impact of homeopathically diluted thyroxine and its slowing effect upon the development of frogs:Graunke H, Endler PC, Scherer-Pongratz W, Spranger H, Frass M, Lothaller H.

Rather than condemning a therapy because, "naw...it don't add up do it?" and gleaning all your information from pharmaceutical company mouthpieces, perhaps it would be prudent to broaden one's sphere and think for oneself.

You wrote Actually, the meta analysis conducted by Linde et al 1997, did not support the hypothesis that reaction to homoepathic remedies were placebo effects.

He did, however conduct a later examination of the same material and after, eliminating some trials previously included (new criteria not cited) and conducting a little more number-crunching he was able to be more damning of homeopathy.

I was talking about Linde et al. 1999, which is the later article you mention. I linked to it in my post, so it would have taken you one mouse click to check that for yourself. What the Linde article I cited did was evaluate the set of homeopathic trials for bias. According to the authors, the evidence of bias "weakens the findings of our originalmeta-analysis". You might disagree with that conclusion, but you need to explain why.

The meta analysis conducted by Shang was little better - any researcher worth his salt would know that the study was also flawed in its number crunching and exclusion criteria.

Rather than condemning a therapy because, "naw...it don't add up do it?" and gleaning all your information from pharmaceutical company mouthpieces, perhaps it would be prudent to broaden one's sphere and think for oneself.

Actually, any researcher worth his salt would read the Shang et al. paper and draw his conclusions from that. I spent quite a lot of time reading that paper in the light of the objections to it from homeopaths. I even wrote a post on it, which you can find here. In other words, rather than simply accepting what homeopaths said about the paper, I thought for myself. I concluded that most of the criticisms were based on fundamental misunderstandings of the paper. Again, you may disagree, but I would suggest that the ball is in your court.

Finally, it's not true that I dismiss homeopathy only because "naw...it don't add up do it?". That is one reason why I dismiss it. The other reason is that the best available evidence shows that it doesn't work.

This is particularly poor. Note that, out of 7 time points in the study, differences between control and treatment groups were significant for only three. This is called cherry picking. I think there should be a correction for multiple comparisons too (i.e. comparison at 7 time intervals), but there is nothing in the paper about this being done. Also note that the treatment group starts out more developed than the control group (at t=1). That this remains the case throughout the experiment is not a surprise.

[If Mojo is reading, I think this study was discussed at JREF somewhere? It's possibly another one of Dana Ullman's favourite useless studies].

This is an odd one. Although the treatment tested claims to be 'homeopathic', it isn't clear that that is the case. Here's what the study says about Relieva, the treatment that was tested:

The active group was treated with a study preparation that contained Reliéva (IGI, Inc, NJ; Canadian Custom Packaging, Canada), a homeopathic product. Its active ingredient is a highly concentrated, proprietary extract of M. aquifolium 10% known as Psorberine, in Novasome, a patented liposome preparation from IGI, Inc, and formulated in an emulsion cream base for topical administration. Matching placebo preparations were prepared with excipients but no active components (IGI, Inc, NJ; Canadian Custom Packaging, Canada).

There is no mention of serial dilution or succussion, and this seems to be more of a herbal remedy than a homeopathic one. If the active ingredient is at a concentration of 10%, then it probably isn't homeopathic.

As for the trial, it seems to have been reasonably well-conducted. One oddity is that subjects that dropped out of the trial were given the 'worst-case' score for Psoriasis Area Severity Index. 3 people dropped out from the treatment group and 27 dropped out from the placebo group. This seems like a potential bias in the results.

Apart from that, psoriasis is a condition in which patients experience flare-ups and remissions throughout their lives. So improvement in trial subjects could be due to this (i.e. regression to the mean), rather than any intervention.

Overall, I don't think that the studies posted by Vicky give me any reason to change my opinion that homeopathy doesn't work.

If any homeopaths out there want to post more studies, please be ready to defend them. It's very time consuming to read them. And also, post links to the studies.

Paul WilsonYou probably know that I am referring to Martin Walker's book (Cultural Dwarfs and Junk Journalism). In it he exposes Goldacre as being in the pay of pharmaceutical industry lobby groups which attempt to undermine all alternative therapies and anyone generally who makes a claim for environmental or Iatragenic cause of disease or who proposes nutritional therapy as a cure. In so doing the future interests of the pharmaceuticals are protected.

This latest wave of anti-homeopathy is just another episode in a 200 year turf war which began with the development of the AMA in America and has seen the pharmaceuticals victorious ever since. Ultimately, its the public who decide though, and the fact is many are turning turning away from orthodox interventions...

It amazes me however that, considering the near-drowning of the NHS and the incidence of Iatrogenic disease ( 1 in 16 hospital admissions and 4% of hospital beds) and in the face of millions of people claiming relief from the therapies like homeopathy and acupuncture, that there are those who seek to deny them that relief.

It is the height of hypocrisy for orthodox medics to the point the finger at homeopathy and other CAM as lacking a scientific basis when currently only 15% of current medical intervention is evidence-based.

My point about Linde is that he says one thing in 1997 and a different thing in 1999. Based presumably on who was paying the piper and how much data dredging he conducted to receive the desired result.

Orthodox medicine is of course invaluable. Lives are saved. Quality of life is improved. Pain is relieved. It is also extremely flawed. And limited. Homeopathy is very effective in psychological aetiology of illness ( an area of which most orthodox medics remain stubbornly ignorant). It provides excellent palliative care for people with long term disease and provides relief in minor injuries and ailments. It is excellent in prevention of disease onset and is light years ahead of conventional medicine in its consideration of the genetic legacy in chronic disease. (Hahnemann identified genetic tendency to certain illnesses whilst orthodox medics were still applying leeches).

This kind of discussions always descends into tedious research pedantry with the patient being forgotten or worse, dismissed as gullible idiots.

The anti CAM movement is based on fear, ignorance and protectionism. Why not let the public receive the therapy they need? Isn't that one of the first tenets of the NHS?

The definition of homeopathic is a substance which is capable of inducing the symptoms in the well which it is trying to cure in the sick. Homeopaths make use of a variety of different potencies of a homeopathic substance. Some are ultra high dilutions (the ones most of you have find most troublesome) and others may not be. The substance Mahonia Aquafolium is homeopathic to dry rough, scaley skin, which begins in small patches (often on the head) and spreads and is relieved in cold water. For this remedy to be applicable, the psoriasis must fit this description.

No, I don't know anything about Martin Walker's book. If it really does accuse Goldacre of being in the pay of "pharmaceutical lobby groups", then I should imagine it's a lot of rubbish. Particularly as those lobby groups aren't getting much value for money, as jqh points out above. In fact, Goldacre has been relatively sympathetic to homeopathy, considering that it might have some use as a placebo therapy, even though the evidence indicates that it lacks specific effects.

My point about Linde is that he says one thing in 1997 and a different thing in 1999. Based presumably on who was paying the piper and how much data dredging he conducted to receive the desired result.

If Linde et al. were working to discredit homeopathy, why did they publish the 1997 paper that suggested it had some effect? The papers say different things because the later one used meta-analytic techniques to evaluate bias, which the previous one did not. Your attempt to impute financial motives for this is nothing more than an ad hominem.

This kind of discussions always descends into tedious research pedantry with the patient being forgotten or worse, dismissed as gullible idiots.

Well, you brought up the papers. But the point of doing the research is to find out whether the patients actually receive any objective benefits. If your case is that homeopathy works, then the research is the evidence for or against that case. So it needs to be discussed.

The anti CAM movement is based on fear, ignorance and protectionism. Why not let the public receive the therapy they need? Isn't that one of the first tenets of the NHS?

My opposition to homeopathy is based on there being no credible evidence that it works. Other than that, I think that people should be able to buy homeopathic medicines if they want. But I don't think public money should be spent on 'therapies' that don't work.

For the record you can download a copy of Walker's `book' here. I wouldn't recommend it, I tried to read it but it is not short and suffers from the random use of fonts/styles that cranks often use. It's, at least, a borderline slanderous piece of crap from my brief experience.

It is so typical that a reasoned debate once more is halted by the closed minded bigotry which allows the goal posts to be shifted so shamelessly.

The sole argument you pose here comes from a meta analysis (Shang et al) which is SERIOUSLY flawed. FAILS to meet basic quorom guidelines. FAILS to be transparent in its reporting and referencing. FAILS to explain why of the 114 trials it considered only 8 form the basis of its findings (we are not even allowed to know WHICH 8 were finally used).

But when I refer to a book which is thoroughly researched and FULLY referenced to highlight the political agenda of an influential writer posing the same arguments as you, I am 'laughed out of the room'.

Q: Is Ben Goldacre affiliated to organisations such as Glaxo Smithkline Wellcome - funders of the British Science Writer's Awards he won in 2003? A: Yes

Q: Is Goldacre also affiliated with the Institute of Psychiatry (funded by the pharmaceutical industry) whose main aim has been to push for patients who claim to have been affected by environmental pollutants (sufferers of ME, CFS and Gulf War Syndrome) to be psychiatrically assessed and declared to be suffering from 'false illness beliefs'A: yes

Q: Is Goldacre the jobbing Junior doctor he claims to beA: No. He is a research scientist at the Maudlsey hospital at the 'Liaison Psychiatry Unit', the aims of which I describe above.

Q: Can we infer from this the possibility of some conflict of interest in his reporting of Homoepathy/CAM and nutritional therapies?A: Yes

It is pointless debating with those who cling so desperately to 'scientism' as a way to avoid truth. I rest assured that the dogma you spout in an attempt to hold back an inevitable tide is as fatuous as my attempting to broaden this discussion into a reasoned consideration of the efficacy of a treatment which I and millions of others have genuinely benefitted from.

I just caught sight of a Victor Hugo quote:"There is no power on earth that can stop an idea whose time has come." How apt.

The sole argument you pose here comes from a meta analysis (Shang et al) which is SERIOUSLY flawed. FAILS to meet basic quorom guidelines. FAILS to be transparent in its reporting and referencing. FAILS to explain why of the 114 trials it considered only 8 form the basis of its findings (we are not even allowed to know WHICH 8 were finally used).

Your interpretation of the meta-analysis is incorrect. See here. (seriously, if you want to argue about the validity of Shang et al, read my post on it first. It's tedious to post the link again and again). And it is not true that 'we are not allowed to know which 8 [studies] were finally used'. Incidentally, Prof Edzard Ernst is putting up £10,000 for anyone who can show that his research findings that homeopathy does not work are incorrect. I say go for it. [But before you say anything about this, check out the Quackometers list of pre-emptive responses].

As for your comments about Ben Goldacre; firstly, this has nothing to do with the scientific evidence on homeopathy. Secondly, your assertions of conflict of interest don't stand up to scrutiny. The sole evidence we get of Dr Goldacre being 'affiliated' with GSK is that he once won a science writing prize that was partly sponsored by GSK. Dr Goldacre seems to be working at the Institute of Psychiatry under a National Institute of Health Research Biomedical Research Centre fellowship, suggesting he is funded by the NHS. There is no evidence he receives any funding from industry. The fact that other people at the Institute of Psychiatry receive industry funding is really immaterial. This seems be an attempt at smearing Dr. Goldacre by association.

You claim that I am clinging to 'scientism' as a way to avoid the truth. You seem to pre-suppose that the 'truth' is that homeopathy works. What I'm actually trying to do is evaluate the scientific evidence in order to decide what the truth is. For me, that evidence is consistent with homeopathy working no better than placebo. So far, you haven't actually done much arguing about the evidence. You've posted three studies which you've failed to defend against criticism, and you've posted some incorrect criticisms of Shang et al. This is hardly a convincing performance.

I wonder what nefarious connections Vicky would acuse Ernst of having? Afterall he's come down against homeopathy being anything other than a placebo - so he must be a bad man, right?

As Paul points out homeopathic aplogists whinge on about Shang et. al. and yet don't seem to have either read the paper or the swift clarifications made after their regrettable omission of some details.

If it is, "SERIOUSLY flawed" what are the specific flaws? As for saying it, "FAILS to be transparent in its reporting and referencing" I'd just refer the curious to the copious details made available:

This seems pretty comprehensive documentation to me. If that's not enough then why not try The (UK) NHS’ National Library for Health “Complementary and Alternative Medicine Specialist Library“ homeopathy evidence. This contains not one reference to a clinically effective intervention.

Not really surprising as the Linde (1997) paper homeopaths like stated that they, "found insufficient evidence from these studies that homoeopathy is clearly efficacious for any single clinical condition."

As for assessing the quality of CAM studies generally, here is some advice from a research methodologist, R. Barker Bausell:

1. Do not rely on journal press releases or second hand accounts2. The article itself should be approached with extreme skepticism (this being a scientific virtue)3. Give more credence to trials published in well known medical journal and no credence at all to those published in CAM journals.4. Give more credence to trials conducted in English and Scandinavian language speaking countries.5. Begin by reading only the study’s abstract. If the results section says that there were no statistically significant (or reliable or clinically significant) differences between the placebo (or sham) group and the treatment group, that says everything consumers need to know about whether they should seek the CAM intervention in question.6. (a)Did the trial employ a randomly assigned placebo-sham control group indistinguishable from the therapy being evaluated?6. (b) Did the trial employ at least 50 participants per group?6. (c) Did 25% or more of the participants drop out of the study before it was over? (If so this will invalidate the study regardless of its results). If even one of these three conditions is not met, the trial is worthless.

Thanks Paul. Accusations of 'scientism' seem to be a substitute for argument/evidence in the minds of some.

It's interesting to note that the charge, if the word is used in the usual perjorative sense, can only be sustained if there is an exaggerated trust in the methods of natural science applied to other areas of investigation.

Here of course we are seeking to apply the methods of science within medical science - and critically evaluating the result. No exaggerated trust or scientism here.

On the list of things that woos don't read, we can add a decent dictionary to Shang et al.

A quick comment on Bernstein et al. It could still be seen by some as homeopathy if it follows the 'law of similars' (like cures like). Ultra-high dilution is not a necessary feature for some of the various 'schools'. For example, Clinical Homeopathy uses potencies varying from D1 (1/10) to C30. This could be described as a D1 remedy.

Of course, members of the other schools (particularly classical or Hahnemannian) sometimes say that there's no such thing as Clinical Homeopathy, usually when a trial of such has not given the 'right' answer. (see here for this kind of critique of Shang et al) Though homeopaths are generally happy with this sort of homeopathy if a trial seems positive.

That Science Writers' Award. If you will indulge us by reading one the litany of excuses from Holford's account of why he did not participate in the Radio 4 programme:"The approach from Radio 4 was quite aggressive and suggested a preconceived agenda to trash nutritional therapy with a highly-biased presenter, who has won numerous awards funded by the big pharmaceuticals."

We strongly suggest that Holford (and Martin Walker) should have a chat with Jerome Burne, his co-author on Food Is Better Medicine Than Drugs. You see, one of those Science Writer awards, one of the ones that is "funded by the big pharmaceuticals", it turns out that Burne wanted one. In 2005 (oddly, at the time when Burne was collaborating with Holford) both Burne and Goldacre were shortlisted for the award and Goldacre won.

I can't really understand the whole thing about Ben Goldacre's science writing prize. It seems to be suggested that Goldacre should note this as a potential conflict of interest every time he writes anything. It's nuts.

Day job...

Fieldwork in Egypt

Welcome to Hawk/Handsaw...

This is the blog of Paul Wilson. You can find science- and pseudoscience-related things here, as well as occasional posts on my current research. There is also some stuff about what I do in my spare time (mostly cycling and complaining about politics).