"I am but mad north-north-west: when the wind is southerly I know a hawk from a handsaw." --Hamlet, Act II, scene ii.

Friday, 30 November 2007

What's wrong with Shang et al.?

Shang et al. recently published a meta-analysis comparing homeopathy with 'conventional' treatments in the Lancet (Lancet 366; 726-732). This is the most recent of the meta-analyses cited by Ben Goldacre in his excellent Guardian piece on homeopathy. Homeopaths have not been amused by this paper, and have tried very hard to discredit it. This is perhaps predictable since the paper concludes that homeopathic treatments have no effect beyond placebo, once the poorly-designed and biased studies have been stripped out. The consensus amongst homeopaths seems to be that the study is outrageously flawed, and scientists show themselves up by referring to it. Some recent discussions of it involving homeopaths can be found here and here. Despite the large amount of noise surrounding the discussion of this paper, I haven't seen a lot of signal, and so far I haven't seen a critique that stands up to scrutiny.

What did the paper do? The authors set out to test the hypothesis that homeopathic treatment effects can be attributed to the placebo effect. If that were the case, then any positive trial results for homeopathy would have to result from poor study design and/or bias. The authors tested this proposition by identifying 105 papers reporting 110 trials of homeopathy, and matching them with 110 trials of 'allopathy', or conventional treatments, on the basis of disorder treated and type of outcome measured. The authors then assessed the methodological quality of the papers, based on factors such as whether the trial was adequately blinded and whether it was adquately randomised. The paper found that for all the homeopathic trials, there was an effect beyond placebo. However, when the trials that were of low methodological quality and/or had sample sizes that were small were stripped out of the analysis, the remaining 8 trials showed no effect beyond placebo. On the other hand, when the same procedure was followed for the conventional medicine trials, the six remaining trials did show an effect beyond placebo.

So far, so good. What have been the criticisms of the paper?

One criticism has been that the trials deemed to be large and of higher quality were not identified, and that the reporting of the meta-analysis was inadequate. This criticism does carry some weight, and the reporting in the original paper was not good enough. However, the authors recognised the problem, and rectified it by identifying the trials in a reply to published criticisms that appeared in the Lancet (Lancet 366: 2083). You can find all the details of the study via apgaylard's blog here. So, this criticism is no longer valid.

Another criticism has been that the meta-analysis only uses 8 papers out of 105 to conclude that homeopathic remedies are no better than placebo. This seems to totally miss the point of the study. For one thing, it's a meta-analysis, so it pools studies in order to get more statistically significant results than single studies. The eight studies of homeopathy have a total n of 1,923, which is quite respectable. Also, Shang et al. have not employed some sort of sleight of hand to dismiss the other 97 papers. They have filtered them out because they are of inadequate methodological quality and/or size, based on clearly stated criteria. This allows the authors to compare the results from all the studies with the results from the best studies. When you use only the best studies, there is no longer any benefit for homeopathy beyond placebo. In contrast, using the best studies of conventional treatments, there is an effect beyond placebo. Again, this is the whole point of the study, and criticising it on the basis that it seeks to use the best-quality studies seems somewhat misguided.

Another common criticism from homeopaths is that the study doesn't test 'real' homeopathy. Shang et al. split studies of homeopathy into four types:

1. Classical homeopathy: individualised treatment based on homeopathic history-taking2. Clinical homeopathy: no history-taking involved, each patient gets the same remedy3. Complex homeopathy: patients take a mixture of several different remedies4. Isopathy: the agent judged to be the cause of the disorder was used

For example, here's a website where they state flat out that there is no such thing as clinical homeopathy. This would be news to anyone who has wandered into Boots and seen the homeopathic remedies on sale there. More commonly, the criticism is that only 'classical homeopathy' is really homeopathy, and the other types don't count. Even if we allow this criticism, the fact is that 18 of the included trials were of 'classical homeopathy', as defined by the authors, and two of those made it into the group of eight large, high quality trials. The statistical analysis also showed that there was little evidence that effects differed between different types of homeopathy. So, not only did the study include trials of individualised homeopathy, it showed that these were no more effective than the other forms of homeopathy.

So, on the whole, it seems to me that the methodology of Shang et al. is reasonable, and the conclusions justified. I think it's probably true that no study is entirely without flaws, and I'm willing to be corrected on this. But so far I've seen no good criticism of the Shang et al. study that invalidates its conclusions.

Edit: Just as an aside it's interesting to read the second last paragraph of Shang et al., where they discuss the place of homeopathy in treatment systems. I take the liberty of reproducing the paragraph below:

"We emphasise that our study, and the trials we examined, exclusively addressed the narrow question of whether homoeopathic remedies have specific effects. Context effects can influence the effects of interventions, and the relationship between patient and carer might be an important pathway mediating such effects. Practitioners of homoeopathy can form powerful alliances with their patients, because patients and carers commonly share strong beliefs about the treatment’s effectiveness, and other cultural beliefs, which might be both empowering and restorative. For some people, therefore, homoeopathy could be another tool that complements conventional medicine, whereas others might see it as purposeful and antiscientific deception of patients, which has no place in modern health care. Clearly, rather than doing further placebo-controlled trials of homoeopathy, future research efforts should focus on the nature of context effects and on the place of homoeopathy in health-care systems."

This seems to be entirely reasonable, and suggests that Shang et al. have no particular bias against homeopathy.

16 comments:

Excellent post. Good comment on what constitutes homeopathy. The definition that the homeopaths I have read seem to follow is if the trial result is positive then, and only then, is it homeopathy.

A nice example are Ullman's comments on "Oscillococcinum". This is a proprietary homeopathic 'medicine' that was given to people with "... influenza and influenza-like syndrome ...". No individualisation.

One of the (many) recorded outcomes was marginally significant; so he calls it effective.

Paul Wilson’s defense of the Shang “comparison” doesn’t answer several important questions:

Shang self-selected the 110 conventional medical studies. Because Shang and Egger (his co-author) are known antagonists to homeopathy, how can anyone feel confident that they didn’t choose certain conventional trials that showed particularly positive results? Ironically, the 110 conventional medical trials had only 9 that Shang defined to be of a “high quality.” Because Shang found 21 of the 110 homeopathic trials to be a “high quality,” this finding alone suggests that TWICE as many homeopathic studies were designed and conducted to a higher level of scientific investigation. Because Shang choose to generalize his conclusion from 6 conventional trials and 8 homeopathic trials, how come one of Shang’s conclusion wasn’t that homeopathic investigations are substantially more scientifically based?

The most glaring error of the Shang analysis (and one that Paul Wilson purposefully chose to ignore, despite the fact that many homeopaths, including myself, have persistently pointed it out) is what was Shang’s results of comparing the 21 high quality homeopathic and the 9 conventional ones? Curious minds want to know. It is also intriguing to know that apgaylard highlighted that he uncovered the “missing 8 trials,” but curiously enough, he too completely ignores ask what were the results of the high quality 21 and 9 trials. The fact that Paul and apgaylard are obviously smart and scientifically-oriented people, their omission seems to be pre-meditated and purposefully secretive (not exactly good science at work).

Why did Shang’s definition of large enough trials happen to include Wallach’s trial of 98, but not several larger trials that tested, for example, Oscillococcinum (there have been 3 trials testing this medicine in the treatment of influenza, but he only included one of the these 3 trials). Shang also only selected 1 of the 3 trials that Jacobs conducted on childhood diarrhea, and he didn’t include the pooled data from these 3 trials that have been published in a major pediatric journal…and he didn’t include her high quality study that was published in PEDIATRICS.

Just selecting the largest trials isn’t always the best way to evaluate whether homeopathy worked or not. While it is certainly true that sometimes there is one single medicine that can be effective in treating a specific ailment, homeopaths find that this is an exception, not the rule. For instance, Oscillococcinum seems to be effective in the TREATMENT of the flu, but not in its prevention. One of these large trials that was a part of the “magic 8” was one that tested one medicine (potentized thyroid hormone) for weight-loss. The fact that it didn’t work didn’t surprise anyone I know…and the negative result doesn’t disprove anything about the system of homeopathy.

There are many other sharp criticisms that I can give of the Shang article and of Wilson’s defense of it, but the above is simply a good place to start.

The bottomline here is that skeptics of homeopathy tend to be extremely critical of various homeopathic studies, but whenever “junk science” comparison articles, like Shang’s, suggest a negative result to homeopathy, the skeptics seem to forget their high scientific standards

I personally think that skeptics and advocates of homeopathy should join together to sharply condemn the Lancet for publishing this piece of junk science. Skeptics and advocates of homeopathy should join together to encourage high quality scientific investigations.

What an odd comment from Mr Ullman. I don’t claim to have “… uncovered the “missing 8 trials,”; I have just pointed out that they have been clearly identified in the open literature and on the web for nearly two years. I remain interested in why this development seems to have passed many homeopaths by.

The group of 21 “high quality” homeopathic trials that Ullman wants to talk about are, in fact, identified in my piece (they are listed in table 1 and my graphic).

It is true that I didn’t discuss the results of these trials. I didn’t discuss the results of any trials in this piece: I was quite clearly addressing the narrow question of the ease with which trials some homeopaths still contend cannot be identified can actually be identified.

Given Ullman’s concern about what the 21/9 comparison would show, it’s a surprise to me that he hasn’t done it himself. The identity of these trials can be found in the pages of the Lancet and on the web; if “… Curious minds want to know …” why don’t they do the work?”

The suggestion that not discussing the results of these trials is “…pre-meditated and purposefully secretive (not exactly good science at work)… " is only half true. Of course I chose, in a pre-meditated manner, what I was going to discuss. Doesn’t everyone? The accusation of secrecy is bizarre. If anyone really wants to find out what the results of individual trials are they can. They could even click on a couple of the links in my piece to get started.

I provide links to Shang et al.’s original paper, critical letters published in the Lancet, Shang et al.’s reply and links to the following documents on the ISPM website: List of excluded homoeopathy studies; Study characteristics of homoeopathy studies; Study characteristics of allopathy studies.

“… purposefully secretive ...”? Surely you’re joking Mr Ullman?

Anyway, perhaps Ullman (a.k.a. James Gulley) would now like to reconsider the statement he made on 29th November on a JRF Forum concerning the 21 "high quality" studies of homeopathy selected by Shang et al.:

“… Perhaps, SOMEONE can finally tell us which were the 21 homeopathic trials and the 9 allopathic ones. Shang NEVER divulged, most likely because this review would show real benefits from homeopathic treatment. Isn't anyone suspicious of "black box" comparison studies like this? Why are only the homeopaths complaining here about junk science? Hmmmm. …”

My piece very clearly does this; and that’s only because I actually read the Lancet correspondence.

As long as we are talking about answering questions I see that Ullman still has not provided an answer to the challenge set on that thread: “GIVE ONE, YOU ONLY NEED ONE, INCONTROVERTIBLE EXAMPLE, WITH REFERENCES, OF HOMEOPATHY CURING A NON-SELF-LIMITING CONDITION”.

Now I think about it neither has he talked about the criticisms of the Rao et al. paper he was trumpeting rather strongly?

He also left us hanging on this blog on his daft assertion that silica has a tendency to store and broadcast information.

My curious mind would really like to see these topics addressed.

Finally as for a more detailed disussion of Shang et al.this JREF forum thread is probably a better venue.

apgaylard said: "I have just pointed out that they have been clearly identified in the open literature and on the web for nearly two years. I remain interested in why this development seems to have passed many homeopaths by."

Well, the germ theory of disease and the existence of atoms seem to have passed them by as well, and those have been in the public domain for rather longer than two years.

Dana states that "Shang self-selected the 110 conventional medical studies". In fact, in the original paper the authors state that:

"For each homoeopathy trial, we identified matching trials of conventional medicine thatenrolled patients with similar disorders and assessedsimilar outcomes. We used computer-generatedrandom numbers to select one from several eligible trials of conventional medicine. Outcomes were selected and trials matched without knowledge of trial results." So, unless Dana wants to claim that the authors are lying, the criticism is invalid.

Dana also talks about the finding that there were fewer trials deemed to be of good quality in the group of 110 conventional trials. This is an interesting finding, and might suggest that homeopathic trials were better than conventional trials. Then again, it might not, as the 110 conventional trials are a small part of the published literature, whereas the 110 trials of homeopathy represent every published trial that matched the inclusion criteria.

You say that the 'most glaring error' in Shang et al. is that they didn't compare the 21 homeopathic trials and 9 conventional trials that were deemed to be of higher quality. Sine this group of trials was shown to be biased, there seems to be little point in comparing them. Still, as apgaylard has pointed out, the information is available if you're really interested.

As for the inclusion criteria, these are stated in the original paper, where the authors write that "The commonest reasons for exclusion were insufficient information (precluding the calculation of odds ratios), ineligible study design, multiple publication, and inability to identify a matching trial of conventional medicine." Since you don't give full references for the studies, it's difficult to tell, but it's probably worth checking out the list of excluded studies here. If the studies were not in the original 165 identified potentially eligible studies, it may be because the studies were not indexed in the databases used, or that they were not found using the keywords in the study.

Dana also says that "Just selecting the largest trials isn’t always the best way to evaluate whether homeopathy worked or not". But the point of selecting the larger trials was to look at the effect this would have on the statistical benefit of the trials. In homeopathy, there was no effect. In conventional medicine, there was. This was the whole point of the study: to see what effect potential biases had in homeopathy vs. conventional treatment.

Dana calls the Shang et al. study 'junk science', but as yet has failed to show that this is the case. Meanwhile, he is silent on the Rao et al. study on memory of water that he has touted previously, but which has since been shown to be extraordinarily flawed. Despite this, Dana accuses skeptics of ignoring work they don't like.

Mojo:Fair point. However, some homeopaths argue that such things are not in their 'paradigm' (daft, I know). I wasn't aware that reading and counting were held to suffer from similar incommensurability.

dullman:I've some comments on the four treatment trials using homoeopathic Oscillococcinum covered in the Cochrane Reviewhere and here.

As for the Jacobs studies: all three were selected as part of the initial 110 by Shang et al. Jacobs (48) was deemed not to be of high quality. Jacobs (49) was the ninth largest study, so didn’t make it into the final eight.

It baffles me why homeopaths and their apologists are so hung up on the inclusion of small/low quality studies.

Cheers for these links. I had seen your demolition of the oscillococcinum 'literature': excellent stuff. I would like to think that Dana Ullman would read it, but even if he does I'm sure it won't prevent him touting the studies around as evidence for homeopathy. He seems very fond of a study in Chest that was taken apart here. Still, he does seem to have shut up about the Rao et al. water memory study, for some reason...

Congrats on fighting up the unbelievably tedious Dullman, Paul and apg, though no doubt he will be back. I expect he's gone to annoy LCN over at the Quackometer

I always smile when anyone mentions the staggeringly ludicrous Oscillococcinum, aka "The Twenty million dollar duck remedy" (Reqd: one dead duck. Global sales $ 20 million. Ker-chingg). It is so truly mad that Oscillococcinum is a textbook demonstration of placebo power all on its own.

What struck me about the Cochrane review of Oscillo was that the 95% CI for "reduction of symptoms" almost overlapped zero days:

"Oscillococcinum treatment reduced the length of influenza illness by 0.28 days (95% CI 0.50 to 0.06)."

Hence all those jokes about your 'flu getting better on its own in seven days, or in a week with homeopathic treatment.

I remember Peter Fisher was quoting the Cochrane meta-analysis when he debated Ben Goldacre at the Science Museum, which in turn made me doubt Fisher's sanity.

Amused that D.Ullman has used "James Gully" as his alias on the JREF. Though given D.Ullman's self-importance and sense of himself as a Martyrdom-seeking Warrior for Truth, I'm only surprised he didn't go to the Motherlode and call himself Hahnemann. Perhaps some other homeopathic blusterer got there first.

I note that the other link you give is to Martin Chaplin, who despite being an academic scientist seems to have followed Rustum Roy to The Darkside and made a career out of giving homeopathic WaterWoo a misleading veneer of science. So his inability to interpret the meta-analysis accurately is not surprise.

I thought that Chaplin's statement that "It should also be noted that placebo effects constitute real clinical effects [121c], should be judged positively and probably account for a significant proportion of the success of prevailing established medicine" was particularly bizarre. The point is that you can't distinguish the effect of a homeopathic 'remedy' from that of a sugar pill. So why should it be 'judged positively'?

Day job...

Fieldwork in Egypt

Welcome to Hawk/Handsaw...

This is the blog of Paul Wilson. You can find science- and pseudoscience-related things here, as well as occasional posts on my current research. There is also some stuff about what I do in my spare time (mostly cycling and complaining about politics).