"I am but mad north-north-west: when the wind is southerly I know a hawk from a handsaw." --Hamlet, Act II, scene ii.

Wednesday, 8 October 2008

Shang study remains firmly in the water

In the comments to a post at Respectful Insolence, my favourite homeopath Dana Ullman weighs in with the suggestion that the Shang et al. meta-analysis of trials of homeopathy and conventional medicine (which has been written about extensively by me and apgaylard), had been "blown out of the water". Ullman makes this assertion based on a new paper by Ludtke and Rutten, entitled "The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials", that has been accepted by the Journal of Clinical Epidemiology. It's nice to see that this paper does exist after all. So does the article really blow Shang out of the water? A quick look at the conclusions tells us that the answer is no:

Our results do neither prove that homeopathic medicines are superior to placebo nor do they prove the opposite...it occurs that Shang’s conclusions are not so definite as they have been reported and discussed.

What Ludtke and Rutten have done is actually quite interesting, though, so I'd like to explore it in a bit more detail. What they've done is taken the 21 trials of homeopathy that Shang et al. considered to be of "higher quality". They have then performed various analyses on this subset of trials to see what happens if you exclude trials based on their size or other parameters.

The authors plotted a funnel plot of odds ratio (a measure of the size of the effect of the intervention: values below 1 indicate a benefit over placebo) versus standard error (which is dependent on trial size). For all of the 21 trials, they found that there was substantial (but not statistically significant) asymmetry in the plot (if the funnel plot is asymmetrical, then biases are present in the data: these might be publication bias, small study effects, or a whole host of other effects). They also note that no evidence for asymmetry was found for the 8 largest trials of the 21. This actually re-iterates one of the main points of the Shang analysis: that a large number of trials is needed to identify asymmetry, and therefore to get an idea of bias in the dataset. That is why Shang et al. looked at all 110 trials that matched their inclusion criteria; that enabled them to identify a highly significant asymmetry in their funnel plot. This was important because it showed that the smaller the study size, the larger the apparent effect.

The thing in the paper that homeopaths will probably emphasise is that for the set of 21 higher quality trials, the pooled odds ratio (from random effects meta-analysis) was 0.76, suggesting a benefit over placebo. But wait! What are the 95% confidence intervals? 0.59-0.99. This indicates anything from a unimpressive benefit to a clinically negligible one. In other words, it's a deeply uninspiring result, but homeopaths will be trying to tell you that it shows that homeopathic remedies do something.

The interesting thing that the authors then did was to take the 2 largest trials, and look at what happens to the results when you add trials in descending order of patient numbers (Figure 2 of the paper). Once you get to the point where you've included the 14 largest trials, the resulting odds ratio is always less than 1 (except for the case of 17 trials). This is interesting in a way, but all it really does is demonstrate what Shang et al. said: that smaller trials are more likely to give positive results. So the more trials you add, the more positive but less reliable the results are; with 14 or more trials you might just about scrape a statistically significant benefit, but that result is not as reliable as the analysis restricted to the eight largest trials. It's also worth noting that the upper limits of the confidence intervals in Figure 2 are always close to 1, showing that any benefit is likely to be clinically insignificant. They perform a similar analysis in their Figure 3, except they use a meta-regression analysis rather than a random effects meta-analysis, and for that they show no statistically significant benefit no matter how many studies they include.

Another thing that homeopaths will probably jump on is that if one very negative trial (on Arnica for muscle soreness) is omitted from the set of 21 trials, the results appear more positive (odds ratio 0.73, 95% confidence interval 0.56-0.93) when a random effects meta-analysis is used. There are a number of other trials that can be removed from the dataset to give apparently positive results, but only when random effects analysis is used: a meta-regression analysis shows that there is no statistically significant benefit no matter which study you remove. Also, when performing a similar analysis on the 8 large, higher quality trials originally identified by Shang et al., no statistically significant benefit is found whichever trial you decide to remove. Again, note that the 8 largest trials are intrinsically more reliable than the smaller ones.

All the way through the paper, it is noticeable that meta-regression analysis shows more negative results than a random effects meta-analysis. In fact, the authors point out that in their meta-regression analysis "no single predicted OR [odds ratio] could be shown to differ significantly from unity". So which should be used? The authors write "... there is no guideline which tells a researcher when to prefer meta-regression to random effects metaanalysis or vice versa. As the statistical test for asymmetry only has a small power, Egger suggests to perform metaregressions when the respective P-value falls below 0.10. Applying this criterion there seemed to be no need to perform a meta-regression in most of the subsets we analyzed". But this conclusion is based on the restricted analysis of 21 higher quality trials. Shang et al.'s original analysis of 110 trials of homeopathy showed asymmetry with p<0.0001, suggesting that a meta-regression analysis would be more appropriate.

So, the upshot is that the paper's title is misleading. The conclusions on the effectiveness of homeopathy do not highly depend on the set of analyzed trials, if an appropriate test is used. Asymmetry is not adequately identified in the dataset because too few trials are used. And, even if you can convince yourself that you can get a statistically significant benefit by playing around with the numbers, the actual clinical benefit is negligible. In some ways, the paper actually reinforces the conclusions of Shang et al., and it certainly doesn't show that homeopathic medicines work.

21 comments:

Woobegone
said...

I find this kind of paper absolutely fascinating, although not for the reasons that Ullman would have hoped.

As far as I'm concerned it's more likely that a hundred clinical trials get botched or just fabricated, than that our understanding of physics and chemistry is fundamentally flawed. Since everything we know about chemistry says that homeopathic remedies are plain water, I'm just not going to be convinced by any amount of clinical trial evidence to the contrary, until someone shows me a plausible mechanism of action. (Evidence based medicine is all very well, but there comes a point when you have to say, it's a frickin' glass of water.)

so I see all the homeopathy trials as making up a kind of "model organism" for studying the way science and scientific publishing works. Given that homeopathic remedies are known to be completely inert, any positive conclusions or even suggestions of positive conclusions that homeopathy researchers come up with must be either chance findings, mistakes, or fraud.

So homeopathy lets us look at how a community of researchers can generate a body of published papers and even meta-analyze and re-meta-analyze them in great detail, in the absence of any actual phenomenon at all. It's a bit like growing bacteria in a petri dish in which you know there is nothing but agar.

The rather sad conclusion I've come to is that it's very easy for intelligent, thoughtful scientists to see signals in random noise. I fear that an awful lot of published work in sensible fields of medicine and biology is probably just that as well. Homeopathy proves that it can happen. (the problem is that we don't know what's nonsense and what's not within any given field.) It's a warning to scientists everywhere.

woobegone, I'd agree. It shows how difficult it is to eliminate all potential sources of bias. That must go for proper evidence-based medicine as well. It's worth noting that in Shang's original paper, the asymmetry was similar for homeopathy and conventional medicine, suggesting that the level of bias of various kinds was similar for both.

I too find this stuff much more fascinating than I probably ought to. This post has been linked to from Ben Goldacre's miniblog, with the words "More homeopathy metaanalysis details, for those interested". Which sums it up fairly well, I think.

I would have to agree with woobegone that clinical trials of homeopathy can never be sensitive enough to demonstrate a true effect. The probability that a strong positive result is the result of a botch job or even fraud is always going to higher than magic sugar pills having a clinical effect.

Shang, and analyses like these, provide good warnings about how messy data can be when looking at totally implausible treatments and should be warning when looking at the results of more plausible treatments not to jump to conclusions too quickly.

but I think it's more a lesson for researchers and publishers than for readers. I mean readers are in an impossible position: if Thing X is a perfectly plausible treatment for Disease Y, and the evidence says it works, then this probably means that yes it works, but it could be all an illusion due to shoddy work and publication bias.

As a reader you just have no way to know, the only way to improve the system is at the publication end (e.g. require pre-registration of clinical trials, and I see no reason not to extend that to some kinds of preclinical work too).

although registration of trials wouldn't solve the problem of trials that are just crap being conducted. My ideal journal would be one where you have to submit the Introduction and Methods of your paper before you've done the research. If they pass peer review (i.e. if you have a decent proposal), then you're guaranteed publication of the results so long as you stick to the methods you promised to use originally. (Including, of course, the stats.) And if you don't submit the results within a reasonable time-frame they publish a note saying that you didn't.

Next time, 1) Please respond to the exclusion of the polyarthritis study. 2) Please respond with an explanation for why several studies that were defined as "high quality" by other researchers would not be defined as such by Shang (be specific). The 2 Reilly studies, for instance, were mysteriously missing, even though editorials in the Lancet and the BMJ have referred to them as extremely rigorous.3) Please respond to why Shang selected the primary outcome measure with "negative" findings when the studies found primary outcome measures that were "positive." 4) Please explain why Shang purposefully ignored the "adverse events" problems from conventional and homeopathic trials (could it have been because the conventional medical trials had too many adverse events?).

Ultimately, Shang's review proves two things:1) On average, homeopathic trials were of significantly higher quality than conventional trials (21 vs. 9!)2) Conventional drugs seem to work more often than homeopathic medicines do, but 3 (!) of the 6 conventional trials tested drugs that have been taken off the market due to their extremely dangerous side effects. So, conventional drugs work, but there's a 50/50 chance that they get rid of some symptoms but you get worse disease in the process (is that science?).

Ullman, all of the points you make nos 1 to 4 have been more than adequately answered in various arenas that you have not re-entered. Your hit and run tactics are not attractive. For other readers see the train wrecks at wikepedia, jref and more evidence based blogs than I care to list. If you have anything to support your claims of a treatment that defies all we know of science just answer the Badly Shaved Monkeys's question.

What really needs to be done is an analysis of all the poor quality studies. That way it can be shown how positive results for homeopathy correlate with the shoddiness of the studies.Dullman loves all this stuff though, and he'll argue about it until the cows come home because it serves as a distraction for his intended audience. What he doesn't like and cannot do is provide properly documented, incontrovertible examples where homeopathy has actually cured anything non-self-limiting. With more than 200 years of miracle claims he can't produce one example. Any halfwit can make claims for curing self-limiting problems like the common cold because no intervention is required.Dullman is a huckster, a charlatan, a pill-pushing parasite with nothing to offer except a way to shaft the gullible.

Dana's link is this one. You need to hand code html in the comments box.

Dana, your problem with Shang is that you persist in imagining some great conspiracy against homeopathy. If you were able to view Shang as a scientific document, rather than as an attack on homeopathy, this would all be a lot easier.

You know why the polyarthritis study was omitted. There was no matching study of conventional medicine. The point here was to get two comparable datasets to which to apply a meta-analysis. It doesn't matter that the higher quality and large studies are not matched: the point is that those trials were drawn from comparable populations. So, by the design of the study, excluding this trial was not just sensible but necessary.

As for quality, I suspect that Shang's definition was different to previous ones. In any case, the studies you mention were very small and therefore not reliable.

Regarding primary versus secondary measures: since I wasn't there when they did their data extraction, I don't know why they picked one measure over another one. But really, Dana, what you should do before you ask me these questions, is read the damn paper for yourself. Perhaps the authors thought to include the information that you seek. Here's what they wrote:

"We used prespecified criteria to identify outcomes for inclusion in the analyses. The first choice was the main outcome measure, defined as the outcome used for sample-size calculations. If no main outcome was specified, we selected other outcomes, in the order: patients’ overall assessment of improvement; physicians’ overall assessment of improvement; and the clinically most relevant other outcome measure (for example, the occurrence or duration of an illness). Outcomes were selected randomly if several were judged equally relevant...Data were extracted independently by twoobservers, and discrepancies were resolved by consensus".

Now, you might or might not agree with those methods, but what you shouldn't do is impute research fraud on the basis of no evidence.

Finally, Shang et al. ignored adverse events because that had nothing to do with the hypothesis they were testing, which was, as per the title of the paper, "Are the clinical effects of homoeopathy placebo effects?" Again, why not read the paper? The question was specifically adressed in the discussion:

"Another limitation of our study is the exclusive focus on the beneficial effects of homoeopathy and conventional medicine, rather than on both benefits and risks. However, the trials included in the study were small and lacked the power to reveal infrequent but important adverse effects. Furthermore, reporting on adverse effects is inadequate even in larger trials. A comprehensive and valid assessment of adverse effects would probably not have been possible within the framework of this study".

So, you're attacking a paper that you haven't apparently read, based on another paper that you probably haven't read. This reminds me of your championing of the infamous Rustum Roy paper on water memory, which you seem to have mysteriously stopped mentioning since it was totally debunked.

Now, it takes time and effort to respond to your bullshit and whining, so I'm not answering any more of your questions until you answer some of mine. You could start here.

Dana, I'm really interested to know what you would say to those homeopathic enthusiasts who maintain that homeopathy cannot be evaluated by randomised trials (and like to use words like "Quantum entanglement" and produce impressive looking equations to prove it).

In fact, that sort of thing has been done. In the Shang analysis, they calculated an "odds ratio of odds ratios" for the higher quality trials versus the lower quality trials (i.e. the odds ratio for the lower quality studies, divided by the odds ratio for the higher quality studies). This was 0.62 (95% CI 0.43-0.9), demonstrating that lower quality studies showed more positive results.

There's a paper by Ernst and Pittler, where they took data from the Linde et al. 1999 meta-analysis. They plotted JADAD score (a measure of trial quality) against odds ratio, and found that in general the higher the quality the less positive the results.

Excellent analysis once more. On the topic of uninspiring confidence limits I quite like the expression used in the 2007 MTHR report on mobile telecoms: a lower limit of 1.01 was described as of "borderline significance"

Nice post. I too was underwhelmed by Ludkte & Rutten. It really doesn't tell us anything new but it is good to see CAM enthusiasts re-doing meta-analysis rather than just whining a la Dana. Is it fair to call them "enthusiasts"? I think so. Rutten is a homeopath, and Ludkte works for an organization whose purpose is "Promotion and Support of Complementary Medicine", according to their website (this is somewhat misrepresented as "dedicatedto research funding in homeopathy" in the declaration of interests at the end of the paper). The foundation's philosophy may also be gleaned from the titles of the research papers on show here and here: several begin "Proof of effectiveness of..." which is not what you would expect scientists to say, even if their evidence was strong. So, without resorting to an ad hominem attack it still seems fair to assume that L&R were trying to put across the strongest possible case in favour of homeopathy. The fact that this is the best they can do actually provides pretty good evidence in support of Shang et al.

Heh, on that Science Based Medicine discussion, Dana Ullman has returned to make pretty much the exact same points as he did here, for all the world as though they hadn't been addressed. Vintage Ullman.

Day job...

Fieldwork in Egypt

Welcome to Hawk/Handsaw...

This is the blog of Paul Wilson. You can find science- and pseudoscience-related things here, as well as occasional posts on my current research. There is also some stuff about what I do in my spare time (mostly cycling and complaining about politics).