Methodolatry: The profane worship of the randomized clinical trial as the only valid method of investigation.

Many of you have e-mailed me and other SBM bloggers about a recent article in The Atlantic by Shannon Brownlee and Jeanne Lenzer, two reporters whose particular bias is that we as a nation are “over treated.” That may be true, although not to the extent that Brownlee, at least, seems to think, and her article on swine flu was truly biased and painful to read. Moreover, “methodalatry” perfectly describes one of the complaints we at SBM have about the “evidence-based medicine” paradigm. So I’m really glad that revere took it on and demolished it.

The hero of The Atlantic article, Tom Jefferson clearly has an agenda about flu vaccines. Indeed, he has such an agenda that he was invited to the National Vaccine Information Center’s vaccine conference in early October. The NVIC is the oldest and biggest antivaccine organization there is. Either he didn’t know that, in which case he’s clueless, or he didn’t care. In any case, it was clear that he was invited there because of his stance on flu vaccination, and he was even going to be awarded the NVIC “Courage in Science” Award. To his credit, Jefferson backed out when he found out that he would be sharing the stage with Andrew Wakefield, who was to be given the NVIC “Humanitarian Award.” He was appropriately horrified. Still, he should never have accepted in the first place, given that the NVIC clearly wanted to coopt him and use his gadfly status to make its anti-vaccine stance seem reasonable and science-based.

That’s just one reason why I don’t take Tom Jefferson particularly seriously anymore. I tend to agree with revere that Jefferson is drifting perilously close to crank territory with respect to flu vaccines. Indeed, “methodolatry” is an awesome term to describe his approach. Actually, it’s a great term to describe some of the Cochrane scientists responsible for analyzing the efficacy of mammography screening, as well; their conclusions and methods rather remind me of Jefferson’s.

Finally, you might also want to reread (or read for the first time if you haven’t read it already) Mark Crislip’s article on flu vaccine efficacy, which, although not directly written in response to Brownlee’s article, does address many of the shortcomings in its analysis of H1N1 vaccine efficacy.

42 thoughts on ““Methodolatry”: My new favorite term for one of the shortcomings of evidence-based medicine”

Damn that Revere. I’ve been privately muttering about methodolatry for a few years. Scooped again, but as usual, he’s smashed it out of the park and analyzed it far more deeply than I could have.

The RCT is a great tool for the right problem, but it falls down hard in my own area of interest, psychotherapy research.

It’s absolutely gruesome to see some of the ways psychotherapy trials get ham-handedly shoved into an RCT-esque model. We have “wait-list” controls (aka “Do nothing” – against which I dare say witch-doctors and homeopaths would prevail mightily), “head-to-head” trials (previously plagued by investigators pushing their pet Brand X against a pitifully done Brand Y), psychotherapy vs “empathic listening” (aka “What exactly *is* the control here?”), and so on.

The group I collaborate with is slowly struggling out of the forest and back into the sunlight, and the main thing we’ve learned: good quality psychotherapy research is just damn hard, and we might only be about 40% of the way toward decent experimental methods to properly assess it.

By this I mean, you call it methodolatry when the most stringent, skeptical science disconfirms or threatens your beliefs. So you push back, challenging the idea that the RCT is an appropriate “gold standard.”

Yet, when the most stringent science confirms your bias (as in RCTs of acupuncture), you approvingly call it evidence-based skepticism. What gives?

Actually, it is entirely consistent with our view at SBM that the entire evidence-based medicine paradigm is flawed in that it values RCTs above all else and ignores basic science and epidemiology. RCTs are the best means of studying certain interventions. However, there are many questions for which placebo-controlled RCTs can’t be done, either because they are unethical (cancer chemotherapy, many vaccine trials) or because they are impractical (surgery). In those cases, the totality of the scientific and clinical evidence need to be examined.

An example: Antivaxers sometimes call for a randomized, placebo-controlled trial of vaccinated versus unvaccinated children in order to see if vaccines cause autism. Such a trial, although it would be the most scientifically rigorous, would be profoundly unethical and in clear violation of the Helsinki Declaration, as it would leave the control group completely vulnerable to infectious disease. So we use the trials we can. This is the case with flu vaccination, given the background evidence that it does indeed prevent the flu. In fact, this issue was discussed in the Atlantic article, with Jefferson wanting a randomized, placebo-controlled trial of flu vaccination. In a pandemic, such a trial would be highly unethical.

As far as acupuncture studies go, a double blind, placebo-controlled RCT is the way to go whenever possible because (1) there are no ethical constraints against it and (2) it is the most rigorous way to test therapies for conditions for which a large placebo effect can be expected.

What ceekay may fail to realize (or maybe not) is that it’s about doubt, not confirmation. We don’t advocate tossing out evidence we don’t agree with; we advocate tossing out garbage evidence, or at least discounting it to an extent. RCTs, as Dr. Gorski points out, can be very useful, but also deceptive. To ask that we take into account basic scientific plausibility is not a stretch.

Can someone on SBM address the points of the article rather than just saying RTC for flu vaccine is unethical? I understand that point very well.

But, can someone discuss the challenges the authors made to the 50% mortality reduction statistic? How about the 2004 low production year where mortality didn’t fall? And, I’m most interested in their assertion that the ethical moratorium on RTC for aggressive breast cancer treatment increased mortality risk. What about situations like that?

It is true that many of us were once naively responding to CAM claims by calling for controlled trials. Positive studies promptly appeared. It must look to many like a double standard if we now easily dismiss them.

Of course, we should have known better. We already knew to await replication of conventional studies by independent centres before accepting their findings.

And in defence of the RCT, they have not been uniformly or even generally positive for any unlikely CAM claim.

They thus do produce results consistent with those predicted by other scientific considerations, so long as you perform enough studies of sufficient quality, and especially if you are permitted to apply some additional weight to negative studies, one that takes into account known biases favouring positive results, such as the low level of statistical significance usually accepted in medical studies and the difficulties in blinding some procedures.

We have also since learnt to our horror how drug companies have been able to manipulate the RCT system. We now have ot be somewhat wary of their RCT results.

RCTs are obviously not all created equal. Being persuaded by the findings of only some of them is not a scandalous double-standard: it’s just critical thinking at work. “What gives” is that RCTs need evaluating, obviously.

Maybe you can’t compare A to B. But you can compare A to C and B to C, and so get your answer.

Example: Give subjects a vaccine and controls a placebo, then measure antibody titres. Use historical data regarding antibody titres and immunity to calculate percentage of patients with effective immunity in both groups.

@Robin, I’m also a layperson with regards to vaccination, but I’ll take a stab at one aspect of the mortality-rate question: evidence of reduced mortality has been an important line of evidence, but it is not by a long shot the sole basis for vaccination. There are other ways of inferring efficacy (and Dr. Benway just explained one of them). Multiple lines of evidence converge to indicate that vaccination works.

It also seems important to note that reduction in mortality rate does not need to be large to be valuable, particularly given that the risks are so low. If the power of vaccines to reduce mortality has been historically over-estimated due to research flaws, that’s interesting and important, but it doesn’t mean that there’s no effect. The Atlantic asserts that “the healthy-user effect explained the entire benefit that other researchers were attributing to flu vaccine.” But the slightest change in wording there changes everything: does Jackson’s data really explain the “entire” benefit? Or just a chunk of it? Who’s making that interpretation? With what bias? It’s just too easy to exaggerate the data to make a point, ignoring the fact that Jackson’s results also have error bars.

The “50% mortality” issue is also a bit of a straw man argument about which the authors make a huge deal. As Mark pointed out in his podcast version of this post, most deaths due to flu are from secondary causes; so it is not unreasonable to find or infer that decreasing flu would have an effect on mortality from other causes.

In any case, over 2 million people a year in the U.S. die from all causes. Of these, around 631,000 are due to heart attacks and 124,000 are due to respiratory disease. By comparison, only roughly 36,000 of these deaths are due to the flu or its complications. Decreases in flu mortality are fairly hard to tease out of overall mortality statistics.

In any case, Paul makes a good point in that it is the convergence of evidence, rather than any one single study, that suggest the efficacy of the flu vaccine. As Mark put it, it’s not our best vaccine, and it depends upon scientists making a correct prediction about what strain will be circulating each season, but it does work. It’s shades of gray, though. Not black and white.

In any case, if you read revere’s post, as well as Mark Crislip’s post and podcast, a lot of the points Robin asks are addressed, just not in convenient bullet points or “point-counterpoint” format.

I think it’s important to note that even randomized controlled trials can be messed up pretty badly, if your methods are wrong. Like, if you do about fifty tests, and then forget to count the huge number of tests into your statistics, so you get false significance for one of them; or if you’re making assumptions that are incorrect and basing your otherwise-legit research on them… you know, that kind of thing.

I’ve actually seen people claiming to have done proper tests of their alternative medicine stuff, with control group, etc., only when you went to look at their reports, either they weren’t anything near complete reports, or else they were just sciency-sounding things that didn’t mean much when you actually dug through them a bit, evidently meant to fool people who’d never had to sit and analyze journal articles while getting peppered by questions from the Professor of the Week.

They thus do produce results consistent with those predicted by other scientific considerations, so long as you perform enough studies of sufficient quality, and especially if you are permitted to apply some additional weight to negative studies, one that takes into account known biases favouring positive results, such as the low level of statistical significance usually accepted in medical studies and the difficulties in blinding some procedures.

Known biases like the file-drawer effect. Positive outcomes get reported disproportionately, giving the illusion of significance where it doesn’t actually exist. It’s perceived to be comparatively difficult to get a paper published in a journal when the paper basically says “we did this and nothing interesting happened”. Yet these are usually the most important research outcomes. This is a persisting problem in current scientific culture, and not just biomedical science.

I say “perceived to be” because this effect is not due to any actual reticence on the part of journal editors or peer reviewers. It’s entirely self-inflicted censorship, the conscious decision to leave the negative outcomes in the drawer. I think it comes down to the psychological conception of productivity. Negative outcomes are not seen as being productive in the same way as positive outcomes.

One thing is that the article points out, on page 2, that “In 2004, for example, vaccine production fell behind, causing a 40 percent drop in immunization rates. Yet mortality did not rise. In addition, vaccine “mismatches” occurred in 1968 and 1997: in both years, the vaccine that had been produced in the summer protected against one set of viruses, but come winter, a different set was circulating. In effect, nobody was vaccinated. Yet death rates from all causes, including flu and the various illnesses it can exacerbate, did not budge.”

I got my flu shot yesterday, but I’m still wondering about that. Is there really no change in mortality when the vaccine doesn’t occur? Are they looking at the data wrong? Is the real world different from the lab?

Another problem with “methodolatry”, and one which is very significant in my field (engineering) because of how often it’s ignored, is that you can have the best, most rigorous scientific experiment in the world (and an RCT is one of the most rigorous forms of experiment available in medicine), but it will be completely meaningless if you’ve asked the wrong question in the first place. In the context of the RCT, that relates to how you’ve selected your subjects, and what your control is. An RCT that isn’t sure what question it’s asking will be completely worthless, even if it’s done with double-blinds, randomization, and everything.

The examples presented in the Atlantic monthly suggest years when fewer than average got the flu vaccine. We should expect deaths due to flu or flu-related complications should go up. According to the author, we did not see a rise. Therefore the vaccine does not have the effect it should or the effect is so small as to be indistinguishable from noise.

My first big problem is I can find no references in the article to the data the claims are based on.

Second, there are other explanations.

1) The author’s interpretation. For example:

“In 1989, only 15 percent of people over age 65 in the U.S. and Canada were vaccinated against flu. Today, more than 65 percent are immunized. Yet death rates among the elderly during flu season have increased rather than decreased.”

Death rates in general or death rates due to flu? It’s important to specify. The author fails to. Old people might die more now during flu season because they have very different lifestyles. Flu season is also winter. Maybe more old people are driving and they die in car accidents in winter, for example. The author even fails to indicate if this rise is significant. Lots of old people die over any given period. If the flu vaccine reduces an additional 1,000 deaths in the USA during flu season, that might not be statistically significant against the number of old people who die over several months of flu season.

2) There are many variables contributing to deaths from flu. Years listed with mismatches might have had very mild flues. They might have also featured very snowy winters keeping people indoors. You have to control for a lot of variables between flu seasons to get a good comparison.

The recent Canadian study that compared Ontario vs the rest of Canada is a very powerful study. Ontario has a vigorous program to vaccinate the whole province. The other provinces don’t. So: Same population. Same flu season. Same vaccine. As the authors themselves note it’s a great natural experiment. We should expect a statistically significant reduction in flu deaths in Ontario. And we did see it, lending support that trying to attain herd immunity in a population reduces flu deaths.

3) We’re told about years when there was a poor mismatch but deaths did not increase. Like I say, it could be those years we got lucky and the flu was mild. But did the author consider years when there was a poor mismatch and deaths increased?

[Influenza epidemic in a nursing home caused by a virus not included in the vaccine]

In the autumn of 1992 two-thirds of the population of a nursing home in Amsterdam was vaccinated against influenza. However, in March 1993 an outbreak of an influenza like illness occurred with a morbidity rate of 49% and a mortality rate of 10%. There was sufficient serological evidence to show that the vaccine as such had induced adequate immunity. As the causative agent an influenza A/H3N2 virus was identified. The failing activity of the vaccine in this instance was apparently caused by the absence of sufficient antigen similarity between the A/H3N2 vaccine component and the epidemic virus.

4) “In 1989, only 15 percent of people over age 65 in the U.S. and Canada were vaccinated against flu. Today, more than 65 percent are immunized. Yet death rates among the elderly during flu season have increased rather than decreased.”

@PeterLipson
You state that “We don’t advocate tossing out evidence we don’t agree with; we advocate tossing out garbage evidence, or at least discounting it to an extent.”

But as soon as someone present good and evidence-based science you do not agree with your preconceived notions you quickly resort to ad hominem arguments:

@DavidGorski “I tend to agree with revere that Jefferson is drifting perilously close to crank territory with respect to flu vaccines.”

This whole debate is reeking of the old tobacco industry tactics of creating confusion and attacking people who does not share your beliefs.

And as Mark Crislips biased selection of studies in his oct. 9 article show, this iste is not about tossing out or discounting garbage evidence either, but about cherry picking studies supporting your position.

I’d like to comment on a topic that has been bothering me. Anonymous blogging abounds on the internet, but anonymous blogging by individuals who claim to be experts on a topic is especially disturbing. Who is revere anyway? Why the secret identity? The Effect Measure site says that: “The Editors of Effect Measure are senior public health scientists and practitioners. Their names would be immediately recognizable to many in the public health community. They prefer to keep their online and public lives separate to allow maximum freedom of expression.” So revere is an amalgamation of editors. A group effort. So far I haven’t seen any blogs that are remotely controversial enough to require a secret identity.

Let’s look further. I do like to check sources.

About ScienceBlogs:
“We believe in providing our bloggers with the freedom to exercise their own editorial and creative instincts. We do not edit their work and we do not tell them what to write about.”
And this: “We have selected our 60+ bloggers based on their originality, insight, talent, and dedication and how we think they would contribute to the discussion at ScienceBlogs.”

No mention of the bloggers’ scientific/medical credentials or their objectivity/lack of conflicts of interest. From the CDC to the AAP to the AMA, etc. etc., consumers are admonished to beware of information gleaned from the internet. Anonymous bloggers rank right up there at the top of my list of sources of which to be wary. With all due respect, Dr. Gorski’s adulation of revere does not equate to actual credentials.

Why all the mystery? Science is based upon the examination of facts. Due diligence should include researching the background of so-called experts. Otherwise praise of anonymous sources veers close to idolatry.

wales, if anonymity were an either/or position, you might have a point. But most of us –yourself included– can be both “on the record” and “off the record.”

Off the record conversation allows people a more casual, spontaneous manner that humans seem to need and enjoy.

You can have a pub conversation with the reveres. You also can follow their links to publications that support their opinions.

You can have both things rather than either/or iz all I’m sayin’.

Source credibility is a major factor for non-experts trying to figure out what to believe when the source contradicts the scientific consensus or when one is unaware of that consensus. Otherwise, not so much.

I am using a pseudonym, but I do not purport to be an expert nor do I give medical advice. There is also a difference between generating a blog piece and responding to one in the comments section.

Source credibility is important when expert opinions are bandied about. Experts discussing facts are one thing, facts are verifiable. But experts giving their opinions, well the opinion is only as good as the credibility of the expert. When that cannot be verified, the opinion is no more authoritative than the next guy’s.

The responses to revere’s defense of pseudonyms are interesting. The “spineless” example was particularly ironic. Revere wants the freedom to call someone “spineless” without consequences. One could point to revere as an example of “spineless” in this case.

I disagree with the revere collective, the use of pseudonyms is no different than the use of anonymity, especially in the case of revere where there are multiple contributors under one pseudonym.

With the current ghost writing problems in medical journals, I am more wary than ever.

Good points Wales. Same thoughts have been bothering me. Who are these guys? Who do they consult for? And so on. Why are they so acerbic and unwilling to discuss e.g. that flu vaccines may be useless (the T Jefferson topic above)? And so on.

The site claims to be science-based, but as I see it, it is mostly science opinionated. That is fine by me if it is clearly stated. Because in that case I am clearly warned that I have to double-check the statements made.

Instead most readers of this site (not part of the like-minded cabal) need to be fortunate enough of having a good higher science education (or interest) that start mental alarm bells ring when the opinions presented are at odds with say Cochrane or evolutionary biology.

The flip side of anonymity is the appeal to authority. People with impeccable credentials can spout nonsense, and people without them can see through nonsense. I don’t care who wrote it. I concentrate on the content and whether it is supported by credible references.

The other part about anonymity is this: Some people rely far too much on authority in evaluating an argument and not enough on critically examining the argument itself. If they can convince themselves that a source is not “authoritative,” then they ignore it, regardless of how good its arguments are. Similarly, if they trust a source, they can turn off their brain and just accept what it says. It’s a lazy shortcut that we all use to some extent, but cranks “crank it up to 11,” so to speak. Moreover, as someone who’s blogged under a pseudonym in other venues, I can tell you one thing. Cranks love the ad hominem attack. They can’t win on science, reason, or evidence; so they launch personal smears. The use of a pseudonym robs them of that weapon; consequently they go to great lengths to “out” anonymous or pseudonymous bloggers.

I assume you are anonymous here for good reason. We are all anonymous in the voting booth for good reasons. Investigative journalism would grind to a halt if unnamed sources were no longer allowed. However, good journalists do make a point of cross-checking information from unnamed sources before going to press.

That’s why I’d recommend you cross-check information from me or any other anonymous source on the webs.

Probably best to cross-check information from named sources also, come to think of it.

Why are they so acerbic and unwilling to discuss e.g. that flu vaccines may be useless …

Perhaps because

1) sixty years of world-wide experience with flu vaccines has established a benefit vs risk ratio in their favor with only minor quibbles and

2) very few interventions in medicine enjoy such a high benefit vs risk ratio

3) phobias intensify when people are encouraged to obsess over them.

If you have a friend afraid to fly, don’t urge him to “educate himself” about what might go wrong with a jet engine, variables that diminish pilot alertness, gaps in airport security, freak weather conditions, crash statistics, etc.

How does one know that a reasonable concern has crossed the line into phobia? When the time spent attending to the risk is far greater than the amount given to every other health risk humans face each day.

Anyone using the word “squalene” in a sentence while smoking a cigarette or driving without a seatbelt is probably vaccine phobic.

It is a given that cross checking sources and facts is necessary, whether the source of the fact or opinion is a verified expert or not. I do not advocate blind faith in authority. I encourage accountability and proof of credibility on the part of experts.

Experts publishing under their actual identities do not encourage intellectual sloth and worship of authority in people unless they were already prone to those characteristics.

There is a middle ground between blind faith in authority and extreme skepticism of authority. Credible experts who are actively posing as experts in online forums should not shy away from being held accountable for their views on public health matters. When they do it raises questions. People who prefer to verify facts generally include information about the credentials and conflicts of interest of experts in the realm of verifiable facts.

Thanks for proving my points on being opinionated and resorting to ad hominem arguments “Dr” Benway.

<1) sixty years of … established a benefit vs risk ratio in their
According to Cochrane essentially no benefit proven

<2) …enjoy such a high benefit vs risk ratio
According to Cochrane essentially no benefit proven

<3) …phobias intensify …obsess over them.
Do you mean your phobia of Cochrane perhaps being right, instead of you?

<If you have a friend afraid to fly, …etc.
Hey, the topic was medicine, not avionics…

<… crossed the line into phobia? …each day.
?????? you seem to have lost your train of thought

<Anyone using the word “squalene” in a sentence
<while smoking a cigarette or driving without a seatbelt
<is probably vaccine phobic.
Hmm, interesting non-sequitur. Am I to assume people not sharing your belief in the unproven effectiveness of influenza vaccine are cranks. Ergo the Cochrane Institiute experts are cranks because they have proven you wrong? Thus anyone doubting your creed is a crank?

For Healthy adults there is not enough evidence to decide whether routine vaccination is effective

In elderly people the conclusion was that there was no correlation between vaccine coverage and ILI attack rate, but decreased pneumonia incidence was observed in nursing homes. Problems with the quality of included studies.

“There is no credible evidence that vaccination of healthy people under the age of 60, who are HCWs caring for the elderly, affects influenza complications in those cared for.”

But vaccinating healthy persons under 60 may reduce cases of influenza.

skepsis, you’re all over the map with your bag of stats. You’ve got to pick a specific outcome, such as antibody titres, mortality, morbidity, or hospitalization else you risk comparing apples to oranges.

Note how outcomes can vary depending upon the match between the antigens in the vaccine and the circulating virus.

There are areas of legitimate debate. For example, older people mount a weaker antibody response than younger people. Be careful not to extrapolate this statement a generalization about flu vaccine efficacy for everyone. Even Jefferson concedes that the vaccine is effective for younger people.

@weing
<What do you think that means?
Good question. The Cochrane reviews do show is that something is not quite right with the flu studies, their reproducibility and their methodology. It is a bit hard to accomodate all the propositions (placebo, effective, not effective and perhaps effective to a small degree in some). I think the "Relation of study quality…" (see above) paper at least answers some of those questions but not all.

@Benway
My stats? The Cochrane reviews you mean? I am pretty sure those working for the "gold standard of evidence based medicine" are aware of all points you make.

Currently there is not enough scientific evidence to either prove or disprove any of the flu vaccine hypotheses (effect vs no effect vs limited effect). It may even be so that all possibilities are true at various points in an individuals lifetime.

We just plain do not know enough yet.

The big problem is that this has led to a stupid emotional debate about whether flu vaccines do work or not. And to the current swine flu panic absurdity. And to the absurd division in two almost sectarian camps of almost religiously vociferous acolytes.

This leaves both the scientifically educated and the layman totally at loss. So when I read a site like this, with arguments made by persons anonymous (like me) or unknown (like Gorski and Benway), I have to make some decisions on how to evaluate the arguments presented (Wales’s discussion above).

And in my case I definitively put more weight in Cochrane reviews than on a guy calling a senior Cochrane reviewer with 20 years of experience a crank !

Such a biased and unscientific ad hominem line of argumentation is just the creationist style of argumentation. What is the agenda of someone making such categorical and statements? Are those really interested in science?

But not only that, it also makes me put all statements related to influensa vaccinations, made by authors on this site, into question. You can’t pretend to advocate science and then just go emotionally ballistic if the science does not equate with your personal notions.

My question was about what you thought the quote “There is no credible evidence that vaccination of healthy people under the age of 60, who are HCWs caring for the elderly, affects influenza complications in those cared for.” means. I’m trying to gauge your comprehension.

That if you get sick the current studies prove no difference in outcome between clinics with flu vaccinated staff and not. On the other hand morbidity decreased if staff was vaccinated (and that was high quality evidence).