Subscribe to this blog

Follow by Email

On literature pollution and cottage-industry science

A few days ago there was a minor
Twitterstorm over a particular paper that claimed to have found an imaging
biomarker that was predictive of some aspect of outcome in adults with autism.
The details actually don’t matter that much and I don’t intend to pick on that
study in particular, or even link to it, as it’s no worse than many that get
published. What it prompted, though, was more interesting – a debate on
research practices in the field of cognitive neuroscience and neuroimaging,
particularly relating to the size of studies required to address some research
questions and the scale of research operation they might entail.

What kicked off the debate was a question
of how likely the result they found was to be “real”; i.e., to represent a
robust finding that would replicate across future studies and generalise to
other samples of autistic patients. I made a fairly uncompromising prediction
that it would not replicate, which was based on the fact that the finding
derived from: a small sample (n=31, in this case, but split into two), an
exploratory study (i.e., not aimed at or constrained by any specific
hypothesis, so that group differences in pretty much any imaging parameter
would do) and lack of a replication sample (to test directly, with exactly the
same methodology, whether the findings from the study were robust, prior to
bothering anyone else with them).

The reason for my cynicism is twofold:
first, the study was statistically under-powered, and such studies are theoretically
more likely to generate false positives. Second, and more damningly, there have
been literally hundreds of similar studies published using neuroimaging measures
to try and identify signatures that would distinguish between groups of people
or predict the outcome of illness. For psychiatric conditions like autism or
schizophrenia I don’t know of any such “findings” that have held up. We still
have no diagnostic or prognostic imaging markers, or any other biomarkers for
that matter, that have either yielded robust insights into underlying
pathogenic mechanisms or been applicable in the clinic.

There is thus strong empirical evidence
that the small sample, exploratory, no
replication design is a sure-fire way of generating findings that are, essentially, noise.

This is by no means a problem only for
neuroimaging studies; the field of psychology is grappling with similar
problems and many key findings in cell biology have similarly failed to replicate. We have seen it before in genetics, too, during the “candidate gene
era”, when individual research groups could carry out a small-scale study
testing single-nucleotide polymorphisms in a particular gene for association
with a particular trait or disorder. The problem was the samples were typically
small and under-powered, the researchers often tested multiple SNPs, haplotypes
or genotypes but rarely corrected for such multiple tests, and they usually did
not include a replication sample. What resulted was an entire body of
literature hopelessly polluted by false positives.

This problem was likely heavily compounded
by publication bias, with negative findings far less likely to be published. There
is evidence that that problem exists in the neuroimaging literature too,
especially for exploratory studies. If you are simply looking for some group
difference in any of hundreds or thousands of possible imaging parameters, then
finding one may be a (misplaced) cause for celebration, but not finding one is
hardly worthy of writing up.

In genetics, the problems with the
candidate gene approach were finally realised and fully grappled with. The
solution was to perform unbiased tests for SNP associations across the whole
genome (GWAS), to correct rigorously for the multiple tests involved, and to always
include a separate replication sample prior to publication. Of course, to
enable all that required something else: the formation of enormous consortia to
generate the sample sizes required to achieve the necessary statistical power
(given how many tests were being performed and the small effect sizes expected).

This brings me back to the reaction on
Twitter to the criticism of this particular paper. A number of people suggested
that if neuroimaging studies were expected to have larger samples and to also
include replication samples, then only very large labs would be able to afford
to carry them out. What would the small labs do? How would they keep their
graduate students busy and train them?

I have to say I have absolutely no sympathy
for that argument at all, especially when it comes to allocating funding. We
don’t have a right to be funded just so we can be busy. If a particular
experiment requires a certain sample size to detect an effect size in the expected
and reasonable range, then it should not be carried out without such a sample.
And if it is an exploratory study, then it should have a replication sample
built in from the start – it should not be left to the field to determine
whether the finding is real or not.

You might say, and indeed some people did
say, that even if you can’t achieve those goals, because the lab is too small
or does not have enough funding, at least doing it on a small scale is better
than nothing.

Well, it’s not. It’s worse than nothing.

Such studies just pollute the literature
with false positives – obscuring any real signal amongst a mass of surrounding
flotsam that future researchers will have to wade through. Sure, they keep
people busy, they allow graduate students to be trained (badly), and they
generate papers, which often get cited (compounding the pollution). But they
are not part of “normal science” – they do not contribute incrementally and
cumulatively to a body of knowledge.

We are no further in understanding the
neural basis of a condition like autism than we were before the hundreds of
small-sample/exploratory-design studies published on the topic. They have not
combined to give us any new insights, they don’t build on each other, they
don’t constrain each other or allow subsequent research to ask deeper
questions. They just sit there as “findings”, but not as facts.

Lest I be accused of being too preachy, I should
confess to some of these practices myself. Several years ago, while candidate
gene studies were still the norm, we published a paper that included a positive
association of semaphorin genes with schizophrenia (prompted by relevant
phenotypes in mutant mice). It seems quite likely now that that association was
a false positive, as a signal from the gene in question has not emerged in
larger genome-wide association studies.

And the one neuroimaging study I have done
so far, on synaesthesia, certainly suffered from a small sample size (at the
time it was considered decent), and no replication sample. In our defense, our study was itself designed as a replication of previous findings, combining
functional and structural neuroimaging. While our structural findings did
mirror those previously reported (in general direction and spatial distribution
of effects, though not precise regions), our functional results were quite
incongruent with previous findings. As we did not have a replication sample
built into our own design, I can’t be particularly confident that our findings
will generalise – perhaps they were a chance finding in a fairly small sample.
(Indeed, the imaging findings in synaesthesia have been generally quite
inconsistent and it is difficult to know which findings constitute real results
that future research studies could be built on).

If I were designing these kinds of studies
now I would use a very different design, with much larger samples and in-built
replication (and pre-registration). If that means they are more expensive, so
be it. If it means my group can’t do them alone, well that’s just going to be
the way it is. No one should fund me, or any lab, to do under-powered studies.

For the neuroimaging field generally that
may well mean embracing the idea of larger consortia and adopting common
scanning formats that enable combining subjects across centres, or at least
subsequent meta-analyses. And it will mean that smaller labs may have to give
up on the idea of making a living from studies attempting to find differences
between groups of people without enough subjects. You’ll find things – they
just won’t be real.

Even if the study was adequately powered and a marginally significant result was obtained, does it really tell you much about autism? The crisis is more about random design than statistics and poor theoretical models.

Even if the study was adequately powered and a marginally significant result was obtained, does it really tell you much about autism? The crisis is more about random design than statistics and poor theoretical models. No amount of p-enhancement can tell you what is worth looking for.

is it really "worse than nothing"? That's a strong claim. I'd guess about 98% of neuroimaging studies since their inception in the late 70's involve samples smaller than 31. None of this has contributed cumulatively at all? Just noise.

It is important to note that power is not just a function of sample size. Many factors impact power including variability, measurement error, significance level, choice of statistical model, assumptions of statistical model, effect size, sample design, genetic architecture, etc. I could go on and on. Sample size is important but so are all of these other factors that are almost always ignored in both candidate gene studies and GWAS.

Often our sample sizes are small because clinical populations are hard to recruit. What can we do to make the best of this situation?

You mentioned one: pre-registration. In the same vein, I guess the more specific a prediction the better. Similarly, the more reliable a measure, the better (so I should be pre-calibrating my experiment on a non-clinical sample to ensure I know the reliability of the measures I plan to use, and can add that into my power analysis). Anything else?

(yes, I take the point that underpowered studies shouldn't be funded, but doesn't that make ideas about improving underpowered studies even more relevant? I'm not going to get funded for this pilot study, but I still want to get the most evidential value out of it!)

Thanks Tom. Yes, there are several things that could be done to improve the situation. Definitely pre-calibrating your measures is a key way to tell what sample size you should use, to have power to detect an effect of a certain size against a background of whatever variance you observe.

In the studies themselves, bigger is better, generally speaking, unless of course there's a lot of hidden heterogeneity, in which case adding in more heterogeneous cases (eg of "autism") won't necessarily help! That's a whole other level of problem - for another blog I think!

But given that, all the more reason, if you find some effect in an exploratory analysis, to have a replication sample ready to test it on! Otherwise you simply won't know if it's a generalisable finding or not.

As an Autism experienced parent I am delighted to see an academic pointing out the scientific flaws in the type of studies that proliferate in our field, generating so much empty vessel amplifying noise in the media. With the argument of "giving small labs something to do" is raised - why are the academics deciding what needs to be studied? Why not listen to the community and engage with people who live and love with Autism? I'm not talking about huge organisations who have invested millions of donated funds in trying to find a "smoking gun" while contributing to an atmosphere of fear and hate. I'm talking about recruiting and surveying people themselves. This has to include self advocates and those who need family or others to advocate on their behalf. As a parent I am so fed up with post grads doing meaningless research which contributes zero to improving the lives of people with autism. It has no effect on policy other than to give a few outspoken politicians a chance to rush off a quick press release and try get a sound bite while hitching a ride on the latest blame wagon.

Post a Comment

Popular posts from this blog

Can molecular memories of our ancestors’
experiences affect our own behaviour and physiology? That idea has certainly
grabbed hold of the public imagination, under the banner of the seemingly
ubiquitous buzzword “epigenetics”. Transgenerational epigenetic inheritance is
the idea that a person’s experiences can somehow mark their genomes in ways
that are passed on to their children and grandchildren. Those marks on the
genome are then thought to influence gene expression and affect the behaviour
and physiology of people who inherit them. The way this notion is referred to – both in
popular pieces and in the scientific literature – you’d be forgiven for
thinking it is an established fact in humans, based on mountains of consistent,
compelling evidence. In fact, the opposite is true – it is based on the
flimsiest of evidence from a very small number of studies with very small
sample sizes and serious methodological flaws. [Note that there is, by contrast,
very good evidence for this kind…

I recently wrote a blogpost examining the
supposed evidence for transgenerational epigenetic inheritance (TGEI) in
humans. This focused specifically on a set of studies commonly cited as
convincingly demonstrating the phenomenon whereby the experiences of one
generation can have effects that are transmitted, through non-genetic means, to
their offspring, and, more importantly, even to their grandchildren. Having
examined what I considered to be the most prominent papers making these claims,
I concluded that they do not in fact provide any evidence supporting that idea,
as they are riddled with fatal methodological flaws. While the scope of that piece was limited
to studies in humans, I have also previously considered animal studies making
similar claims, which suffer from similar methodological flaws (here and here).
My overall conclusion is that there is effectively no evidence for TGEI in
humans (contrary to widespread belief) and very little in mammals more
generally (with one very…

GWAS (genome-wide association studies) for
psychiatric illnesses may be about to become a victim of their own success. The
idea behind these studies is that common genetic variation – ancient mutations
that segregate in the population – may partly underlie the high heritability of
common psychiatric and neurological disorders, such as schizophrenia, autism,
epilepsy, ADHD, depression, and so on. The accumulating evidence from over ten
years of GWAS strongly supports that idea, with many hundreds of such risk
variants now having been identified. The problem is it’s not at all clear what
to do with that information. GWAS are a method to carry out a kind of
genetic epidemiology, based on a simple premise – if a particular genetic
variant at some position in the genome (say an “A” base, as opposed to a “T” at position 236,456 on chromosome 9) – is associated with an increased risk
of some condition, then the frequency of the “A” version should be higher in
people with the condition than pe…