It don’t mean a thing if it ain’t got that swing

– or why there’s no point doing a systematic review if you don’t report it properly.

So this is going to be a little bit of rant, plus a bit of pleading, and a call to arms.

What’s the point of doing any form of research if you report it so inadequately that people are almost morally obliged to ignore it?

Or if you are so selective in your reporting that it gives a completely wrong impression of the results?

The question of not reporting it at all (non-publication of PhDs, for example) is for another day.

There have been plenty of examples where poor reporting of randomised controlled trials has resulted in a false understanding of the efficacy of a treatment. Think Tamiflu or Reboxetine. Both of these, and plenty more like them are the reason behind AllTrials, (and all power to it.)

The topic of the papers was, I think, just a way of keeping the workload manageable (analysing 162 papers is enough of a workload on top of your day job), and so too was the range of years covered, but the point of it was to see if any of the SRs or MAs actually conformed to the reporting guidelines that are available, and was a contribution by a librarian to the whole process visible.

It produced some very interesting (depressing?) results. Here’s just a few of them…..

80% of paper did NOT mention PICO

68% of the searches were NOT fully described & transparent (absolutely essential for any reproduction of the work in the future)

66% did NOT use controlled vocabulary (MeSH, etc)

48% did NOT explicitly use synonyms (put this together with the previous point, and I don’t actually think it equates to a literature search, far less a systematic review!)

49% did NOT recognise any risk of bias in their work (eg language limitations)

most studies only used one database – a significant source of bias, and a virtual guarantee that relevant papers will be missed.

less than 10% of papers mentioned a contribution by a librarian

This is a relatively small study but alarm bells are clanging like crazy.

spoiler:
“Problems remain with SR search quality and reporting. SRs with librarian or information specialist co-authors are correlated with significantly higher quality reported search strategies. To minimize bias in SRs, authors and editors could encourage librarian engagement in SRs including authorship as a potential way to help improve documentation of the search strategy.”

We can help researchers publish better work!!

There are some tremendous reporting guidelines for all sorts of research (they’re pulled together in one place by EquatorNetwork (and all power to them!)) and plenty of specific advice around reporting of systematic reviews:

[…] Systematic reviews are often published with poor methodologies or are poorly reported. This was the finding from Spanish, French and Swiss librarians who evaluated a set of systematic reviews and meta-analyses on a specific condition, and found that, for example, 80% of the papers didn’t mention PICO, and 68% of the searches were not fully described and transparent. This led to an EAHIL act which was put forward by Isla Kuhn at the EAHIL General Assembly meeting, and you can read more about this on Isla’s blog: https://ilk21.wordpress.com/2016/06/12/it-dont-mean-a-thing-if-it-aint-got-that-swing/ […]