Editorial

Systematic reviews seek to identify and summarise all the relevant research evidence relating to the clinical or health policy decision under consideration. This rationale can be fatally undermined if the review only includes a proportion of the research conducted, due to the restricted availability of some studies or their data. The recognition of reporting bias and its potential to confound the results of evidence synthesis is not new,[1] and there is now ample methodological research demonstrating that studies with 'positive' results are more likely to be published and published sooner, than those with 'negative' results.[2][3] If systematic reviews fail to address reporting biases they are at risk of generating misleading results. We should, therefore, pay particular attention to reviews that either identify strong evidence of a risk of reporting bias or adopt innovative methods to overcome it.

The January 2012 issue of The Cochrane Library sees the publication of an updated Cochrane Review of the neuraminidase inhibitors oseltamivir (Tamiflu) and zanamivir (Relenza), which are antivirals used to treat and prevent influenza.[4] The methods have been modified to address the risk of reporting bias, and the review is the first of its kind to be published in the Cochrane Database of Systematic Reviews. The review was supported by funding from the National Institute of Health Research in the UK. The methods used to locate, appraise and summarise the relevant evidence have been developed by the review team after they became aware of multiple clinical studies whose results had never been published in scientific journals.[4]

The Cochrane Collaboration encourages authors of Cochrane Reviews to search for and incorporate information from both published and unpublished sources.[5] Jefferson et al had included in their previous review a small subset of all trials conducted in this area, casting some doubt over the reliability of their conclusions. The authors therefore developed an alternative strategy, going beyond the information contained in published articles based on the trials of neuraminidase inhibitors to longer, more detailed 'clinical study reports' (CSRs). These documents provide much fuller accounts of the study and contain information that often does not get included in the six to eight pages that are typical of published journal articles. Indeed, CSRs frequently run to thousands of pages in length.[4] In view of the amount of information they can provide, their value to a systematic review is substantial.[6]

The authors have been assiduous in their attempts to compile the full set of clinical trial data for the published and unpublished studies. They have spent many hours sifting, reading and annotating pages of data and commentary obtained from drug licensing authority websites, Freedom of Information requests, and study sponsors. By this approach the authors identified 67 trials conducted in people with an influenza-like illness. They hoped that finding such a large number of studies would help generate more reliable and definitive answers than relying on studies published as journal articles.

So does the review provide us with any greater assurance of the efficacy of neuraminidase inhibitors than we had previously? Crucially, the authors have not been able to access the entire set of documents they need to carry out their planned analysis. Only 25 of the 67 trials (38%) identified contribute any data to the analysis of outcomes in the review. The findings we do have confirm that oseltamivir shortened the time to alleviation of symptoms by around 21 hours compared with placebo, which is concordant with the effects reported elsewhere.[7] However, the review did not find evidence of a reduction in hospital admission, and the effect of oseltamivir on preventing complications of influenza could not be assessed.

Despite the monumental task that the authors have completed, many uncertainties remain unresolved, and more have surfaced as a direct result of their efforts. The questions now raised by the review are, if anything, more thought-provoking than the answers it provides. The evidence that has emerged questions the mechanism of action of these drugs, and in particular that of oseltamivir. The consistent imbalance in the number of people with raised influenza antibody levels between treatment groups in the oseltamivir treatment trials (the influenza-infected population) runs contrary to the hypothesis that oseltamivir does not affect antibody production. To further compound this problem, much of the data available to the authors came from the study participants with raised antibody levels rather than all participants randomised.

Looking beyond the findings of this review, we now need to ask which other review questions are likely to necessitate such a fundamental re-evaluation of methods, and what steps should be taken to address the issues the review has raised. Does the inclusion of data obtained from regulatory agencies change the results of meta-analyses? Over time, will the maturation of trial registration repositories and better disclosure of data reduce the need for approaches that include exploration of licensing agency data? Should data obtained through litigation related to drugs be searched for and included in meta-analyses? And even with such a body of documents accessible to a team of researchers, is such an approach feasible without a substantial increase in the investment of time and money?

There is some way to go before the methods used in this review become routine for Cochrane Reviews. The Cochrane Collaboration is supporting a new research project that aims to identify when exploration of regulatory and licensing agency data is needed, and the strategies for performing such work efficiently and effectively. In the meantime, the development of methods by Jefferson et al has undoubtedly elucidated an otherwise dark corner and has raised challenges for all those engaged in conducting systematic reviews. By so doing, Jefferson et al have met a core aspiration of Cochrane Reviews and a key principle of The Cochrane Collaboration: to identify and minimise bias.