Summary Effect Sizes in Meta-analyses After Removal of Retracted Studies From the Pool of Primary Studies

Daniele Fanelli,1 David Moher2,3

Objective This study aimed to assess the magnitude and direction of change of summary effect size in meta-analyses (MAs) after retracted papers are removed from the pool of primary studies.

Design We aimed to identify a homogeneous sample of recent MAs that contained, among primary studies, 1 or more studies that were later retracted, to compare pooled odds ratios with and without such studies. On December 16, 2016, we retrieved all retracted publications recorded in the Web of Science (WOS) and then retrieved a list of records that cited these retracted publications. We selected all records containing “meta-analysis” or “systematic review” in the title, abstract, or keywords and then restricted the initial list of potentially relevant titles to records published in 2016. The full text of these studies was retrieved and inspected for selection based on the following exclusion criteria: limited to a systematic review and not a formal MA; not a standard MA (ie, a weighted pooled summary of primary studies, which excludes network MAs, genome-wide association studies, and MAs of functional magnetic resonance imaging, microarray, and genomic data); does not contain primary summary data in the full text (ie, the retracted cited article is not among primary studies of the MA); or does not use odds ratio–convertible metrics (including risk difference, proportion, mean, or other unusual metrics designed for the specific purposes of a study).

Results A total of 3834 records of potentially retracted articles were identified in WOS. We retrieved 83,946 records that cited these potentially retracted publications; from these, we identified 1433 records containing “meta-analysis” or “systematic review” in the title, abstract, or keywords. Of the 109 potentially relevant MAs published in 2016, 17 did not match any exclusion criteria and were included in this study. Each of these MAs had included in its weighted summary 1 retracted study. Three pairs of MAs cited the same retracted study; therefore, the number of distinct retraction events covered in our sample is 14. All MAs had been authored by independent research teams, and only 1 author appeared in 2 MAs. Two MAs were published in the Cochrane Database of Systematic Reviews, and 15 were published in different journals that were classified by WOS in different biomedical fields, from molecular biology to surgery. Additional analyses are ongoing.

Conclusions The 17 MAs included in the study are representative of multiple biomedical research areas and retraction events. For each of these MAs, we will calculate summary effect size with and without the retracted primary study and obtain a ratio of odds ratios across the sample. Pooled results will yield a preliminary estimate of the possible impact that retractions may have on the biomedical literature.