Discrepancies in Risk of Bias Judgments for Randomized Trials of Acupuncture Included in More Than 1 Cochrane Review

Objective To assess consistency in risk of bias judgments for randomized trials of acupuncture included in more than 1 Cochrane Review.

Design We identified randomized trials of acupuncture that appeared in more than 1 Cochrane Review and retrieved all risk of bias judgments for these trials. We assessed the consistency of judgments (high risk of bias, low risk of bias, and uncertain) for the 5 domains in the Cochrane risk of bias tool: random sequence generation, allocation concealment, blinding, incomplete outcome data, and selective reporting. Reviews that did not report all 5 domains were included in the analyses of the domains they did report.

Results We identified 90 Cochrane Reviews that included at least 1 randomized trial of acupuncture, comprising a total of 1692 trials. After checking the reviews, 31 trials were identified in more than 1 review (in a total of 28 Cochrane Reviews). Thirty trials appeared in 2 reviews and 1trial appeared in 3 reviews. For all 31 trials, we found a total of 121 judgments for the 5 domains. Overall, 50% (60 of 121) of these judgments were different (Table), with most of these differences being the categorization as uncertain in 1 review but high or low risk in another. Relatively good agreement was found for random sequence generation (68%) and incomplete outcome data (63%). Five of the 9 discrepant trials with incomplete outcome data had the most extreme inconsistency (ie, high risk of bias in 1 review but low risk of bias in the other review). Agreement was 52% for allocation concealment but only 27% for blinding and 36% for selective reporting.

Conclusions Use of acupuncture as example of the assessment for bias in Cochrane trials may be a limitation of this study given the concerns about blinding in trials of acupuncture. However, this analysis shows that there are large discrepancies in risk of bias judgments between Cochrane Reviews that assessed the same acupuncture randomized trial, which may cast doubt on the much commoner situation, when a trial is assessed once only. Further work is needed to improve the application of the Cochrane risk of bias tool. The collation of judgements for all randomized trials in a central, standardized database of risk of bias may be helpful.