The challenge with systematic reviews of non-randomised studies in urology

694 views

In this issue, BJU International has made the conscious decision to publish a systematic review (SR) and meta-analysis (MA) by Guo et al. [1]to inform the question of whether patients undergoing nephrouretectomy for upper tract urothelial carcinoma are at increased risk of worse oncological outcomes. This question was also the topic of a similar review by Marchioni et al. [2] published earlier this year in this journal. Both studies were submitted around the same time and underwent independent, parallel peer review that resulted in different editorial decisions. Given their similarity in methodological quality they both deserved similar consideration for publication, which the journal is hereby honouring.

As BJU International takes a lead in promoting high-quality SRs in urology, the journal has seen a considerable increase in the number of submissions, including SRs of non-randomised studies (NRS). Whilst much of what we practice on a day-to-day basis is based on evidence from NRS, studies of those designs have infrequently been included in the Cochrane Library, which has pioneered much of the underlying methodology. This is for a few reasons: First, the ‘garbage in–garbage out’ phenomenon; if the underlying individual studies only provide very low-quality evidence, combining these studies will rarely enhance the confidence we place in their results. Second, the need for methodological advances in the assessment and analyses of NRS. Third, when high-quality evidence from randomised controlled trials (RCTs) is available, it may be inefficient to review the NRS literature.

However, progress is being made on the methodology front. Members of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group are credited with having developed an approach for rating the quality or certainty of evidence from randomised and NRS to inform decision making [6]. While a body of evidence from RCTs, which starts as high-quality evidence, may be downgraded for study limitations, a body of evidence from NRS, which starts as low-quality evidence, may be upgraded for one of three reasons, most commonly for large magnitude of effect [7]. The underlying assumption is that, whilst we have to assume that bias is likely to be present in these studies, it is unlikely to explain the entire observed effect.

It nevertheless remains critical to assess the risk of bias of NRS. While the Newcastle-Ottawa scale (as used in both of these SRs) is a widely used instrument to evaluate risk of bias in NRS, it has critical limitations that the recently developed Risk Of Bias In Non-randomised Studies – of Interventions (ROBINS-I) seeks to overcome [8]. ROBINS-I evaluates NRS by using a standardised comparison to an RCT (i.e. target trial) [9]. In this way, ROBINS-I captures the bias inherent to studies without proper randomisation or allocation concealment, namely the lack of a balance of known and unknown confounders and selection of participants. ROBINS-I allows users to fundamentally start all studies at the same quality level, providing the transparency requested by some SR authors conducting SRs of NRS.

While ureterorenoscopy before nephroureterectomy may indeed increase the risk of intravesical recurrence as the authors suggest, additional exploration would be needed to make a statement about the causality of the relationship. Guo et al. [1] conducted sensitivity analyses to describe the potential for bias introduced by confounders of previous bladder tumour history and bladder-cuff management, thereby increasing our confidence that the observed effect may be closer to the truth. It seems equally important to note that Guo et al. found no increased risk in cancer-specific, recurrence-free or overall survival, which are other outcomes of potentially greater patient importance.

Understanding the inherent limitations of NRS, and placing their findings into appropriate clinical context are critical to the conduct of SRs. Moving forward, BJU International will continue to seek out the highest quality reviews that make use of the best, up-to-date methodology. We hope that these efforts will both serve as a beacon for the research community, but more importantly, result in improved evidence-based care for our patients.

Lack of randomisation leads to increased risk of bias and decreased accuracy/validity, resulting in erroneous evidence, quite critical to healthcare evaluation. Therefore, performing a systematic review (SR) on Non Randomised Studies (NRS) would necessitate a check on bias. ROBIS has been the first tool developed to assess the risk of bias with a domain-based approach using signaling questions, currently aimed at SRs involving intervention, etiology, diagnosis, and prognosis (http://www.jclinepi.com/article/S0895-4356(15)00308-X/fulltext).
Recently, ROBINS-I has finally filled the void for an acceptable domain-based assessment tool for NRS. Seven domains (2 at pre-intervention, 1 at intervention, 4 at post intervention) are used for assessing the type and interpretation of these domains (judgements) assesses the magnitude of risk of bias (http://www.bmj.com/content/355/bmj.i4919).

Please note that all submitted comments will be reviewed by the BJUI Web Team before they are considered for publishing on the site. Comments may take up to 48 hours to go live. If you have made a comment which has not appeared live after this time and you wish to discuss this matter further, please contact us.

BJUI Social Media

NEW: Infographics

Featured Video

This week’s featured video: Association between T2DM, curative treatment and survival in localized PCaClick here for more