The automation of systematic reviews

Re: The automation of systematic reviews

Optimism is important but reality matters most. We agree with this editorial (1) : comprehensive, complete and up to date systematic reviews are desirable. However, automating systematic reviews to do this is a utopian aim that is flawed in its dismissal of language, its scientism, and a mis-placed faith in technology.(2)

That the authors describe systematic reviews as involving ‘basic tasks’ of retrieval, appraisal, synthesis and publication lays bear simplistic misconceptions. Language in systematic review is inescapable and necessarily involves interpretation.(3) For example, descriptions of health interventions are often messy, inconsistent, and incomplete. These descriptions themselves are interpreted when re-described, classified or quantified.(4) Appraisal of bias in studies, though protocol-led, still involves all manner of interpretations and a fusion of judgment, nuance and fairness that computers cannot provide. Finally, conclusions are never inherent in pooled data: do null pooled findings indicate there is ‘no evidence to support an intervention’ or should this intervention ‘not be practiced?’(5) Arguments that these difficulties can be addressed via superior technology, better or more rigorous methods and / or the removal of humans altogether are predictable and amiss. The belief that method and technology can conquer the challenges created by language, as the authors do, is historically common but wrong.(6)

Humans should always be involved in systematic reviews. While ‘technology’ can assist them identify relevant studies, classify these using taxonomies, appraise their quality via tools and calculate synthesized results software – what studies, data and conclusions mean is always about interpretation and needs to involve humans. To do so is not to admit failure in method or science or technology but to recognize properly the nature of language and reality.