Day 9: Evidence in the real world

The blogs over the past two weeks have demonstrated that embedding rigorous evidence in decision making is not always a straightforward task. As the quote below shows, this is further complicated by data not always showing a single course of action for decision makers to take.

"You say 'evidence'. Well, there may be evidence. But evidence, you know, can generally be taken two ways" - Dostoevsky, Crime & Punishment, 186

Then there may be other instances when the evidence may not yet be available to provide specific solutions or guidance. Systematic reviews are seen as an effective way of putting research studies in a scientific context. However, a key criticism is that they “often conclude that little evidence exists to allow the question to be answered”[5] (although there is a counter-argument that this a useful finding in its own right). If further analysis is then needed, “how can the need for rigour be balanced with the need for timely findings of practical relevance?”

Identifying effective programmes and policies are not the end result of course. We have already noted the need to drive the demand for such evidence, but we also need to improve our ability to ensure that they are implemented with fidelity[6] to the original model to help increase their chances of success when they are rolled out. For instance, if a programme is evaluated and found to be effective, can we rely on these findings to implement it in a different area or context? Improving our understanding of implementation science is crucial.

Even when a programme has been successfully identified, implemented and scaled, the need for evaluation does not end. We need to ensure that the programme or policy continues to be effective. The most intensive work is arguably over, but we should still ensure that its impacts are as optimal as they could be. Yet we need to ensure that we recognise what types of evaluations are needed at different stages. The National Institute for Health and Clinical Excellence[7] (NICE) has a grading system which starts at ‘very low’ to indicate where any estimate made is uncertain, up to ‘high’ which shows that any further research is unlikely to change confidence in estimates of effects. Do we need similar systems in other areas to ensure the most appropriate evaluations are being undertaken at different stages, and prevent efforts in those areas where further research may not reveal anything new?

What has become clear over this blog series is that we need to be strengthening the supply and generation of research, as well as the demand for it. The next blog post will discuss the development of the UK Alliance for Useful Evidence[8], a new initiative being developed to play this role. The Alliance for Useful Evidence will explore the infrastructural improvements and changes that are needed to embed rigorous evidence in decision making across social policy and practice.

As with all the challenges we have discussed over the past few days, those outlined here should not excuse rigorous testing and evaluation. Instead we hope that they set the need for rigorous research to be set within context.