#OxfordImpact2018

The most memorable sentence from this one-day workshop held at Jesus College, Oxford, on 21 June 2018 was from keynote speaker Mark Taylor. Reflecting on his experience as a patient with multiple sclerosis, he said: “I am old enough to have lived through two NICE guidelines on my condition. Neither of them was relevant to me.”

Mark’s comment illustrated one of the central challenges of what has become known as the ‘impact agenda’ in healthcare research. The gap between what science delivers and what many patients need remains wide. Good research does not, in and of itself, produce good impact for the people intended to benefit from it.

The workshop, which was funded jointly by the Oxford Biomedical Research Centre and the UK Cochrane Collaboration, brought together patients and patient advocates, policy makers, clinicians, service managers, knowledge brokers, research funders, frontline researchers and university strategists to “think differently about research impact”.

Most of the day was spent in interdisciplinary small group work, in which participants were encouraged to challenge current definitions and metrics, try to close the gap between what is measurable and what is desired, and consider the counterfactual (“what would the world be like if we gave up trying to measure research impact?”).

Some groups were lucky to include some talented artists, who captured ideas in visual form.

By the end of the day, we had papered the walls with flip chart paper and multi-coloured post-it notes.

Despite the best efforts of some individuals to drive us to consensus, we did not agree either on a definition of what research impact is (most people felt that it will always mean different things to different stakeholders) or on a single way of achieving it (many groups emphasised the need for plurality of approaches).

There was, however, near-unanimous agreement on a number of principles. For example:

The potential impact for a wide range of end users should be considered at the outset of a research study and borne in mind throughout the study and beyond it. There was much discussion on the benefits of a co-design (‘Mode 2’) approach in which multiple stakeholders (patients, clinicians, policy makers) are involved from the outset in setting the research agenda and co-producing the findings.

Even when research impact is a high priority from the outset, achieving it is a messy, iterative, pragmatic and unpredictable challenge whose methodology cannot be tightly prescribed.

Achieving research impact beyond the standard academic outputs (journal articles and conference presentations) depends critically on cross-sector relationships and trust – which need to be built over time and nurtured by engagement activities. This ‘soft’ activity underpins effective impact and needs to be resourced and valued.

Whilst the patient’s need for meaningful improvements in survival, morbidity and quality of life has to be central to the impact agenda, it is rarely clear how this ideal can be aligned with organisational and policy realities. Research will have limited enduring impact, for example, unless the service has the capacity to accommodate change (for example, training staff and introducing new service models) and the next generation of researchers and knowledge workers are successfully trained.

At a final plenary session, we used a nominal group process to identify the most important issues to take forward as ‘impact challenges’.Below, we list the top five, along with an unanswered question for each:

Impact Challenge 1: better use of routinely-collected data

Currently it is difficult, if not impossible, to demonstrate and track research impacts using NHS data. The unanswered question identified was “How might we work better with health organisations and services to maximise the secondary use of routinely collected data in impact narratives?”

The primary use of NHS data – at an individual level – is of course to identify disease and assess its progress of disease, including whether an individual patient has benefited from a particular intervention. Aggregated data from these same NHS records can be fed into organisational learning to ask (for example) what proportion of a denominator population have been reached by an intervention that is based on a particular research finding – and what proportion of those have actually benefited.

As electronic patient records become more detailed, and as big data analytics become more sophisticated, the potential for impact narratives to be enriched by high-quality NHS-generated quantitative data is increasing.

Impact Challenge 2: optimising learning from impact narratives

The science – and art – of generating case studies of research impact is currently in its infancy. The unanswered question here is: “How can we make impact narratives more authentic, reproducible and compelling?”

We know that both qualitative and quantitative data (‘stories and numbers’) need to be combined to produce a detailed and auditable account of what happened and why. Workshop participants shared the view that narratives of research impact are extraordinarily difficult to produce but that, once crafted, can provide rich learning. Negative impacts and partial successes offer important learning opportunities for both individuals and organisations.

We should therefore avoid the temptation to sanitise our impact case studies into ‘good news stories’. We also need to create the space and engage key players to reflect, individually and collectively, on these impact narratives and tease out the implications for practice and policy.

Impact Challenge 3: democratising the impact agenda

The diverse attendees at the workshop had varied (and often strongly-held) views on the question: “Whose interests does the impact agenda serve?”

Those of us who work in the UK higher education sector stand to gain, organisationally if not personally, from a persuasive narrative of research impact shaped as a case study for the 2021 Research Excellence Framework. But as Mark Taylor’s opening pitch illustrated, high scores in the REF impact exercise (achievable by getting a prominent mention in a NICE guideline) may not translate to tangible benefits for patients.

Whilst everyone seemed to agree that impact metrics should be ‘patient-centred’, it is also the case that patients don’t run the REF. Huge strides in patient and public involvement activity in recent years notwithstanding, there were few concrete suggestions as to how to share power democratically with the ultimate beneficiaries of medical and healthcare research.

Impact Challenge 4: removing perverse incentives

Every system is perfectly designed to get the results it gets.

The unanswered question here might be phrased thus: “To what extent should researchers pursue a set of externally imposed impact metrics even though this pursuit distracts them from activities that would actually achieve more impact in the long term?”

On the one hand, most research is funded from the public purse and impact metrics are often designed to achieve the worthy goals of transparency and accountability. On the other hand, goals such as responsible and sustainable science in dialogue with society as articulated by the DORA group are not well captured by existing UK metrics.

Research policymakers and others who set and police impact metrics are keen to help bridge the gap between what is desirable for patients and society and what is measured (though it is widely recognised that not all desirable impacts are measurable). By bringing different stakeholder groups together, the workshop set the stage for some fruitful dialogue on how to adjust, curtail or even abolish existing impact metrics while also working to identify and pilot new (leaner, fitter, more socially responsible) metrics in different sectors.

Impact Challenge 5: capacity building

The unanswered question here is “How should we train and support the next generation of researchers and knowledge brokers in the impact agenda?”

Early and mid-career researchers, and people responsible for the training and development of such researchers, brainstormed a long list of practical skills for achieving impact (see picture). An impact curriculum, they felt, should be human-focused not technology-focused, and address both ‘hard’ (science-based) and ‘soft’ (humanities-based) skills.

Such a curriculum would likely require apprenticeship and mentoring as well as classroom-based (or online) teaching. It should include (but not be limited to):

How to influence people

How to convene, chair and record meetings

How to build and strengthen networks beyond academia

How to identify and build key strategic links with policymakers

How to communicate research findings to the non-expert

How to respond to policymakers’ questions in real time (when there is insufficient time to do research or even undertake an academically sound review)

How to understand the perspective of non-researchers (policymakers, patients, industry)

How to access, select, analyse, combine and synthesise real-world data sources for the purposes of to measure impact

How to write impact narratives

How to balance the costs of measuring impact with the benefits of doing so

Wrap-up and next steps

Towards the end of the workshop we called for radical ideas. Whilst we had fun “getting rid of academic journals” and “putting citizen juries in charge of metrics” over the afternoon pastries, most people present recognised that such suggestions would not be simple or straightforward to implement – nor were they likely to increase research impact without unintended consequences elsewhere in the system. There was much appetite to run the workshop again next year, with the following items already on the (provisional) agenda:

A keynote speaker who is an expert on how to use NHS data to measure the impact of research findings (please contact us with suggestions)

More on “value” and how to measure its many dimensions (but perhaps some specific input from health economists – ditto)

The opportunity to share examples of impact narratives, perhaps as work-in-progress, in oral or poster sessions

More people from industry to be invited (“they’re not the bad guys”)

At the time of writing, we have not yet sent out a light-touch evaluation of the day (which will include asking for more suggestions for next year). Subject to the analysis of that evaluation we hope to be inviting you to register for #OxfordImpact2019 in a few months’ time.