Experts will assess UK research 'impact' to award funding

UK universities will soon be paid depending on the impact of their academics' research - not just its quality.Purestock

UK research funders have struggled during the past four years to work out how to measure the benefits of research to society and the economy, as part of reforms to a system for judging science in UK universities. But the Higher Education Funding Council for England (HEFCE), the body in charge of the reforms, says that it has now hit on a feasible method.

A year-long pilot study published today finds that using peer-review panels to judge the 'impact' of research is "workable" and "robust".

"We view it as a success," HEFCE's director for research, David Sweeney, told reporters at a press conference.

The conclusions were keenly awaited by UK university scientists, as by 2014 the method is set to become a new addition to the nation's research audit system, the results of which are used to divide more than £1.5 billion (US$2.4 billion) per year in public funds between universities. Research impact is expected to contribute up to 25% of a university department's overall rating of research quality.

Many institutions say that their initial reservations about the scheme, including that the added focus on research impact would skew funding towards more obviously applied research, are now resolved. But other concerns remain about how the system will operate.

How to measure impact

The planned system — called the Research Excellence Framework (REF) — will replace the Research Assessment Exercise (RAE) as the nation's research audit, which lacked judgements of research impact on society.

In the impact pilot study, university departments were asked to submit case studies — one for every ten of their academics — describing the impact of the research that academics had conducted during the past 17 years. These were reviewed by academics and industry scientists on subject-specific panels, and awarded rankings ranging from 4 * (the best) to unclassified.

In the pilot, 11 university departments took part in a physics exercise, and 10 took part in assessments of clinical medicine and of Earth systems and environmental science. (Assessments of English literature and language and of social work were also included.)

The impact measures considered in the pilot included the establishment of spin-out companies, clinical trials or the development of drugs; impact on policy relating to the environment; or the development of industry-specific products and services, such as computer software or technology.

"Once we got over the shock of what we had to do, we were more comfortable," says Anna Grey, research manager at the University of York, UK, who was in charge of the institution's participation in the pilot study for physics.

According to the peer-review panels, "expert review of case studies is an appropriate means for assessing impact."

Jonathan Grant, president of RAND Europe, a research consultancy based in Cambridge, wrote a report last year criticizing the REF, but now says, "If you are going to measure impact, this is the way to do it."

Remaining concerns

But there are some unsolved difficulties. Although Grey says that the pilot had addressed many of her university's concerns, she added that difficulties could still arise over describing impacts in which universities have collaborated with industry.

Some of the University of York's industry partners were not happy to release the details it needed to demonstrate impact, she says. These included financial savings made as a result of products or services developed by the academics at the university.

"Unless we can prove to the companies that the information will remain confidential, we will struggle to get hard evidence of impact," she says.

Other issues include how far back to judge impact. The pilot study's findings recommend that academics be allowed to submit research for the assessment that they did between 15 and 25 years ago. For physics, Grey wants the full 25-year window.

A further uncertainty is exactly how much the impact assessment contributes to the overall REF. Grant says it should be between 10% and 20%, not 25%.

The cost of the new audit framework is unclear, Grant adds. HEFCE told Nature that it expects the REF to cost the same to run as the RAE — about £60 million per assessment. But HEFCE did concede that in the REF "there will be some additional effort involved in evidencing and assessing impact" for universities, over and above the data they had to gather for assessment in the RAE.

Rodney Phillips, an immunologist at the University of Oxford, UK, led his university's participation in the clinical-medicine pilot study and helped to judge submissions to the scheme. Rather than reducing the workload for academics — the government's stated motivation for the reforms — he says that the scheme "will be more work than the RAE".

ADVERTISEMENT

Peter Main, director of education and science at the Institute of Physics in London, says that the general concerns about the procedural changes caused by the introduction of an impact measure to the REF have not been resolved. "Universities take research funding very seriously, and could put pressure on departments to continue research in a specific area to reap benefits from advances made a long time before, even when more future impact might be generated from new directions," he says.

HEFCE says that it will discuss the recommendations over the next few months, and expects to announce its decisions on the final form of the exercise by February 2011.