A New Approach to Research Evaluation

By Vaughan Beck

The system used to assess the quality of Australian research needs refinement to recognise the value of applied research.

Despite considerable progress in developing the Excellence in Research for Australia (ERA) scheme, it still runs counter to the Australian government’s innovation policies because of its focus on “pure” research that advances knowledge – to the detriment of “applied” research that targets problem-solving and opportunity creation.

The ERA is principally concerned with academic recognition through publication and citation measures, and is an effective mechanism to assess the quality of research in academic institutions. Highly cited research is a key determinant of a university’s international reputation and a factor in its ability to attract students both locally and internationally.

However, the current ERA measures are wholly inadequate to assess quality or national worth in the full range of applied research. Their restricted scope will force research institutions increasingly to concentrate on research aimed at meeting the ERA’s academically driven publication criteria.

This situation is potentially damaging to the government’s innovation agenda, which aims to foster greater collaboration between those performing higher education research, mostly at postgraduate professional level, and organisations with the potential to use the outcomes of this research – put simply, the closer integration of research and industry.

The Australian community expects to see a greater dividend from its investment in research. It wants to see new products, new jobs, new business opportunities and new exports.

The problem caused by the current focus on the quality of publications also threatens to impact undergraduate education. Universities will recruit future academic staff on the basis of their ability to write peer-reviewed papers rather than their experience outside academia or their understanding of what the future employers of their students require. It will be particularly harsh on many of the newer and regional universities whose main research and education focus is serving the needs of their local communities.

Several options are available to overcome this situation. One is to suggest separate funding to universities that rewards collaboration and outcomes arising from collaboration. Whether this is a new fund (like the UK’s “Third Stream”) or an enhancement of existing funding (e.g. ARC Linkage or CRCs) is a matter of detail but is highly desirable.

Several frameworks have been developed to assess the quality of applied research. For example, as an extension to the UK’s Research Excellence Framework (REF), the UK research funding bodies have decided to identify and reward the impact that excellent research has had on society and the economy, and to encourage the sector to build on this to achieve the full potential impact across a broad range of research activity in the future.

During 2010 a pilot exercise was conducted to test the feasibility of assessing research impact, and to develop the method of assessment for use in the REF. Following the completion of the pilot exercise, decisions have been taken on a broad framework for the assessment of impact and its weighting within the overall REF. Membership of the pilot expert panels included a broadly even mix of practising researchers and research “users” from business, public sector bodies, charities and other third sector organisations.

By employing case studies and expert opinion via pro forma documentation (perhaps including user evaluation) and by defining a number of application categories under which the research can be addressed, the problem in measuring and ranking applied research outputs can be addressed.

Collaboration to encourage industry take-up and commercialisation of research is too important to overlook in any assessment of the ERA system, despite evidence that impact can be measured relatively effectively and relatively cheaply.

For Australia to receive an increased dividend from its R&D funding, the answer may be to institute separate funding to universities and other publicly funded research organisations that rewards applied research, collaboration with industry and commercial outcomes.

Dr Vaughan Beck FTSE is Executive Director – Technical at the Australian Academy of Technological Sciences and Engineering (ATSE) and former Pro Vice Chancellor of Victoria University.