Performance indicators for science – Increasingly diversified, distinguished and European research and development

Amid the many changes that are shaking the foundations of Finland’s scientific community, it is important to understand what Finnish science actually entails. Recent hot topics have included proposed budget cuts, initiatives to streamline the role of universities, choices of strategic priorities, and universities’ performance targets.

Indicators have become increasingly important as measures of performance. The number of degrees awarded, innovations created by universities and research organisations, and scientific articles published are only a few examples of the kinds of indicators that we use today to understand the value that science adds to society. In our view, ranking the performance or productivity of organisations on the basis of indicators provides a very narrow perspective on science. We believe that, instead of focusing on specific indicators, discussions on the status, standard or performance of science should be based on understanding more comprehensive, systemic changes. We echo the criticisms voiced by Professor of the Sociology of Education Osmo Kivinen and Researcher Juha Hedman: Using performance indicators as the basis for evaluating the performance of science creates a risk of misunderstandings and decisions that can cause irreparable damage.

Our Co-evolution of knowledge creation systems and innovation pipelines (CEK) research project is aimed at exploring the background of indicators in order to better understand the changes that are taking place in Finland’s scientific community. We also strive to draw qualitative conclusions on the basis of these performance indicators. The following are four observations that we feel have received too little attention in indicator-based debates.

First and foremost, Finnish science has become increasingly diversified. Traditionally, Finnish science has been built around two fields: medical research and natural sciences. Now, however, these are being joined by a new, internationally significant field, which comprises elements of information technology, social sciences, and economic research. This new combination of research fields has been the biggest growth area of Finnish research, in terms of the number of international scientific publications. Although it is impossible to tell which of the currently growing research areas will feature the most prominently on the map of Finnish science in the future, this change already shows, on a general level, that Finland’s scientific community has begun to prioritise themes that have relevance for the economy.

The diversification process has also brought about an important organisational change, which leads us to our second observation: New research units with new research themes have joined the front line of research and broken Finland’s tradition of focusing on cutting-edge medical and natural sciences research. It appears that this is a concious choice by the scientific community. Our results highlight several universities and research institutes that have, by making choises, been able to move closer to the front line of international research. In other words, the pursuit of international competitiveness has already forced universities to make choices and redistribute their resources. Whether individual universities actually need to stand out in this manner is another question.

Global competition has made it imperative for researchers to engage in extensive international cooperation. Cooperation has been seen not just as a way to bring more know-how to Finland but also to promote cutting-edge research. Our third observation is the fact that our findings indicate a complex relationship between international research cooperation and the standard of research. Although the volume of international research cooperation is increasing, our findings suggests that we should question whether current investments will lead to internationalisation and giving Finnish researchers a marginal role in international top-level research projects, and whether these outcomes are worth pursuing at the expense of developing Finland-based research teams. It is not at all clear which kinds of international partnerships add the most value for Finland.

Finally, although we were unable to draw unambiguous conclusions on the effect of international co-authorship on the quality of Finnish research, our findings clearly point to increasing integration within the European scientific community. This has happened at the expense of North American research. The data used by Finnish researchers are more and more exclusively based on European scientific studies, which further reinforces the integration of European science.

In conclusion, Finnish science has become increasingly diversified, distinguished and European.

Analysis of outputs, such as scientific articles, can generate valuable information to support evaluations of the performance and impact of science. As our examples show, interesting conclusions can be drawn if indicators are used instead of outputs as variables in further analyses. The risks associated with an indicator-led science policy stem from inadequate understanding of the context of the results, genuine misunderstandings, and the vulnerability of indicators to manipulation. We argue, for example, that researchers are highly adaptable to the introduction of mechanical performance indicators and science policy coordination more generally. This is why new performance indicators and steering mechanisms should incorporate a critical assessment of the side-effects of the scientific community adapting to these indicators and mechanisms. For example, if we begin to reward researchers for new patents, we are sure to get more new patents. Whether this benefits the standard of research is questionable.

Finland is currently facing substantial social restructuring, and the process will make the performance of our research and innovation system an increasingly important factor in political decision-making. However, decisions concerning science and innovation policy should be based on scientific, reliable information. The proposed changes to research spending, the roles of research organisations, and the balance between basic and applied research need to be accompanied by quantitative indicators that generate information about how the scientific community is changing. Evaluations of the impact or standard of research, as well as conclusions, must nevertheless always be based on (peer-reviewed) qualitative analysis.

Suominen and Toivanen are research scientists in VTT’s Innovations, Economy and Policy team. Suominen is also a post-doctoral researcher at the Academy of Finland, where he is working on a project called Modelling Science and Technology Systems Through Massive Data Collections.