Monday, December 22, 2014

Bang for buck in REF 2014

The Research Excellence Framework 2014 evaluates research in terms of outputs (e.g., journal publications), impacts (e.g., how research translates to real-world impact) and research environment (e.g., facilities, funding, people). Universities try to maximise the proportion of 4-star and 3-star ratings they get which indicates world and internationally leading research. To the wider world, the only things that count are the end products of research: outputs and (research's indirect cousin) impacts. So what is the point of assessing research environment? One possibility is that it can be used to ascertain which departments are squandering resources and which departments are doing fantastic research on a shoestring, and using that information to guide funding decisions. For example, if a department is managing to produce a high proportion of 3-4* outputs on a 2* environment, then quality-related (QR) research money should be thrown at them because one can only imagine what they might achieve with better facilities. On the other hand, if a department has a 4* environment but its modal outputs tend to be in the 2-3* range, then serious thought should be given to holding back some of their QR money because they are clearly wasting their excellent research environment facilities on relatively mediocre research. How can we examine - and reward - greater bang for buck?

We attempt to quantify "bang for buck" by looking at the quality of research produced by departments (outputs and impacts) relative to the quality of their research environment. Are people in 4* research environments consistently producing 4* research outputs? Are people in lower rated research environments producing poorer quality outputs?

To examine this, we calculate each department's grade point average (GPA) based on their combined outputs (weighted at 65% in REF2014) and impact (weighted at 20%) and compare it to the GPA based on their environment alone. In short this is the score for outputs and impacts divided by the score for research environment. This gives us a "Bang for Buck" ratio, where a score of 1 indicates a university/unit is producing research in line with the quality of its environment. A score of greater than 1 indicates higher quality outputs relative to the research environment, while a score lower than 1 indicates outputs of a lower quality relative to the environment. As my interest is in psychology, I've done the analysis for Unit of Assessment 4 (Psychology, Psychiatry and Neuroscience) - if time permits I’ll add the analysis for universities overall later. I’m focussing on top research departments, which I've defined as those that received above average scores for their 3* and 4* outputs/impacts combined (the average score in UoA 4 is ~65%). Table 1 below shows the rankings according to the Bang for Buck ratio for the top research departments.

Table 1. Bang-for-buck scores for the top 46 institutions in UoA 4. Overall GPA is included for context, but not used in the calculation.

Rank

Institution name

Overall GPA

Environment GPA

Impact GPA

Outputs GPA

b4b

1

Nottingham Trent University

2.894

2.50

3.73

2.73

1.19

2

University of
Stirling

3.269

2.88

4.00

3.14

1.16

3

University of East London

2.931

2.63

4.00

2.67

1.14

4

Goldsmiths' College

2.981

2.75

3.60

2.84

1.10

5

University of Aberdeen

3.221

3.00

3.00

3.34

1.09

6

University of
Portsmouth

2.799

2.63

3.53

2.61

1.08

7

University of Dundee

3.182

3.00

3.80

3.03

1.07

8

University of
Warwick

3.290

3.13

3.47

3.27

1.06

9

University of East Anglia

3.099

3.00

3.60

2.97

1.04

10

University of Hull

2.805

2.75

3.27

2.68

1.02

11

University of Essex

3.302

3.25

3.53

3.24

1.02

12

University of
Plymouth

3.040

3.00

3.20

3.00

1.02

13

Roehampton University

2.785

2.75

3.40

2.60

1.02

14

Bangor University

3.278

3.25

3.73

3.15

1.01

15

Lancaster University

3.014

3.00

3.27

2.94

1.01

16

Queen's University
Belfast

2.987

3.00

3.73

2.76

1.00

17

University of Sussex

3.355

3.38

3.84

3.20

0.99

18

City University
London

2.728

2.75

2.80

2.70

0.99

19

University of Leicester

3.062

3.13

3.00

3.07

0.98

20

Swansea University

3.176

3.25

4.00

2.91

0.97

21

University of Surrey

2.858

3.00

2.73

2.86

0.94

22

Royal Holloway,
University of London

3.439

3.63

3.73

3.31

0.94

23

University of Kent

2.963

3.13

3.40

2.79

0.94

24

University of
Liverpool

3.138

3.38

3.80

2.88

0.92

25

University of Southampton

3.212

3.50

3.80

2.96

0.90

26

University of
Glasgow

3.205

3.50

3.20

3.14

0.90

27

Imperial College London

3.405

3.75

3.84

3.19

0.89

28

University of Oxford

3.623

4.00

3.86

3.47

0.89

29

University of Reading

3.043

3.38

3.68

2.77

0.88

30

University of York

3.460

3.88

3.47

3.36

0.87

31

University of Exeter

3.223

3.63

3.20

3.14

0.87

32

Cardiff University

3.522

4.00

3.80

3.33

0.86

33

University of Cambridge

3.508

4.00

3.64

3.35

0.86

34

University of Durham

3.038

3.50

3.27

2.86

0.84

35

Birkbeck College

3.462

4.00

4.00

3.17

0.84

36

University of
Bristol

3.191

3.75

3.80

2.87

0.82

37

University of Birmingham

3.400

4.00

3.68

3.18

0.82

38

University of
Nottingham

3.188

3.75

3.33

3.01

0.82

39

University of Leeds

3.066

3.63

3.36

2.85

0.82

40

Newcastle University

3.366

4.00

3.87

3.07

0.81

41

University of Edinburgh

3.349

4.00

3.82

3.06

0.81

42

King's College
London

3.329

4.00

3.78

3.04

0.80

43

University College London

3.307

4.00

3.71

3.02

0.80

44

University of
Sheffield

3.196

3.88

3.60

2.92

0.79

45

University of St Andrews

3.281

4.00

3.40

3.08

0.79

46

University of
Manchester

3.232

4.00

3.60

2.94

0.77

The first thing that strikes me is the relatively small number of UoA 4 departments that have a bang for buck (b4b) ratio above 1. Only 16 of the top 46 departments are producing better outputs/impacts than their research environments would suggest. The second thing I notice is that Oxford (.89) and Cambridge (.86), almost always considered to be in the top 3 research departments, are well down the table in terms of b4b. Now, Oxford and Cambridge departments could hardly be considered underachieving, given their impressive outputs and impacts, but the b4b scores suggest they could be doing still more given the resources at their disposal.

Of the 35 institutions in Table 1 that don't receive top marks for their environments, 25% of outputs and 58% of impacts are rated as 4*, indicating that huge swathes of world class research are done outside of traditional research powerhouses and elite institutions. For me, the real stars in this analysis are Aberdeen, Dundee and Warwick - all three have very impressive GPAs for outputs and impact, but are still scoring very highly on the bang-for-buck measure. On average, 35% of the outputs and 49% of the impacts from these three institutions are at 4* level, despite not getting top scores for their research environments (averaging only 8% at 4*).

Does size matter?

Scanning the table, it also seems that larger departments tend to do less well in terms of bang-for-buck. Table 2 shows the 10 largest departments (by number of staff submitted to UoA 4) along with their b4b scores.

Table 2. Bang-for-buck scores of the top 10 departments by size

Rank

Institution name

FTE Staff Submitted

b4b2

1

University
College London

286.6

0.80

2

King's College London

238.9

0.80

3

University of
Edinburgh

117.3

0.81

4

University of Oxford

98.3

0.89

5

University of
Cambridge

76.0

0.86

6

Cardiff University

69.3

0.86

7

University of
Bristol

68.8

0.82

8

University of Manchester

67.7

0.77

9

University of
Nottingham

53.2

0.82

10

Newcastle University

51.6

0.81

The mean b4b score of the top 10 largest departments is .825, meaning that the quality of research outputs and impact are considerably below what one might expect given their research environments. There is a general negative relationship between the size of department and the b4b score (r ~= -0.3) Are the departments in Table 2 too large to function optimally in terms of research outputs? Is there an ideal department size in terms of producing quality research consistent with the research environment? Table 3 shows the b4b scores for groups of departments ordered by size (e.g., the median number of staff for the top 10 largest departments is 72.64, the next 10 largest is 37.3 and so on).

Table 3. Groups of departments ranked by size with median number of active researchers and b4b scores

Rank
by Overall GPA

Median FTE

b4b

1-10

72.64

0.825

11-20

37.3

0.908

21-30

29.275

0.908

31-40

21.75

1.033

41-50

14.85

1.038

According to this data, the sweet spot for department size is somewhere between 15 and 22 active researchers. Beyond that, people are not making optimal use of their research environments. It's not clear why this would be the case - it may be because resource management becomes extra problematic over a given size or because a culture of resource wastage becomes normalised in larger departments. There may be many reasons for such a relationship.

What have we learned?

Overall, I think this analysis suggests a couple of very important points. It is clear that there is huge value in not focusing research funding in a small number of elite institutions. Whatever the flaws of the REF, the overall research profiles clearly indicate that there is high quality research taking place across the UK - there are 46 departments with >65% research considered in world-leading or internationally-leading. Furthermore, most of the world-leading research in the UK is taking place in departments that lack 4* research environments. What the bang for buck calculations show is that many departments are punching well-above their weight in terms of research produced and are demonstrating extremely good value for money. In addition, as departments increase in size there is a noticeable decrease in bang-for-buck scores.

In terms of allocating research funding (QR money), there are at least two ways to utilise this information. One way is to allocate extra money to those departments with high bang for buck ratios, because giving them the opportunity to improve their research environments will likely lead to matched improvements in research outputs and impacts. Another way is to disregard environment completely when allocating QR money, and focus only on outputs and impacts, which rewards departments purely by the research they actually produce, rather than factoring in the environment they already have in place.

No comments:

Post a Comment

About Me

Current research interests include embodied views of language processing, conceptual combination, linguistic creativity and issues of mental representation in general. Of late I've been looking at what an embodied cognition approach can tell us about people's decision making behaviour. I'm also interested in using statistical linguistic analysis to model and make predictions about human behaviour, decision making and language processing. See my homepage here.