Tuesday, 15 October 2013

The Matthew effect and REF2014

For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath. Matthew 25:29

So you’ve slaved over your departmental submission for REF2014, and shortly will be handing it in. A nervous few months await before the results are announced. You’ve sweated blood over deciding whether staff publications or impact statements will be graded as 1*, 2*, 3* or 4*, but it’s not possible to predict how the committee will judge them, nor, more importantly, how these ratings will translate into funding. In the last round of evaluation, in 2008, a weighted formula was used, such that a submission earned 1 point for every 2* output, 3 points for every 3* output, and 7 points for every 4* output. Rumour has it that this year there may be no money for 2* outputs and even more for 4*. It will be more complicated than this, because funding allocations will also take into account ratings of ‘impact statements’, and the ‘environment’.

I’ve blogged previously about concerns I have with the inefficiency of the REF2014 as a method for allocating funds. Today I want to look at a different issue: the extent to which the REF increases disparities between universities over time. To examine this, I created a simulation which made a few simple assumptions. We start with a sample of 100 universities, each of which is submitting 50 staff in a Unit of Assessment. At the outset, we start with all universities equal in terms of the research quality of their staff: they are selected at random from a pool of possible staff whose research quality is normally distributed. Funding is then allocated according to the formula used in RAE2008. The key feature of the simulation is that over every assessment period there is turnover of staff (estimated at 10% in simulation shown here), and universities with higher funding levels are able to recruit replacement staff with higher scores on the research quality scale. These new staff are then the basis for computing funding allocations in the next cycle – and so on, through as many cycles as one wishes.
This simulation shows that funding starts out fairly normally distributed, but as we progress through each cycle, it becomes increasingly skewed, with the top-performers moving steadily away from the rest (Figure A). In the graphs, funding is shown over time for universities grouped in deciles, i.e., bands of 10 universities after ranking by funding level.

Simulation: Mean income for universities in each of 10 deciles over 6 funding cycles

Depending on specific settings of parameters in the model, we may even see a bimodal distribution developing over time: a large pool of ‘have-nots’ vs an elite group of ‘haves’.
Despite the over-simplifications of the model, I would argue that it captures an essential feature of the current funding framework: funding goes to those who are successful, allowing them to enter a positive feedback loop whereby they can recruit more high-calibre researchers and become even more successful – and hence gain even more funds in the next round. For those who are unsuccessful, it can be hard to break out of a downward spiral into research inactivity.

We could do things differently. Figure B shows how tweaking the funding model could avoid opening up such a wide gulf between the richest and poorest, and retain a solid core of middle-ranking universities.

Simulation using linear weighting of * levels. Each line is average for institutions in a given decile

Figure C, on the other hand, shows how a formula that predominantly rewards 4* outputs (weighting of 1 for 3* and 7 for 4*, which is rumoured to be a possible formula used in REF2014). This would dramatically increase the gulf between the elite and other institutions.

Simulation where 4* outputs get favoured. Each line is average for institutions in a given decile

I’m sure people will have very different views about whether or not the consequences illustrated here are desirable. One argument is that it is best to concentrate our research strength in a few elite institutions. That way the UK will be able to compete with the rest of the world in University league tables. Furthermore, by pooling the brightest brains in places where they have the best resources to do research, we have a chance of making serious breakthroughs. We could even use biblical precedent to justify such an approach: the Matthew effect refers to the biblical parable of the talents, in which servants are entrusted different sums of money by their master, and those who have most make the best use of it. There is no sympathy for those with few resources: they fail to make good use of what they do have and end up cast out into outer darkness, where there is weeping and gnashing of teeth. This robust attitude characterises those who argue that only internationally outstanding research should receive serious funding.

However, given that finances are always limited, there will be a cost to the focus on an elite; the middle-ranking universities will get less funding, and be correspondingly less able to attract high-calibre researchers. And it could be argued that we don’t just need an elite: we need a reasonable number of institutions in which there is a strong research environment, where more senior researchers feel valued and their graduate students and postdocs are encouraged to aim high. Our best strategy for retaining international competitiveness might be by fostering those who are doing well but have potential to do even better. In any case, much research funding is awarded through competition for grants, and most of this goes to people in elite institutions, so these places will not be starved of income if we were to adopt a more balanced system of awarding central funds.

What worries me most is that I haven’t been able to find any discussion of this issue – namely, whether the goal of a funding formula should be to focus on elite institutions or distribute funds more widely. The nearest thing I’ve found so far is a paper analysing a parallel issue in grant awards (Fortin & Curry, 2013) – which comes to the conclusion that broader distribution of smaller grants is more effective than narrowly distributed large grants. Very soon, somebody somewhere is going to decide on the funding formula, and if rumours are to be believed, it will widen the gap between the haves and have-nots even further. I'm concerned that if we continue to concentrate funding only in those
institutions with a high proportion of research superstars, we may be
creating an imbalance in our system of funding that will be bad for UK
research in the long run.

11 comments:

On the focus resources vs spread them widely: just on the science of it, you actually want distributed activity across a range of people in order to allow for true innovation. There are many psychologists doing work I think is wrong but at least they're doing it well in case I'm wrong!

I know not all the people in a given department think the same way; but the loss of diversity strikes me a real problem too.

Liz Pellicano sent me this comment, which she had difficulty posting from an airport!

Excellent post, Dorothy, as always. The issues you raise here are critical for the future of higher education. The Institute for Public Policy Research (IPPR) recently published a report on the future of HE, which thought about the relationship between different kinds of universities and was written by VCs of various different places. It is downloadable here: http://www.ippr.org/publication/55/10847/a-critical-path-securing-the-future-of-higher-education-in-england

The consolidation of resource in science funding generally seems to have been a theme of the past 5-10 years in the UK. The previous RAE funding formula was strongly weighted toward 4* work as you observe (0,1,3,7), and there has been widespread speculation over the past few years that 2* work won't be funded at all in this REF. In this period, we've also seen the almost total devolution of studentship funding to a select group of institutions, and the same is happening for knowledge transfer funding which is increasingly being devolved to the "biggest and best" institutions (consider e.g. the EPSRC impact acceleration accounts). Sadly, it looks like research grants are also going in this direction. Research councils want longer and larger bids, and even the charities like the idea of putting their eggs in fewer baskets (e.g. Wellcome Trust). In this context, I wouldn't be surprised at all to see a 0,0,1,7 funding formula in this REF. Sadly, I think that much of this is driven by the desire to reduce transaction costs (science funding is much cheaper to administer if you are only dealing with 10 'customers'), and little thought has gone into how it might actually affect the quality of UK science and higher education. Further, given that the "biggest and best" institutions benefit from policies that seek to consolidate resource, and so can hardly be expected to argue against them, it is very hard to see the tide turning anytime soon ....

Giovanni Abramo at the Italian National Research Council has published quite extensively on this issue in the Italian national assessment system. In one of his papers, I recall him essentially asserting that it is logical (but politically very brave) to only block fund 'excellence' in research universities. I think the paper I'm recalling was published within the last 5 years or so, and contained analysis not dissimilar to your own. I hope this is helpful!

The head of the main German research funding agency DFG said openly years ago that this Matthew effect is the official strategy to distribute money. Maybe UK officials just shy away to announce it? But in Germany at least the funding of Higher Education is guaranteed and not so much linked to research income. I am afraid that the Matthew effect in combination with the current government strategy will harm the international reputation of UK Higher Education in the long run.

Dorothy I don't understand why you see this as a bad thing. The REF is supposed to generate positive feedback, and positive feedback accentuates differences. If you want a funding system that prevents departments from differentiating you need negative feedback, i.e. to give the most money to the worst departments.

I’m gonna go out on a limb here and disagree with my own PVC (sorry, Andrew). On balance, I think that a funding regime that disproportionately rewards 4* research is probably a bad thing, even if REF scores were completely accurate (correlated with “real” quality at r = 1.0). A system that assumes a "quite good" institution is unable to get better, and thus denies them the funds that would enable them to get better, is probably not an optimal system for promoting merit. A system that rewards in proportion to merit would at least be able to recognise and reflect the dynamism of university research; research groups wax and wane as people come, go, get disheartened, get re-invigorated.

However, this view is based on the assumption that REF is a system for rewarding and promoting merit not a system for implementing research policy long-term on a national scale. If it is actually a system designed to implement research policy, and if that policy is to bring about the concentration of funding in a small number of elite institutions, the formula of predominantly rewarding 4*s will have precisely the desired effect. Dorothy’s calculations demonstrate that quite clearly. Of course, whether you think this policy is a good idea is another matter ...

Thanks Caro - I was just about to reply to Andrew and then realised you had said pretty much what I would say. Yes, any system that rewards success will, in the long term, create divergence, and that could be regarded as a good thing. But it is matter of degree - rewarding only 4* makes divergence much more extreme much more rapidly, and also makes it impossible for an institution to recover easily from one bad round.But, in any case, although I'm sure my own views are evident, my real point in the post is that the funding formula will be imposed from on high, and I think we should be discussing what we actually think the point of the REF is. Maybe most people do think that funding should only be concentrated on an elite, (though if so, we could just give all the money to the golden triangle now and be done with). Personally I think the academic health of the country would be improved with a bit more diversification of funding - as argued by Andrew Wilson above. I'd like to see more of a national discussion of this.

I don't disagree with the point that I understand the two of you to be making here. Attempts to concentrate QR funding even more than it is currently, will probably damage the research base. In the latest funding allocation more than half the money goes to 10 of the more than 120 institutions that get funding. I second Dorothy's call for a wider debate on the question of how much we should concentrate research funding but I suspect that there are powerful voices that would like to restrict funding even more and even to break the link with research assessment which has the inconvenient property that it keeps showing that excellent research is done in institutions that many would like to dismiss.

For this reason I think that the broader complaint that I understand you to make here and in your post in March, Dorothy, that the REF is dangerous and wasteful and should be abandoned in favour of something simpler is wrong and dangerous. I can't think of a better way to improve the quality and usefulness of university research than to allocate funding on the basis of a systematic assessment of the quality of research outputs and the economic and social impact of research. Everything else is detail. Universities may do stupid things in their preparation for the REF and the REF may be expensive but we should not abandon the REF or the principle of funding the best research because of that.

Hello, think it's a really interesting discussion. Just a note on the funding formula, yes it started as something like 9:3:1:0 in the first funding years based on RAE2008, but it has already moved to 3:1:0:0 (that's 4* on the left) over the last six years, so currently, HEIs are receiving no money for 2* and below from the RAE2008 results. (That's why there was not much incentive to return work believed to be below 3* to REF). Actual formula here, see last spreadsheet: http://www.hefce.ac.uk/data/year/2014/201415qr/#d.en.86840