No Time for Muses: The Research Excellence Framework and the Pursuit of Mediocrity

Anthony J. Stanonis, September 2016

Witnessing a growing government bureaucracy during the 1920s, American journalist H. L. Mencken unsheathed his pen. ‘It is the invariable habit of bureaucracies, at all times and everywhere, to assume . . . that every citizen is a criminal’, he wrote in a November 1926 edition of the Chicago Tribune. His pen took no quarter: ‘Their one apparent purpose, pursued with a relentless and furious diligence, is to convert the assumption into a fact. They hunt endlessly for proofs, and, when proofs are lacking, for suspicions’. Mencken’s commentary recognised the tension between the liberty of citizens and the social control undertaken by government, even within a democracy. The expansion of government oversight risked bureaucratic entanglements that restricted personal freedom. If unchecked, government would turn each citizen into a modern-day ‘subject’ with the bureaucrat as ‘lord’. Mencken was certainly prone to hyperbole, but his claims offer a worthy starting-point for evaluating the appraisal procedure known as the Research Excellence Framework (REF).

Systems for evaluating research within UK universities date to the Thatcher years, when government fully funded higher education. In 1985, Keith Joseph, the Secretary of State for Education and Science, published a green paper entitled The Development of Higher Education into the 1990s. The document called for a system to assess research within UK universities. Joseph eyed the vibrancy of universities within the United States. Patrick Minford, professor of applied economics at Cardiff Business School, explains, ‘Indeed, the USA was the model Sir Keith had in mind that the UK, then a state monolith, should get good enough to emulate and he hoped that, once fit, would become a market-driven sector where reputation and quality would get their market rewards’. He argues that rather than a temporary jolt, the evaluation process has evolved into a ‘huge bureaucracy [. . .] devoted to corralling academics to meet the aims of the exercise’. The appraisal process, born in the aftermath of the Falklands War, also reflected concerns with the decline of Britain as a global force. Not surprisingly, the REF, the most recent research appraisal process, fixates on the international reputation and supposed global impact of UK research. Research is also encouraged to address social and economic matters.

The REF in 2014 consisted of four main panels responsible for overseeing the thirty-six discipline-specific subpanels. These totalled 898 academic members and 259 ‘research users’. The REF panels had roughly two years to assess 191,150 ‘research outputs’ from over 52,061 full-time academic staff at 154 higher education institutions . Another 6,975 ‘impact case studies’ were also assessed. Each submission was graded on a 4-point scale: (4) ‘world-leading’ (3) ‘internationally excellent’ (2) ‘internationally recognised’ (1) ‘nationally recognized’. Some items went unclassified. Impact studies were ranked similarly: (4) ‘outstanding’ (3) ‘very considerable’ (2) ‘considerable’ (1) ‘recognized but modest’ and unclassified. These categories then provided a grade point average for institutions and their various schools.

Institutions were thereby ranked for their ‘research power’ calculated by multiplying a school’s grade point average by the total number of its full-time faculty. Universities thus have a vested interest in pushing each faculty member to publish enough for the REF – called being ‘REF returnable’ – and to target projects that might receive higher grades. Academic freedom is thereby constricted as university staff scheme for better REF scores while others with low scores or too few publications – no matter how excellent or even award-winning – risk being purged, a fate already confronted by a few of my colleagues. Furthermore, an anxious, bloated university bureaucracy fixated on REF-oriented metrics has increasingly smothered the university grounded on unfettered scholarship and trust.

Despite the nomenclature and pomp, the REF panels offered shoddy coverage and expertise for the purpose at hand. In REF 2014, 30 per cent of outputs were graded a 4; 46 per cent received a 3; 20 per cent garnered a 2; 3 per cent were given a 1. Only 1 per cent went unclassified. Divided per individual, this means that each panellist evaluated at least 187 articles, books, and other submissions. That amounts to 374 items if panellists worked in pairs to ensure greater fairness – as they do in double-marking significant work by students. This was all supposedly accomplished while panellists undertook their regular teaching, research and administrative demands.

For a process focused on situating UK scholarship in global perspective, the panels paradoxically lacked evaluators from outside the UK. International participation was practically non-existent. The four main panels consisted of a foreign membership of between 10 to 25 per cent. These foreign affiliations gave a veneer of legitimacy to the REF’s claims about identifying internationally important research. But the veneer crumbles upon examination of the sub-panels. None of the forty-one persons involved with Sub-Panel 30, covering the field of History, was from an institution outside the UK. Ironically, the same holds true for the thirty-nine persons who worked on Sub-Panel 28, covering the field of Modern Languages and Linguistics. Even Sub-Panel 21, assessing Politics and International Studies, contained only two members out of thirty-one who claimed affiliations outside the UK – though these two possessed joint appointments within domestic institutions. 96 per cent of UK research outputs in REF 2014 were classed as at least ‘internationally recognised’ meaning they covered material and were of a standard that would garner academic interest beyond the UK. Given the insularity of the sub-panels such claims are barely credible. Rather than trusting individual scholars to engage best with their specialised fields in the tradition of free and equal debate, government and universities have constructed a hollow scarecrow via the REF that, in the tradition of empire, defines international standards from the perspective of the metropole.

Criticism of the REF notes its origins in Thatcherism and the application of corporate production quotas to universities as the government has grappled with declining resources and escalating debt. Reflecting on the increased restrictions placed on academic life over the last thirty years, Terry Eagleton, a professor of English at the University of Lancaster, remarks, ‘Instead of government by academics there is rule by hierarchy, a good deal of Byzantine bureaucracy, junior professors who are little but dogsbodies, and vice chancellors who behave as though they are running General Motors’. The global prominence of American universities – especially their ability to draw international students – has stoked the envy of UK administrators struggling with budgetary cuts. UK universities have eagerly recruited foreign students because they pay higher fees. Flooding the market with publications and self-proclaiming those products as globally significant is equivalent to an American-style advertising campaign. Eagleton notes: ‘What has emerged in Britain [. . .] is what one might call Americanization without the affluence’.

American universities have also witnessed a dramatic growth in administrators as these institutions confront intense competition for students. However, academic freedom among American faculty has been insulated from publication quotas because of the tenure system and the dual public-private university system, which facilitates competitive hiring practices. The emphasis on student development has also given faculty greater freedom for intellectual pursuits. Wendy Brown, a political scientist at the University of California-Berkeley, elaborates:

But only in the United States did a post-secondary education contoured toward developing the person and the citizen, not merely the job holder, come ubiquitously to structure higher education curriculums, and only in the United States was such an education on offer to a wide swath of the population from the 1940s forward.

This emphasis on personal character and citizenship encouraged faculty to be both role models for their students as well as productive scholars. Faculty members, especially after the 1960s, were encouraged to be activists for economic equality and social justice. Brown identifies how the REF eviscerates this function of the university:

The move to judge academic endeavor by its uptake in nonacademic venues (commerce, state agencies, NGOS), as the recently implemented British “Research Excellent Framework” does, is equally damaging. Not only does it abjure humanistic inquiry that explores and builds meaning; it cannot capture the value of basic research from which technical applications derive, thus shutting off the spring waters whose exploitation it affirms.

In other words, the organic encouragement of academic research and social engagement has been ignored within the UK. The REF, like any form of Taylorism, treats researchers as cogs in a machine of mass production. Scholars’ academic outputs are treated as throwaway goods with little consideration beyond the next REF.

On this point, one is reminded of David Lodge’s satirical novel Changing Places from 1975. A well-travelled professor of English at the University of Birmingham, Lodge uses a faculty exchange programme between a US and UK university – ‘Euphoric State’ and ‘Rummidge’, respectively – to highlight the striking differences between the higher education systems and their approach to life as well as research. At Euphoric State, faculty members enjoy not only the intellectual freedom safeguarded by tenure but also a range of funding opportunities from government and private organisations. They ‘picked up grants and fellowships as other men pick up hats’. Those Americans who came to Rummidge eschewed the more rigid administrative demands, convoluted marking procedures, and overall bureaucracy of the UK university system. ‘Hence the American visitors to Rummidge tended to be young and/or undistinguished, determined Anglophiles who could find no other way of getting to England or, very rarely, specialists in one of the esoteric disciplines in which Rummidge, through the support of local industry, had established an unchallenged supremacy: domestic appliance technology, tyre sciences and the biochemistry of the cocoa bean’. The appraisal procedures implemented since the 1980s were meant to galvanise the research agenda at universities like the fictional Rummidge while also making UK researchers aware of global needs. Ironically, appraisal systems such as the REF have instead exacerbated the flaws within the UK’s Rummidges by increasing bureaucratic structures, fixating on ill-conceived metrics, and eroding the freedom of academic staff to undertake research as each sees fit.

The REF is grounded on a good intention. Faculty should be engaged with the intellectual life of their respective fields rather than resting on their laurels. The REF requires a faculty member to submit four research outputs – books, articles, patents, conference proceedings, performances, exhibitions, or other work – in order to be eligible for evaluation. Fewer submissions are required of faculty who undertook maternity leave or a career break during the evaluated period. Some items can be double-weighted ‘if an output is of sufficient scale and scope to merit double-weighting in the assessment’.

However, the REF’s emphasis on outputs and speed encourages mediocrity rather than excellence. Even if a book achieved significance in scale and scope, a tome – at best – it gets double-weighted. Academics in the humanities are thus punished for writing books rather than articles. Their freedom to develop arguments is truncated, even though major international and national associations, especially within the humanities, privilege such work through prominent book awards. Worse, the REF has contributed to the publication of slipshod edited collections and new journals, mainly online, with no track record and little readership. An American reviewer of The Oxford Handbook of Tudor Drama, for instance, noted the curious lack of contributors from US and Canadian institutions and the odd inclusion of self-citations in lieu of more prominent scholarship. The reviewer blamed ‘England’s unforgiving Research Excellence Framework’. He wrote, ‘As ever more rigid assessment regimes worldwide exert greater pressures to publish, to be cited, and to cite colleagues (but not to include or cite others), this collection points to the need for resisting such demands’ . In other words, rapid production and dissemination of scholarship under the REF undermines rather than bolsters the international reputation of UK universities.

An anthropological analysis by Cris Shore and Susan Wright has exposed how scholars and university administrators purposely undermined or manipulated their work in order to satisfy the REF requirements. Efforts to scheme the REF highlight ways academic freedom and integrity have been sacrificed. They argue:

The skewing effects of systems of measuring and grading universities’ research output are now so familiar that they have acquired their own terminology, such as ‘salami slicing’ (cutting research results into small chunks, each published as a separate journal article), ‘rushing to press’ (publishing partial results as soon as they are available rather than making a mature and considered analysis), and the ‘star player’ syndrome (hiring high-profile researchers just before a research assessment exercise.

The result is a profound ‘deprofessionalisation’ within universities.

The emphasis on quantity undermines quality – the hallmark of academic freedom – in numerous ways. The need to publish and to do so in impressive journals has a counterproductive impact. Michael Bailey argues that targeting ‘“top” journals forces academics to fashion their research around what those journals want, which can result in an unwillingness to push beyond the narrow confines of specialist fields of study and, ultimately, intellectual inertia’ (Bailey, 2011, p. 96). He rightly highlights the corrosive effect of the REF on university life. Bailey writes that:

With sails trimmed tight, increasingly academics are forced to cut corners if they are to meet the next publishing deadline, particularly newly qualified academics who are expected to combine research with heavy teaching loads and endless duties (a problem whose sheer scale and mind-numbingly tedious and pointless nature appears to be exclusively British).

The requirement that each faculty member submit four publications further undermines intellectual activity in numerous ways. Within the humanities, for example, this one-size-fits-all approach to assessing academic production ignores disciplinary, geographical, and temporal specialisations. A historian working on Africa or the United States must submit the same number of publications as a historian of the United Kingdom. No consideration is given for the distance required to reach and undertake a prolonged stay at the archival holdings. A literary scholar analysing easily accessible novels submits the same number of publications as an anthropologist, despite the potentially time-consuming fieldwork required for the latter. The REF thereby handicaps much of the internationally important research it seeks to inspire by forcing conformity upon academics.

Individual publication targets are detrimental to creativity and fail to engage with the unique trajectory of each academic project, in which time is the most precious commodity. A perusal of the acknowledgements within academic books reveals the extraordinary number of years invested in these volumes. As already mentioned, the REF discourages such sophisticated scholarly analysis by fixating upon the number of publications. Instead of individual targets, reasonable departmental targets might work better by providing a stronger sense of community within a discipline-specific school. But even departmental targets risk starting a school down a steep slippery slope. A school, for instance, that reaches a 25 per cent target for REF-returnable faculty will soon face pressure to meet a more ambitious target, especially as administrators play musical chairs across the university system and therefore seek a résumé that better assures future employment.

A significant, though often overlooked, problem with the REF rests in the disproportionate representation given to higher-ranking faculty members. This imbalance particularly threatens academic freedom by reinforcing disciplinary boundaries and entrenching lines of thought. Professors dominated the review panels on the 2014 REF. Professors formed 65 per cent of Sub-Panel 21 on Politics and International Relations, 82 per cent of Sub-Panel 28 on Modern Languages and Linguistics, and 70 per cent of Sub-Panel 30 on History. Junior faculty amounted to 10 per cent of Sub-Panel 21, 0 per cent on Sub-Panel 28, and 20 per cent of Sub-Panel 30. The other members on those panels – 25 per cent, 18 per cent, and 10 per cent, respectively – were employees of the British Library, Oxford University Press, BBC, or other agencies that partner with UK universities. This latter group predominantly lacked doctorates. The use of professors resides in the assumption that they are better suited to assess the quality of work because of their greater expertise – an expertise cultivated through years within the profession.

Such reliance on senior members of academia carries a profound cost. Older may mean wiser but it does not generally mean greater openness to new ideas. As Thomas Kuhn has argued in his classic The Structure of Scientific Revolutions, academics forge their careers by promoting their research and defending their claims from challengers. Though focused on the sciences, Kuhn’s framework outlining how new knowledge gains credibility carries weight across the disciplines. A discipline works on particular problems, yet some of these problems may prove unsolvable without a completely new worldview. Einstein’s theory of relativity revolutionised physics, a process unpacked by Kuhn. Yet a similar shift occurred within the field of history during the 1960s. Historians in the United States, for example, jettisoned white supremacist paradigms, such as that of the so-called Dunning School, for one inclusive of African American and minority perspectives. As a result, once marginalised scholars, such as W. E. B. DuBois, gained widespread recognition for being ahead of their time. Since the 1990s, younger scholars have pioneered the now-established fields of tourism studies, queer studies, and food studies. Kuhn explains that innovators tend to be ‘so young or so new to the crisis-ridden field that practice has committed them less deeply than most of their contemporaries to the world view and rules determined by the old paradigm’. Innovation typically occurs despite rather than because of older intellectuals. Kuhn concedes that ‘most of them can be reached in one way or another’. However, some intellectuals, ‘particularly the older and more experienced ones, may resist indefinitely’ the work of younger scholars.

The depth of modern knowledge and the development of the university system have increased the age of innovators during the twentieth century. Whereas scholars typically became research active by 23 in 1900, they did not become research active until 31 in 2000 (Jones, 2010, pp. 1-2). Drawing from the fields of Chemistry, Medicine, Physics, and Economics, Benjamin Jones, a professor at the Kellogg School of Management at Northwestern University, has shown that the ‘mean age at great achievement for both Nobel Prize winners and great technological inventors rose by about six years over the course of the twentieth century’. Yet he found ‘no compensating shift in the productivity of innovators beyond middle age’. In other words, notable innovations are most likely to occur between the ages of 30 and 40, peaking around 38 then decreasing in likelihood later in life. I would contend that a similar curve occurs within the humanities.

The wisdom and experience of professors should not be discarded but, rather, there needs to be balance to ensure more accurate ranking and greater openness to cutting-edge scholarship. Privileging professors over junior faculty dulls the intellectual environment by causing a chilling effect on academic freedom, especially when livelihoods are at risk due to the REF. Unpopular topics or adventurous projects are implicitly discouraged. This is precisely the situation condemned by Jürgen Habermas when he called for the ‘democratisation of the university’ during the late 1960s. Habermas proposed, ‘The university run by professors, which simulates a community of teachers and students, would be replaced by a corporation in whose administration all three parties would take part with the opportunity of asserting their own interests: students, junior faculty, and professors’. The REF falls far short of this ideal and the safeguards this ideal provides for academic freedom, intellectual vibrancy, and open community across the generations. Writing for the Guardian in 2014, Derek Sayer, professor of history at Lancaster University, warned:

The most innovative work – the research that breaks molds, shifts paradigms and redefines fields – may not even make it into the REF at all because universities tailor their submission to what they think REF panels want, and REF panels reflect disciplinary hierarchies.

Echoing Habermas a half century ago, he further stressed that the ‘REF panels give extraordinary gatekeeping power to a disproportionately older, male, white – and overwhelmingly Russell Group and form 1994 Group – academic elite’.

Rather than containing the damage caused by the REF, the bureaucracies of universities and of government increasingly reshape higher education policies to bolster the REF rankings. Growing pressure to attain grants, which the UK universities siphon, is further curtailing academic freedom in the name of fostering outputs. This curtailment is also reflected on the recently introduced requirement to publish in open-access journals, as demanded for grants funded by the UK government and for the inclusion of articles in the next REF. The fixation with the REF reaches beyond faculty to postgraduates. Students studying for their doctorates are increasingly under pressure to work along models forced upon faculty. This undermines their standing on the global job market and constrains their intellectual ingenuity. Like a virus, the REF has penetrated deep into the UK university system, consuming the very host that gives it life.

Studies have shown that innovation in academia requires an intellectual culture that provides faculty ample time to pursue their ideas. Innovation also demands an acceptance that some adventurous projects fail. Both aspects require trust in the wisdom and devotion of scholars – a trust the REF denies by its very nature. The American universities’ tenure system allows for both time and failure; the REF truncates the time available for research and punishes failure to such a degree that faculty are discouraged from pursuing radical new concepts, experiments, or innovations. The REF is converting universities into research treadmills by pushing faculty to great exertions in terms of outputs but encouraging them to go nowhere in terms of ideas. As Mencken warned: ‘Long before the bureaucracy is satisfied, the man is worn out and in despair’.

This essay is taken from the Civitas book, Why Academic Freedom Matters, a multi-author collection which can be bought or downloaded here.

About the Author

Anthony J. Stanonis is Lecturer in modern US history at the Queen’s University Belfast. He is author of Creating the Big Easy: New Orleans and the emergence of modern tourism, 1918–1945 (2006) and Faith in Bikinis: American Leisure and the Coastal South (2014) and editor of Dixie Emporium: Tourism, Foodways, and Consumer Culture in the American South (2008).