The way the money goes

Plans to axe the research assessment exercise (RAE) after 2008 have prompted a vigorous debate about how to judge the best research - the work that gets the funding. For arts and humanities, the dilemma is more acute than for science and engineering, where there are strong links between the quality of research and the amount of income teams attract in grants and contracts.

Historians or philosophers do not work in teams and attract little if any income, but the quality of their work could be rated by whether academics in their field cite it in their own papers - the so-called citation index. But are all academic journals equal? The European Science Foundation (ESF) has proposed lists of the most prestigious journals as the basis for a citation index to be used when funding research across the EU. That would strongly influence the fate of British academics.

We asked two of them to discuss the issues.

From: Dr B Smith

20/06/06 11.49am

Dear MM,

Were we in a university system with abundant resources, all research would be amply funded: some of it would advance knowledge, some make no contribution whatsoever. But resources are stretched. There are calls across Europe for increased funding in humanities, so it is time to make a case for the quality of research produced.

Some will ask why more money should be given to the humanities when they cannot demonstrate the quality of research they produce, or agree on how it should be evaluated. Since the sciences have to bid for funds by providing measures of the quality of their research, it will be hard for us to win increased funds unless we do likewise. We don't want a citation index. We know things work differently in the humanities. But demonstrating the strength of the European research culture by publishing lists of high-quality research journals, assessed by peer review, is the best way to show that academics across Europe make significant contributions to national and international research. If this demonstrates the strength and vibrancy of European research, we should support it and try to ensure it is done properly.

Very best, Barry

From: MM McCabe

20/06/06 3.43 pm

Dear Barry,

Wouldn't it be wonderful to be in a wealthy university system? We live instead with, and fight bitter battles for, diminishing resources. Our paymasters, in the UK and in Europe, press us to make inexpensive the procedures for deciding the allocation of funds; thence the demand to move from the RAE to something cheap and speedy: metrics. But the case has not been made, I believe, to measure humanities research, even in part, by means of a graded list of journals.

Consider three objections: that metrics just cannot evaluate humanities research; that their institutionalisation will damage the journal culture we enjoy; and that to compile such lists properly is impracticable.

Starting with the first: how should we assess the value of work in the humanities? We may have been inexcusably coy about this. As a consequence, they risk caricature as cultural luxuries, expendable, of no account compared with the task of the sciences to save lives and make us comfortable. This caricature - which undervalues both the sciences and the humanities - makes us defensive, vulnerable to threats of citation indices (which promote the self-serving, while avoiding intellectual risk).

But we know what we do, and why it matters: we seek to understand ourselves, in all our complexity; and, in this, we join forces with the scientists on equal footing. But this search to understand ourselves is often indecisive, inconclusive, risky (though not subjective). Its success is not judged by outcomes, but by the quality of the work itself. To find that out, the work must be read; not its citations counted nor its place of publication ranked.

Such a reading is undertaken, however imperfectly, by the RAE. The RAE is expensive, and so under threat. But we should not be bullied by that threat into thinking that metrics are, even if imperfect too, able to do the same job: they just aren't. A journal list does not "demonstrate quality", nor does publishing in a gold-star journal: what demonstrates the quality of our research is the research itself - it must be read.

Very best, MM

From: Dr B Smith

21/06/06 4.15pm

Dear MM,

We agree that a crude metric of assessment won't do. But what are you proposing? There are two audiences here: the scholars, who benefit from high-quality research, and our external paymasters, who want some assurance the research is of high quality. Are you seriously suggesting the latter read the journals and form a view? You make common cause with the scientists, but they have to evaluate their research for their paymasters. Why, they will ask, should we be exempt?

Surely, it is better for academics to evaluate research in the humanities than have bureaucrats force quick and cheap evaluation systems on us. (Though a US-leaning commercial service like Thomson International is not cheap.)

I take it you are not suggesting there are values that cannot be assessed. We all carry around in our heads an informal ranking of journals. Isn't it better to disclose it and see if there is a consensus in the discipline that carries conviction?

The lists the European Science Foundation (ESF) will produce were arrived at in this way. They had input from member organisations, were worked on by a European panel of experts, who then consulted subject associations. The resulting lists will need to be maintained and revised to reflect a dynamic research culture. But all the journals listed contain high-quality research. As for the categories of A, B and C, this is not a simple hierarchy and I would hope people in the humanities could appreciate this point.

Journals of different kinds serve different academic purposes. A journal's ranking implies no value judgment about any individual piece of research. The idea that scholars wouldn't read an article on their topic because it didn't appear in an A-rated journal suggests a low opinion of humanities researchers. Let's agree to tell our paymasters what we do well. Our rich journal culture will survive.

Very best, Barry

From: MM McCabe

21/06/06 7.40pm

Dear Barry,

Up to a point, we agree: no cheap and nasty citation indices; and if our work is to be evaluated, it must be done by the academics themselves. But it is the work that should be evaluated, not the journals in which it appears - and this is just what the present RAE sets out to do. It may be, for many reasons, questionable; but it points in the right direction.

Journal lists, in my view, point in the wrong one, and are dangerous, to boot. Much of the time, the humanities disciplines proceed - in common with all speculative thinking - by open discussion, by exposure to the ideas and criticism of others. Sometimes we can have those discussions face to face; but sometimes they are slower, reflective, and at a distance - and then we are immeasurably indebted to the journal culture we enjoy.

This culture will be doubly damaged by a journals list, graded as you endorse. On the one hand, careful and even-handed discussion will be irrevocably slanted by contributions carrying the grade of the journal in which they appear (is it not naive to suppose otherwise?). On the other, the journals themselves will suffer. Imagine some A journal, central to some discipline in which everyone will want to publish: it will be overwhelmed by submissions, and ossify. Or consider a C journal, small and marginal, perhaps, or new: it will be forced to publish a dull round of poor stuff, or to go under altogether - the lists cannot adapt fast enough for its flourishing. No matter how carefully the lists may be compiled, they will do active harm. And unless the list-makers read everything the journals contain - and reread, regularly - will the lists be carefully compiled?

Very best, MM

From: Dr B Smith

22/06/06 2.01pm

Dear MM,

I understand the worry. Journals provide an important place for the development of ideas, and we must keep this alive. The point of doing research is to communicate ideas and to change minds. But for this to happen, journal articles have to be read, and be worth reading. To ensure this we need firm standards of scholarship and good criteria for selection.

The system I described provides a list, in different categories, of high-quality journals. The reasonable expectation is that a European research index will encourage better practices. Standards of publication should be kept high and policies of accepting or rejecting submissions made clear. Do journals have blind refereeing? Are they truly international? Surely, we don't want journals reflecting subjective preferences of editors. Unfortunately, this still happens in our European journal culture. We can do better.

Even now, we all have judgments about the best journals, but this doesn't mean papers are only submitted to them. We know there are specialist journals, and others with a particular focus, more suited to what we are working on. Likewise, the ESF index is not a simple hierarchy of better and worse: its categories respect the different functions journals fulfil. We both know there is talk of who has published in good journals. But what counts as a good journal is based on one's subjective impressions. How accurate are these? How up-to-date? Of the 600 journals considered by the philosophy panel, would you know their worth? Isn't it better to discuss widely and arrive at collective decisions about the value of our research journals? Democracy is needed here.

Let's separate the issues. The research index ranks journals, not academics, unlike the RAE, which ranks individuals and departments as international, national, etc. And since only 37% of submissions for the last RAE were in journals, the ESF index won't be the ultimate evaluation of output. What it will do is enable the Arts and Humanities Research Council to tell the government that a good deal of research in the humanities is taking place in good quality journals.

Very best, Barry

From: MM McCabe

22/06/06 3.44pm

Dear Barry,

You want an account of a "good quality journal", in order to demonstrate the quality of our research. And you suggest two distinct criteria for this account: standards and transparency. The latter reveals a journal's editorial policy (how refereeing is done, how acceptance rates are measured); the former considers the quality of the work it decides to publish. It is a mistake to conflate them: often good academic judgment does not go hand-in-hand with the procedural propriety that is so much in demand these days. For sure, we can require of our journals that they become more transparent; but transparency without good judgment will not rescue a lousy journal. Good judgment, in turn, cannot be arrived at without - to reiterate the point - reading the work; and it cannot be checked without reading the work. If this is to be a list that registers anything about the quality of research, the list-makers will have to read the work: all of it. Democratised opinion will not do.

Now suppose the list is made, despite the powerful protests of some subject associations. It may be Frankenstein's monster: who will control the use to which it is put?

But I am afraid I don't see either of us shifting our positions, do you, Barry?

Very best, MM

· Mary Margaret McCabe is professor of ancient philosophy at King's College London. Dr Barry C Smith is in the school of philosophy at Birkbeck College and is deputy director of the Institute of philosophy at the University of London. He is a member of the European Science Foundation's expert panel in philosophy