Citation indices do not avoid subjectivity

Peter Higgs used his recent celebrity to criticize the current academic job system: “Today I wouldn’t get an academic job. It’s as simple as that. I don’t think I would be regarded as productive enough.” In this context, it was argued to me that using citation count, publication count, or some other related index during the hiring process for academics is a necessary evil. In particular, single academic job openings are often deluded with dozens or hundreds of applications, and there needs to be some method of narrowing down the search to a manageable number of applicants. Furthermore, it has been said, it’s important that this method is more objective rather than subjective.

I don’t think it makes sense at all to describe citation indices as less subjective measures than individual judgement calls. They just push the subjectivity from a small group (the hiring committee) to a larger group (the physics community); the decision to publish and cite is always held by human beings. Contrast this to an objective measure of how fast someone is: their 100m dash time. The subjectivity of asking a judge to guess how fast a runner appears to be going as he runs by, and the possible sources of error due to varying height or gait, are not much fixed by asking many judges and taking an “objective” vote tally.

Of course, if the hiring committee doesn’t have the time or expertise to evaluate the work done by a job applicant, then what a citation index does effectively do is farm out that evaluative work to the greater physics community. And that can be OK if you are clear that that’s what you’re doing. But this doesn’t make the evaluation any less subjective. Likewise, if you’re trying to determine which person in your high school is going to make the largest contribution to the world, it might be useful to know who’s voted “most likely to succeed”. But you should be clear that (a) this isn’t any less subjective and (b) this popularity contest selects against quiet nerdy types like Bill Gates.

Now, there is an alternative notion of “objectivity” meaning “neutrality”, whose antonym is “partiality” rather than “subjectivity”. For instance, if two people have a legal dispute, they are both likely to agree on a neutral third party (a judge) to help resolve their dispute rather than asking one of their relatives. This doesn’t make the judge’s decision non-subjective at all, just impartial (hopefully). Is this notion of objectivity, i.e. neutrality, the real issue in academic job searches? Maybe.

One type of bias might be between two members of the hiring committee, Alice and Bob. If Alice wants to hire her friend Jill but Bob wants to hire his friend Henry, both may accuse the other of bias. In the legal system this can be partially fixed by requiring judges to recuse themselves on the rare occasion that they know a litigant. But this might not be possible in small academic circles where the tiny number of competent experts in any given topic means that members of the hiring committee often have close pre-existing relationships with applicants. So Alice and Bob may then turn to a crude metric like a citation index. Again: this is impartial but not any less subjective.

Of course, my opinion is potentially colored. I’m currently applying for jobs, I’ve never been on a hiring committee, and I like to think the quality of my ideas outweigh my small number of publications…

[This post was prompted by discussion with Adrian Kent and Jason Ralph. Edited 2013-12-8 to make the foil argument more modest.]

One Comment

One argument that can be made for citation-based indicies is that the speed at which people do experiments and publish papers matters, because people will always have many more ideas than they can share with the academic community and so indicies based on the total number of citations measure not just how good a persons ideas are, but also how well they will share them with the community.