February 26, 2008

On Blind Reviewing (2)

In the first part, I presented a case in favour (favor) of blind reviewing as a bulleted list of ten reasons to have blind reviewing as the most preferable implementation of the concept of peer reviewing.

Excuse again the legal sounding language.

I also indicated that I am not sure that this is indeed correct, i.e., blind reviewing is the best that we can do to ensure quality research. In this post I will try to present a case against blind reviewing, or at least a case against the idea that blind reviewing is only or the best way to go about the business of research.

First, let’s take each of the ten points I had listed:

Human beings can be biased. So, if a reviewer knows that a research paper is written by a person she doesn’t like or has strong disagreement with, she can get biased against the paper and will not be able to review the paper fairly.

Human beings can be biased and their biases can really come to the fore if they know that their identities won’t be revealed. There may be enough in the paper to trigger their biases.

Apart from the above kind of biases, there can be the bias in terms of the weights associated with the names of the authors, their institutions, their countries, their group, even their academic background. Most of the people who have been working in NLP/CL[1] for some time know about the linguistics vs. statistics or machine learning bias. This kind of bias increases the chance of your paper being rejected or accepted depending on whether you seem to be in favour (or favor) of a linguistics heavy approach to NLP/CL or of a statistics (or machine learning) heavy approach. There are variants of this bias in other fields too. For the closest example, we can consider Linguistics. Where your paper is perceived to be situated along the Chomskyan or Empiricist or Cognitive or Computational axes with respect to the chosen position of the reviewer can have a large impact on the decision about your paper, irrespective of what else your paper says. And the chances of such a perception can be increased if the identities are known.

Such biases are especially the ones that can be triggered mostly by the content of the paper itself, because enough is revealed about the authors by just the content of the paper. More on this later.

Human beings can be unduly confrontational and they can also be unduly wary of confrontation. So, if the identity of the reviewer is not withheld, the author(s) may be offended by the reviewer and they may also become confrontational and carry on this confrontation with the reviewer, thus making the process of reviewing difficult and something which a lot of people would like to run away from. Also, the reviewer may avoid making adverse comments, especially if the reviewer doesn’t want to offend the author(s).

This is perhaps the strongest point in favor of blind reviewing. However, the problem is that, like all the other points, it assumes that identities are not revealed at all in the blind reviewing process. The truth is that the content of the paper often gives enough information about the possible identities of the author(s). And this happens more in the cases where there are much higher chances of biases and where a proactively fair reviewing process is needed the most. For example, if someone is working on languages of the Third World, those languages will be mentioned in the paper. From this, and from the language and presentation of the paper, it will be easy to guess that the paper is from some Third World researcher. In many cases, the reviewer might even be able to guess the author(s), or at least the groups of which the author(s) are part. It is these cases where the need for fairness is the highest because it is so easy for a reviewer to become biased against the paper. So much so that she doesn’t even care to read the paper carefully. If you get two papers to review and one of them seems to be from a major research group from a major university from a First World country, while the other is possibly from a graduate student from a second class university in a Third World country, would there be any difference in the way you review the papers? Would there be any prejudgment? Won’t you be more careful in reviewing the second paper if you knew your identity will be revealed?

If the author(s) don’t know who the reviewer is and vice versa, the whole reviewing process may be more fair for the above specified reasons and because of the general association between anonymity and fairness. If you don’t know who is criticizing and the person criticizing also doesn’t know who is being criticized, then you can expect more fairness.

Seems to be a valid point on the surface. And it is: To a certain extent. But there is an even more valid counter-point. If the reviewer knows her identity is not going to be revealed, she can be as biased as wants. Even as biased as she doesn’t want. There is not much ‘incentive’ to read the papers carefully. You can get away with anything, especially if the authors are not ‘prominent’ ones so that even the chair(s) won’t probably notice. Unknown authors, paper rejected, extremely negative comments and ridiculously low scores. So what? Common occurrence. No need to take note.

If the Program Committee (PC) chair(s) also don’t know who the authors are and who the reviewers are, then they can assign equal weight to all the reviews for making the final decision about a paper.

This is actually a hypothetical point. It doesn’t apply even to double blind reviewing. The chair(s) always know who the authors are. The reviewing process is not actually completely blind. In almost all cases, the chair(s) assign the papers to the reviewers. Now, the reality is that the fate of a paper depends on which reviewers are assigned to review it. Moreover, in many cases, the chair(s) or the area chair(s) actually overrule the assessment of the reviewers. They are supposed to do this to ensure fairness, but why should the chair(s) be assumed to be free from bias? They are as much human as other reviewers. Reviewers in one conference or journal are often the chair(s) or the editor(s) in some other conference or journal.

If the author(s) don’t know who the reviewer is, then they won’t have any reason to attribute bias or prejudice to the comments made or ratings given by the reviewer.

Only if they assume that the reviewer didn’t guess anything about the possible identities of the authors. That might happen with their first few papers, but later on they might catch on to the fact that this is not really so. In fact, just as the reviewers can guess the possible identities of the author(s), the vice versa can also happen. One small remark might give away the identity of the reviewer. And it often does.

Peer reviewing of research papers, like the administration of justice, should not just be fair, but seen to be fair. And this can only happen with blind reviewing.

Again a superficially valid point, but there is a more valid counter-point. If everything is ‘blind’ but not really blind, how can the process be seen to be fair? Perhaps there can be some other ‘non-blind’ way which actually seems more fair.

Blind reviewing, through the use of the device of anonymity, gives a true meaning to the idea of ‘peer reviewing’, because if the identities are not known, all the people involved can be treated as peers, even if some of them are senior most pioneering researchers or Directors of first class institutions in first world countries, while some others are graduate students in second class institutions in third world countries.

As discussed earlier, anonymity doesn’t really happen in the cases where it is supposed to matter the most, i.e., in cases where there can be biases. Could it be that ‘blind reviewing’ actually gives an impression of fairness where it doesn’t really exist? This could be a very undesirable situation if we are really sincere about ensuring fairness.

If the identities are not known, both the reviewer and the author can focus on the content of the paper and the review, respectively.

Another strong point in favor of blind reviewing, but again it depends on the assumption of anonymity. Also, could it be that (consciously or unconsciously) the reviewer starts guessing who the paper is possibly from so that she can review accordingly, without spending too much time in understanding the paper? And when the author(s) get the reviews, could they be guessing who the reviewers are (from the remarks they have made). Both of them might reach right or wrong conclusions, and whichever they reach, such a situation will not be an ideal one for the purpose of fair peer reviewing. This is, of course, more likely to happen with papers which get rejected, fairly or unfairly.

Finally, the very practical reason that blind reviewing provides a reasonably fair mechanism to ensure the selection of the best research papers such that everyone can be more or less satisfied with the outcome and no one will have valid reasons to complain.

Not all conferences or journals have blind reviewing. Yes, some studies have been done which show that blind reviewing reduces biases, but these studies have not considered the cases where the biases are most likely (as mentioned earlier). What these studies show that biases are somewhat less in cases where biases are likely to be less anyway.

So the case it not as clear as it might seem at first sight.

Now I will mention the really bleak reality of blind reviewing which made me think about the reviewing process and, over the years, has provoked me enough to write this.

Let me emphatically state first that when I started research and was thinking of publishing my first paper, I was really happy to know that most NLP/CL conferences and journals use blind reviewing process to select papers. My reasoning was exactly as I have listed above as points in favor of blind reviewing. So this post is not being written just out of a whim.

What I have found is that blind reviewing, though it does work to some extent, actually becomes a cover for reviewers to be as irresponsible as they want because they are anonymous. As an analogy, there are some good anonymous commentators on blogs, but many use anonymity as cover for their mean and nasty (completely unjustified) comments with perfect unaccountability. Something similar happens with the blind reviewing process. Many reviewers, including those who are most probably very senior researchers, use the cover of anonymity to let all their biases flow freely into their reviews because they know they are safe. And, unlike on the blogs, the author(s) can’t even reply because most conferences don’t have a rebuttal phase. Even if there is one, the reviewers simply don’t care. They have reviewed and their comments are final and unchangeable, whatever the author may have to say. They don’t change their reviews in response to author(s)’ rebuttals or clarifications. They don’t because they are safe in anonymity.

What makes this kind of situation even worse is the fact that a lot of reviewers review for many conferences and journals and, therefore, your paper if unfairly rejected from one place, is quite likely to run into the same set of reviewers at another place.

It can be quite depressing. I am sure many researchers have thought of or have actually quit from the research arena because of what I have described above.

I think it’s time we gave another look to the reviewing process. I don’t have a solution, but I will try to make some suggestions later. Perhaps more experienced researchers and organizers can say something better.

I know there are many great people out there who put their best in writing a review and actually go beyond what’s expected of them, but I think such people would do the same even if their identities were revealed because they know they are doing what’s right to the best of their abilities.