Archive

Guide to reading reviews

For the upcoming rebuttals of CHI, it might be useful to understand what the reviewers really mean when writing their reviews. This year as I read with interest the reviews of my fellow reviewers, maybe due to my growing experience, or maybe because of the late hour reviewing, I started to see something new in the reviews: the hidden messages. Below is a collection of this years’ CHI, CSCW and past years’ CHI review’s opening remarks with possible interpretations.

What they said

What they meant

What a great paper! It was so easy to read.

It was quick to read and I didn’t pay too much attention to the details that could have confused me.

This paper is needed in HCI. Too many people are not aware of this.

I wish I had known about this before I made my study.

This paper describes a study about…

I’m blank about what to say, so I’m going to state the obvious facts and add an opinion at the end

According to Galileo, Smith and Edison (2061) this line of research is misguided. I agree, which is why I’m giving it a 2.

I’m a jerk with superficial ideas who will not take time to review papers.

This paper attempts to make a contribution…

Oh God, why did I accept this totally meaningless paper?

The paper provides a clear structure and has a coherent presentation.

Structure: check, Presentation: check.

This kind of research has been done within the HCI field since the 1980’s, as the authors rightly point out

I’m not going to bother too much, there is no way there is anything new here.

My main concern about this paper is the practicality of the idea

I don’t like the work and I have to come up with something to justify my opinion.

On a serious note, I believe reviewing is a truly important component of research. A good review should focus on the work at hand. What contribution does it make to our field and to scientific knowledge at large? Does it widen or deepen our understanding of the issues at hand? Can the results or part of the results be generalized and have impact on research or practice?Novelty is important for some types of HCI papers, but far from all.

Next, the review should assess the validity of the work. No study is perfect and nitpicking on details of the design of the study or the analysis performed is not judging the validity of the work, unless of course there are major errors. One study is not enough to prove something; we need more than one. Hence, minor mistakes in one study can be corrected by subsequent studies without invalidating the work.

Finally, the reviewer should give advice about how to improve the authors’ work: either as a pointer for future work or to improve the current paper at hand. This is an important point: conducting research and writing papers is a learning process and we all benefit from a mature community doing great research. We all win with a good reviewing process. Good luck with the rebuttals!

Pernilla, I liked:) I especially appreciate your comment on the educational value of reviews. I’ll share a small anecdote.

I was recently enrolled in a course on how to write a good review and I was appalled by the professor’s emphasis on competition. The benefits of writing a good review were reduced to personal as opposed to communal gain (e.g., “writing a good review will improve the AC’s opinion of you”).

Furthermore, and this is a slight pushback on one of your humorous interpretations, I noted to the professor that I often write a small summary of the paper at the head of my reviews. I think this is valuable because it can help the authors and other reviewers assess whether (or how) I understood the paper and its core contributions. Of course, I follow this short summary (2-3 sentences) with a much longer, meatier review. Cutting to the point, my professor’s response was that writing a summary in the review can leave you vulnerable to criticism if you have misinterpreted the paper! I was shocked.

In short, I agree that reviews are firstly about promoting quality work and educating our peers (and ourselves) about how to produce it. This purpose will best be served if we check our egos at the door and think about wider communal impact.

Stacy, 1) interesting that you have a course on reviewing. This seems like a positive thing. 2) Is the course based on the instructor’s opinion, or are there more objective sources of evidence for effective reviewing strategies? If so, what are they? Where does the notion that ACs form lasting impressions of people based on their reviews come from? I bet that prior experience with the reviewer in a research or other role will have a stronger effect on perceived quality than the actual review.

Thanks for sharing your experience and your comments, Stacy. I agree with you that it is a good practice to start out summarizing the paper. It serves to establish a common ground between you as a reviewer, the authors and the meta-reviewer. In my view, if you as a reviewer misunderstand the paper it is likely that other readers will misunderstand it too. Hence, the authors of the paper did not articulate the contribution well enough. A good meta-reviewer should understand this. And from the rest of the review, it is often obvious if the misunderstanding of the paper arises from the reviewer not really reading the paper or unclarity on behalf of the authors. I often start out with a summary, and I included it as one example because I thought it would not be fair only making fun of others ;-).