VISUALIZATION, ETC.

Peer review doesn't work because by then it's too late

(Equal parts reflection, armchair-quarterbacking, and fantasy-writing
about why peer review feels ineffective, on both the giving and
receiving end. More academic navel-gazing, you’ve been warned.)

Maybe too much has been said about why peer review is broken,
but today I want to highlight a point I don’t think I see discussed as
often, which is that by the time a reviewer gets to a paper, it
might be too late.

What I mean by that is that even though most papers have a kernel of a
good idea, at some point before submission the process took a turn for
the worse that makes the submission unsalvageable. What’s a reviewer
to do? I’ve been on both the giving and the receiving end of this.

In the giving end, reading the paper feels like this: “I see.” “Yes,
good point” “Ok, so now you’re going to… what? huh? no. No. What?
wait, don’t bother with this experiment, what no no ughhhh”. After
three separate sections of this, the reviewer tends to write a review
amounting to “the writing is unclear and the motivation is missing. I
don’t understand the experiments and also they didn’t cite random
reference X”. This is obviously a bad review, because it all points to
local fixes that can be done, rather than global fixes that
should be done. In the reviewer’s mind, those are not the reason the
paper is bad, but they’re a reason for the paper not to be accepted,
and it’s easier to write the latter than the former. I
can definitely relate to the feeling of despair of reading a paper and
going “oh dear, if only I had gotten to this six months ago”.

In the receiving end, reading the review, this is how feel: “This review
says nothing substantial about the paper, and the reviewer clearly did
not like it. It seems like even if all of these fixes were done, the
reviewer still would have rejected the paper. What now?”.

Like the ER doctor treating a hyperglicemic crisis wishes they could
fix the problem by controlling the patient’s sugar intake in the last
10 years but is limited to insulin shots in the present, the reviewer
gets a paper that needed 12 months of care, and writes “I guess you
can fix the typo on Theorem 4.2” or “experiments do not compare
against technique Y”. Eventually the patient is discharged from the
ER, but the long-term damage is sort of done.

Preventive health care for papers

This mess is partly the fault of the reviewing systems that ask for
“concrete ways in which the paper can be improved”. But I think the
real problem is deeper. The reviewer can do the most good at the
beginning of the process, and at the end of the process, the best
reviewers can do is pick out the few submissions that happened to
avoid all the pitfalls. But if we know what the pitfalls are, why not
address them at the best time? This is where journals can actually add
a lot of value to a submission.

Authors announce, early in the process, their intention of doing work
on an area, and submit this to a journal. The journal editor then
assigns a reviewer early in the process. The reviewer gets to nudge
the paper’s progress in this way or that, and hopefully the total
gnashing of teeth is significantly reduced. This can even be done in
the open if you take the Github-overlay-as-arxiv-of-the-future
perspective: have reviewers file bug reports early on. Journals now
serve as the clearing house for this process; good journals will earn
a reputation for good fostering; good editors find good matches
between authors and reviewers, and everyone wins. Forget gatekeeping
or filtering, and think of journals as academic public health. We
shouldn’t be trying simply to find the best work: we should be trying
to make as much good work as possible.

Of course this is tricky because early in the process is the best time
to steal an idea; but this kind of stealing already happens anyway
late in the process, so the cat’s kind of out of that bag. And if you
think having a reviewer early on increases the power of the reviewer
too much, that just tells me you haven’t been in either side of a
knock-down drag-out reviewing war (that we’re waging on each other
forever more).

So besides path dependence, why do we not do this? Maybe this larger
degree of involvement is what the Ph.D. process should be about (or
should have been about in the bad cases). But then the fostering
process would provide a secondary opportunity for best practices to
diffuse through a community.