How Much NSF Grant Money Can One Bad Apple Control?

Second: a friend of mine was rejected for an NSF training fellowship. Two of the three reviewers liked her proposal, but the third review was very negative, which, well… sometimes it happens. But in the section evaluating the “broader impacts”, instead of addressing what she actually wrote, he (reviewers are anonymous, I use the male pronoun here for convenience) noted that she was “truly intelligent” and also a woman, and opined that this alone should fulfill the requirement of broad participation. He hoped that she would resubmit a proposal that was not complete garbage, because she works in an important subfield and has potential. (In case you are curious, the full question to which the reviewer was responding was What are the broader impacts of the proposed activity? In addition, please consider the prospective benefits to the applicant, the scientific discipline and the United States.)

Now, this review made us both want to puke on the reviewer’s shoes, but we found it difficult to articulate why. There are any number of horrible implications one can read into that brief paragraph: that women’s proposals should be judged by lower standards, that truly intelligent women are very rare, that the original broader impacts statement did not contain anything interesting to talk about, that the reviewer was thinking about her gender the entire time he was writing the rest of the review and was too distracted by wondering about boobiez to pay attention to the science, etc. Presumably the reviewer would vehemently deny all of these things, but they aren’t all things that operate at a conscious level.

Personally, what makes me roll my eyes the most is the thought that after he had finished trashing her application, this reviewer gave himself a pat on the back for saying something nice and supporting the cause of women in science. Which is not to imply that he should have held back, if he really thought her proposal was garbage, but he should’ve realized that “She’s smart, and a woman!” tacked on to the end of an extremely negative review would smell like a road that’s been traveled by a caravan of sanctimonious donkeys.

One of the benefits of a multi-person review panel is that if you happen to draw a bad card, someone who’s not going to give your proposal a fair shake because of your ethnicity or gender or institutional affiliation or you’re not using the lab technique he invented or whatever, the other two reviewers can balance that out. However, in a highly competitive grant program like the NSF’s, you will eventually find yourself competing against proposals that didn’t have a weirdo on the panel, or didn’t trigger the weirdo’s biases, and got three good reviews as a result. Which means that adding more reviewers is not a good defense against the political nature of the process.

This particular fellowship was not awarded through a blind review process, wherein the scientific merits of a project are evaluated before the panel knows who submitted it (another criterion for awarding NSF fellowships, the competence/promise of the scientist, obviously can’t be evaluated blindly). I really think it ought to be. I’m not sure whether or not blind review produces significant changes in the racial or gender distribution of fellowship recipients (and am too lazy to research the matter right now, but perhaps the lazyweb can do it for me?), but it’d be really nice if we could get negative feedback on the scientific merits of our proposals without wondering if racism or sexism played a role.

Comments

Alternatively, if erviews had to be signed, then people couldn’t use anonymity as a shield for unscientific grant assassinations.

In any case, the NSF really needs an outlier rejection algorythm, where any review more than a few sigma from the other reviews simply gets tossed out as “electro-sociological noise”.

Then entire topic probably needs a post of its own, though.

As for the sexual comments, damning with faint praise is common methodology of the corrupt gatekeeper class. What they choose to be “complimentary” about is probably trivial. Worrying about whether it’s a woman thing is like wondering whether the bunny and the hat need to be different colors; the more you think about it, the more it distracts from the real issue.

I think the underlying worry is more, “is this negative evaluation the result of something I can control, or not?” – perhaps not the most productive response to reviewers’ caprice, but a very human one.

The congnitive dissonance required to completely trash someone’s research proposals and then call them ‘truly intelligent’ should be sufficient to disqualify this reviewer’s comments from consideration – they might as well write “NOT EVEN REMOTELY IMPARTIAL” in big red pen at the top of their review.