Post navigation

When a series of entirely reasonable decisions leads to biased outcomes: thoughts on the Waterman Award

The National Science Foundation just announced the winner of the 2014 Alan T. Waterman Award, the highest award it gives to a scientist or engineer under the age of 35. The winner is Feng Zhang, a molecular biologist at the Broad Institute and Harvard. In addition to being a huge honor, the award comes with $1 million dollars of research funding. It’s a big deal. And, for that reason, I was concerned to see that, just like the previous 10 winners, this year’s winner was a man.

Now, I want to be clear: Feng Zhang is clearly a very impressive scientist, and is highly deserving of this award. So is each man who won the award in the previous decade. But when male scientists win an award 10 times in a row (in one year, two men won), I would suggest that argues that it’s worth examining the process for unintended biases.

Why this focus on the Waterman Award? Because, in my opinion, this is a good example of a common phenomenon that happens often in academia, including in ecology. It happens with society (another link) and university awards, faculty searches, invitations tospeak atmeetings, and in departmentalseminar series – a committee of well-meaning people who are not trying to be exclusive end up selecting primarily men.* In each case, there’s a pool of people who might be deserving of the award (or position or seminar/talk slot); the concern is when the people chosen from that pool end up being a biased sample.

As I said, even in cases where each individual decision or outcome seems entirely justifiable, if there is a consistent pattern, the process as a whole needs to be examined. Let’s split the process into two halves to consider:

1. The nomination process. It is really, really common for people to initially think of men when asked to nominate people, either for awards or talks. This is part of why things like Anne’s List of women neuroscientists is valuable. People interested in increasing diversity of a seminar series, for example, could skim such a list to look for people to consider inviting. As far as I know, no such list exists for ecology and evolution, though I know that my Michigan colleague Gina Baucom has been doing some behind-the-scenes work to try to get one going.

For something like the Waterman Award, this is more challenging. I would guess that a lack of a diverse pool of nominees is probably a large part of the problem. Given how many of the winners have also won the Presidential Early Career Award for Scientists and Engineers (PECASE), one option would be to write to the departmental chairs of all PECASE winners to suggest they submit a nomination. The program contact for the Waterman Award, Mayra Montrose, already writes to PECASE winners to suggest that they ask to be nominated for the Waterman Award – they are clearly trying to increase the number of nominations (which is good!) I have no idea what percentage of PECASE recipients ask someone else to be nominate them, but I’m guessing that, given things like imposter syndrome, women and underrepresented minorities might be less likely to follow through on that suggestion. (This piece in The Atlantic also suggests this would be the case.) Asking the chairs directly might help overcome this. If you have other ideas for how to increase the diversity of the pool of nominees for this sort of award, please suggest them in the comments!

For the Waterman Award, coming up with a single, specific set of criteria seems like it could be particularly challenging. The award goes to one person, but that person could be in any of a wide range of fields, so the awards committee presumably needs to find a way to compare a sociologist, an astronomer, a cell biologist, and an engineer. Each field will have differences in terms of productivity – I have no idea how to try to compare the publication records of a pure mathematician and a computer scientist, for example.

Another potential challenge with the Waterman Award is that, because it goes to people under the age of 35, it will be evaluating people on their productivity at an age right around where many women have children. (This article talks about the “baby penalty” faced by women in science.) Again, I don’t have good ideas for how deal with this for this sort of award (other than for the selection panel to keep it in mind), but hopefully others will.

Hopefully it is clear that I don’t think there is an easy solution to the male-bias in Waterman Award winners, and I am most definitely not saying that there is any intentional bias going on. But I am saying that having 11 consecutive male winners suggests that there might be biases, and that possible ways to improve the diversity of awardees should be considered. My goal here is to try to start a conversation. Moreover, my hope is that thinking about the biases that might influence this one particular award will hopefully lead us to also examine biases in society awards, hiring, and seminar/talk invitations, which affects most of our readers more directly.

Does the diversity of award winners matter? I think so. In the case of the Waterman Award, there is a lot of prestige (and money!) that come with having won it. For society awards, there usually isn’t much money, but there is prestige. Plus, for younger scientists just starting out, seeing pictures of winners (of the Waterman Award or a society award) and seeing that very few look like them sends the message that maybe they don’t belong. Women and people of color in the sciences often ask “Should I be here?” We want to make sure that our seminar series, our faculty, and our award winners send the message that they do.

If you have thoughts on how to deal with this problem – either for the Waterman Award in particular or for the more general problem – please share them in the comments!

*This pattern tends to be even more extreme for racial and ethnic minorities, of course.

71 thoughts on “When a series of entirely reasonable decisions leads to biased outcomes: thoughts on the Waterman Award”

Thanks for bringing this up — I agree with you that acknowledging and managing our implicit biases is an important topic. I just wanted to add another line to your asterisk’d postscript: those systematic implicit biases can be even worse for other minorities not mentioned here – Disabled, low-vision, Blind, hard of hearing and Deaf professionals. Not only that, they also have to deal with explicit biases, assumptions and societal barriers making access to information and research opportunities that much more difficult. So all the points you bring up here are very important for all of us to consider given that science truly benefits from diversity of thought and approaches.

Yes, I definitely agree and considered adding this to the asterisk. In addition to the groups you mentioned, I was also thinking of LGBT scientists.

My impression is that considering things that would help improve the climate for one underrepresented group tends to also improve the climate for other underrepresented groups, but I imagine that there are additional biases in some cases. For example, as you mention, there can be explicit biases related to the ability of certain groups to do science, perhaps especially empirically oriented work (field work and bench work). In those cases, there’s probably additional measures that would need to be taken to improve the climate, beyond those that would help for women, racial/ethnic minorities, etc.

As a statistician, I have to be pedantic and point out that the odds of having 11 straight men (assuming a fair 50/50 ratio) is 2^-11 or 1/2048 or p<0.0005. Now the 50/50 is pretty close (especially for under 35) in biology but across all STEM it might be off, but even assuming 2/3 men which is too high, this is still significant at p<0.01. No one can pass this off as just random chance.

As someone who worked with Morgan using her methods (your closing link #2) on creating a diverse speaking series (the upcoming <a href=http://www.grc.org/programs.aspx?year=2014&program=unifying)Gordon conference) the methods she described are great. They’re easy, rational and produced a very fair outcome of invitees.

To be even more pedantic, you’re testing a hypothesis that was suggested by inspection of the same data on which you’re conducting the test. Don’t misunderstand, I think it’s absolutely right to notice the streak of male winners and ask whether there might not be subtle biases at work that could be mitigated by changes to the selection process. And I don’t think it’d be correct to just say “meh, the streak could be due to chance, let’s not worry about it.” But it’s tricky to try to do formal null hypothesis testing here. /end hyper-pedantry 🙂

Was Brian doing an hypothesis test here? Couldn’t one argue that this is just probability, that assuming the observations were independent and a 50:50 split what are the chances of seeing eleven 1s? Seeing the data wouldn’t change this probability. (I guess that Brian’s use of the word “significant” means he did venture into hypothesis testing, where his point could equally well have been made from probability density alone, without attaching that particular moniker.)

Also, whilst we are being pedants, it would be entirely correct to state that the streak could have arisen by chance. (There is a finite, albeit small, probability of this happening.) It is just *very* unlikely to have done so. 🙂

Based on the comments below, it seems that an analysis such as this should be split into two stages; they appear to be conflated in the numbers Brian gives above. In the second stage, evaluate the winning streak in light of the sex ratio of the nominees (which Montrose, presumably, knows). In the first stage, the sex ratio of the nominees should be evaluated in light of the pool of potential nominees. This is harder, because defining the pool of potential nominees may not be straightforward. However, splitting the analysis into two stages should indicate where mitigating steps are likely to contribute to the desired effect. Even if the pool of potential nominees remained ill-defined, a divided analysis would indicate whether there was potentially bias in selection of the winners from the nominee pool. If not, it would be reasonable to infer that the bias was occurring at the nomination stage.

Thank you Meg. Great Post. And this is indeed appalling. Have you contacted NSF to ask for a response?

As to Brian’s calculations, it would be interesting to normalize the proportions formally, not just by gender representation but also by # of proposals submitted. Has anyone done that?
How about the gender representation of NSF rotators and POs? Mine are all male (BioOce).

As I just commented on twitter, I do think we’ve got to be careful to avoid cherry picking affirmative examples here. As scientists, it would be more scientific to choose, say 10, lists to quantity gender representation of, then do it, and share all the results. (not just appalling cases). Maybe everything is this biased against woman. I hope not and my guess is that it isn’t.

In the EEOB program in Biology at UNC, we have gender equity in faculty hires (unfortunately not in the MCDB side of the dept). Beyond touting the positive, we got here through a very careful and reflective process, much of which you describe above. Both men and woman are often (unconsciously) biased against woman and other minorities and everyone (not just us white guys) has to be very conscientious and careful and transparent about our decisions and actions in searches, etc.

Hey John, I’ll follow up on your request by adding one more award to the list to start inching towards your N =10.

Every year the Association for Tropical Biology and Conservation names 1-2 Honorary Fellows – the highest award we give, in recognition of outstanding contributions to the field and the Society. Since 1963 we’ve named 83 honorary fellows, of which only 6 were women: Krause, Cheerman, Mathias, Meggers, Emmons, Denslow. http://ow.ly/vRLpT

I agree that all of these issues need to be addressed. But, in the meantime, what about having two awards, one for men and one for women? While it wouldn’t halt the bias that is being observed here and everywhere, it would at least elevate one woman a year to a very prominent status. This might, over time, start to cause change. Thoughts?

I suppose the obvious counterargument would be, it’s all science, why do we need separate awards for men’s science and women’s science? And over time, wouldn’t having separate awards risk enshrining a gender gap rather than breaking it down? I dunno, I guess I could see arguing both ways, but personally I think Meg’s suggestions for addressing the issue are the way to go. Especially when I think about the issue in other contexts, like faculty hiring.

I can see both sides of the issue (and the new comment from Happy below agrees with the suggestion that it can lead to people devaluing the award if they perceive it as having a quota for women). But I think that, on the whole, with something as prestigious as the Waterman (especially when you add in the monetary portion of the award), it seems worth a shot.

I agree, I think this is a dangerous idea—too reminiscent of separate but equal, and thus only reinforces the notion that standards are different for men and women (because if there truly aren’t different standards, why not combine into one applicant pool?). This approach would increase female representation in the short-term, but I think the long-term damage it would do would outweigh the benefits. We need to treat the problem, not the symptoms.

To that end, as others have mentioned, it would be most beneficial to know how unlikely 11 consecutive male winners is given the relative proportion of male applicants/nominations each year—undoubtedly much less unlikely, and perhaps not unlikely at all? Which speaks to Meg’s point that increasing the representation of women in the applicant pool would be an important step forward, and would (I suspect) constitute more permanent progress than simply creating two castes of awardee.

Here’s a link to some data showing the breakdown in awards women have received by ASN and SSE (with one ESA award and one private foundation award included). Young Investigator awards are ~15% female, whereas the student awards are right at ~50%.

Just wanted to add another piece to this–In my experience, early career women are much less likely to put themselves forward. For the Am Nat Student Paper Award (a paper published in Am Nat with a lead student author principally responsible for the content), students have to get their advisor to write a one-sentence email saying they are eligible. In previous years, I’ve chased after likely candidates (male and female) who didn’t take that step and asked them if they are eligible. The majority who turned out to be eligible as students and didn’t take the step to be considered were women. There is nothing negative that can happen about simply putting themselves into the pool and yet they don’t. The Young Investigator awards are also done via self-nomination with the same problem.

Great post. I’ve seen other work on this issue, mostly reinforcing what’s been said here. Check out Lincoln et al 2011 Nature 469: 472. And I am always frustrated by the same response Meg notes of “oh, but the winners are all so wonderful!” Yes, but not the point.

Out of curiosity, I had a look at the 2013 ESA Fellows and ESA Early Career Fellows. 2013 Early Career Fellows: 2/6 women (including Meg–congratulations!) 2013 Fellows: 8/17 women. So I don’t really see any reason for concern with that one.

Thank you, Meg, for the excellent blog. As manager of this award for a few years, it frustrates me that I am still not seeing more nominations for women. The number of nominations has increased significantly since I became manager, but the proportion of women nominated has remained steady. On a great note, I see excellent women nominated in all 6 “discipline categories”. Thank you for pointing out the imposter syndrome. I will keep it in mind when soliciting nominations. I think the biggest issue, in the end, is the committee voting for only one award from 6 very different categories. I like the recommendation that we award two prizes, and will forward it to the NSF Director. I also like the recommendation to ask department chairs to nominate individuals. I welcome concrete recommendations from the community. Please send them to my email: mmontros@nsf.gov

I’m glad you brought up the age issue, Meg. I’ve been thinking about writing about it (and maybe I will)! I think awards for scientists based on age are biased against women in general. For one thing, as you point out (sort of), women have children at a younger age than men, and are more directly impacted by things like pregnancy. While decreased rates of research progress in reproductive years may not matter much over the whole of a career, they certain do take a toll at the “early career” stage.

Another thing I’ve been thinking about is my hypothesis based on personal observation that women enter graduate programs in ecology at an older age than men, on average. I think this may have to do with women having to have more experience to be seen as equally “competent” to their male peers. (Reviewer bias.) This would lead women to be older than men, on average, at the same career stage, and therefore less eligible for prizes/awards/fellowships/etc. based on age.

All that being said, I went to look at the call for proposals for the Waterman, and, in fact, the rule is “under 35” OR “within 7 years of PhD.” That is a very good thing, and I hope all awards will change so that we eliminate age-based awards and simply shift to awards being relative to something like PhD completion. (Although that continues to have a baby penalty probem; at least it’s a step in the right direction.)

Thanks for noting that “OR” portion of the requirement. I had missed that, and it certainly helps.

The baby penalty and biases in terms of who is viewed as competence also argue in favor of trying out a system of having two winners, on of whom must be a woman. But, as discussed above, it does run the risk of people devaluing the award.

Yeah. I read another article recently via the Twitterverse that talked about gender biases (I think in the UK) in terms of who is viewed as “competent”. Its focus was on reviews (for proposals and papers) and how that cascades to all metrics of job evaluation. When double-blinded, there were many more papers by women accepted, suggesting unconscious (hopefully!) gender bias. But blinding for proposals appears more problematic because “reputation” of researcher and institution are more important than with papers. Seems like it would be equally challenging for awards. The author suggested 3 avenues for dealing with unconscious gender bias:
(1) blinding, wherever possible
(2) the sorts of awareness training you suggest
(3) quotas, which is in effect what it would be like to have one award for men, one for women

There is considerable pushback, from what I can see, for #1 and #3. I don’t think all of it is valid. But I also think the people who push back against #1 and #3 don’t think #2 will really work, and so don’t feel the need to push back.

In any case, I’d be very honored to accept a completely unprestigious award with a million dollars attached… 🙂

I’ve been wondering about this–how to rethink an early career award to minimize bias. If an award is based somewhat on productivity and is set for a longer time span, it seems like there’s still a bias. If someone has 7 years to produce but someone else has 4 years due to life circumstances, the bias might still be there. I’m not sure how long it would take to equalize and still be early career. Or perhaps an applicant could choose, say, three publications done within a span of time to be considered rather than a whole corpus?

Very interesting thoughts. It’s a good question. And those other life events could be anything (e.g., birth or adoption of a child, illness of a parent or other loved one, etc), as you know. I think the idea of allowing the applicant to choose a certain number of publications from within a certain time span is a really interesting idea that could help counter that.

What can be done? Well, scanning through the historical data, I see that there was a run of 3 women winners of the Waterman out of 5 immediately before this 11-year streak — which just happened to coincide with the years that Rita Colwell was heading up the NSF. One suspects the lack of a coincidence. I don’t know if she would have explicitly pushed the committee to be inclusive or just led by example, but it’s a good illustration of how just having women at the top rank can help with inclusivity.
I happen to know two of those three women, one slightly and one reasonably well. I think the comments about there being particular barriers to women for an under-age-X award may have something to them, because the winner I know best married and had children “late” by society’s standards– after she had done the work cited in the Waterman, in fact. Again, coincidence? Probably not.

I hadn’t noticed the pattern with regard to the timing of Rita Colwell being director, but Jonathan Eisen noted this on twitter, too. It is definitely noteworthy. It will be interesting to see if the number of women awardees increases now that a woman is taking over as NSF director once again.

To point out how strong the biases that we need to fight against are, I want to comment on the effects of award diversity on the perceived ‘value’ of the awards. I received a major award presented to relatively young scientists that, like the Waterman, is not field specific. It is also fairly evenly presented with respect to gender (~60/40). More than a few male colleagues commented that the award is biased towards women (??) and discredited my achievement with that fact. These were my colleagues and they were willing to say this to me directly. We’ve got a big hill to climb.

Ugh, depressing. As noted above (by myself and others), this is the main concern with making it clear that gender will be taken into account (e.g., by requiring one winner be a woman). In discussing seminar invitations, a few people have talked about inviting only women one semester and seeing if anyone noted or commented. I’m not sure how any of those experiments turned out, but it would be interesting to know!

Would the concerns being discussed be mitigated to any degree by having the award committee publish criteria for the award? Might this not make it easier, and more transparent, to identify which early-career scientists, independent of sex, meet the criteria? A problem w this suggestion, of course, is that there are intangibles that pertain to any selection [e.g., using rate of publication to assess ability to work without interruption].

Having a defined set of criteria that are established before looking at any of the applications is usually a good way to minimize bias. The problem here is that, given how diverse the potential winners are (sociologists vs. pure mathematicians vs. ecologists vs. engineers), I suspect it would be very hard to come up with particularly concrete criteria (but I may well be wrong). But it’s certainly worth trying.

One thing that I don’t think is discussed enough on this topic is the compounding effect that feeds forward and becomes particularly apparent in these top tier awards. NSF admits that they solicit nominees based on PECASE winners. If that award was also gender biased, the bias feeds forward into this award without any implicit bias from NSF. If there is increasing bias in each round of the selection process, there are fewer and fewer women as the award becomes more selective. This would track with males assuming that awards with fewer female recipients are somehow more prestigious (Please note – I strongly disagree!). Splitting awards based on male and female recipients would only exacerbate this effect, reducing the implicit quality of awards earned by women.

Alternatively, if awards are how we gain visibility in our field, we can change the source from which we draw candidates for seminar invitations, job interviews and young investigator awards. Perhaps then, the effect will trickle up into more women being considered for these larger awards. I like the list idea, but perhaps it could incorporate both males and females, being more of a ‘rising stars in the field’ list.

Something that I like about Anne’s List (as I understand it, at least), is that anyone can nominate herself (or a colleague). They have three categories (as well as a breakdown by subfield): tenured, pre-tenure, and senior postdoc. Interestingly, though, when I clicked on the “senior postdoc” link, there is no one listed.

I think having a list of people who might be considered rising stars would be fine, but I would think it should be a separate thing, and that there would still be value in having a list of women in ecology and evolution. (That’s just my opinion, of course!)

HI,
The PECASE awardees are remarkably close to 50/50 gender, government wide. I find that it is an outstanding pool of female scientists. I really appreciate your comments. Please send me any recommendations to: mmontros@nsf.gov.
Thanks!
Mayra Montrose

One strategy that has been successful at my institution is to have a ‘nominations committee’. This group looks for university/national/international awards and society fellowship that faculty in our department are eligible for and works to put in nominations. Since it is their job, they put in consistent effort and consider strategy at a department level. As a result, more women have gotten awards/become fellows. I think because the committee is more directed, it relies less on individual effort and is less idiosyncratic. I also suspect that having a full list of faculty eligible for a specific award in front of them reminds them of people that might not otherwise come to their mind first, e.g. women and other underrepresented groups. Heck, even just having more nominations from a committee like this likely improves the representation of woman at our institution relative to men elsewhere even if they are still underrepresented relative to men at our institution. This obviously doesn’t solve the big problem but it is something practical to try now.

I think this is an interesting post and interesting commentary. I have been on several awards committees for high level awards within societies. One observation I made was that when the nominating committee consists of men and women, the women tend to nominate women and men and the men tend to nominate men. OF COURSE, THIS IS NOT TRUE ALWAYS! Just a tendency. When I have been the chair of the awards committee, I then had the opportunity to put the list of nominees from each committee member on the ballot, and then notice that both men and women voted for men and women. In other words, voting seemed gender-unbiased but the nomination process seemed unintentionally biased. (I have no statistics here). My conclusion, not based on statistics, was that women somehow thought of women to nominate but as often as not, men did not.

The place to focus seems to be getting more women nominated. I have worked on doing this myself, as have many men and women. Making appropriate nominations often takes a lot of time and effort however and sometimes a person needs to be nominated multiple times (more work). For example, I have to admit that I have nominated women for awards, and then had the nomination rebuffed, and then not re nominated the person (just due to the work involved and general time constraints). Since there are fewer women than men at higher levels in academia and science in general, if the nomination packets tend to be gender-biased by the gender ratio of nominators, this will mean fewer women are nominated by high level people. We cannot rely on just senior women to do all the nominating of women! We need to help more women get nominated by both men and women.

I agree that getting more women nominated is key. Happy’s comment above indicates this, too. This is part of why I thought it might help to contact the chairs of PECASE winners — my hope is that doing that would increase the number of nominees, and perhaps especially increase the number of women who are nominated for the Waterman Award.

From the NSF Advance work we found that situations that make schema activation (thence bias) are haste, little information, and close to equal candidates. All of these seem likely in the award process.

Ranking candidates is also discouraged. That seems likely in such a process.

The study that comes to mind is:
Wenneras, C. & Wold, A., “Nepotism and sexism in peer-review,” Nature. 387(1997): 341-43
which found that female applicants to the Swedish Medical Research council had to be significantly more productive to earn an award. Since it is impossible, practically, for one of these ultra-qualified candidates to be that much more productive, it stands to reason that the award is difficult to obtain.

The RAISE Project is a good source of data on awards, and the site includes advice for individuals and institutions on how to increase the chances of women and minorities being nominated and receiving awards in STEMM http://www.raiseproject.org/index.php

In the concluding paragraph Meghan Duffy asks “Does the diversity of award winners [matter]?” The underlying assumption of this statement is that the award winners lack diversity. However, when you look at the fact that there are people of Italian, Indian, Chinese, and French origin who have won the Waterman in the last 10 years, I’d certainly push back on the notion that these award winners are not diverse.

Copyright

(C) 2011-2017 by the author of each individual post (specifically Jeremy Fox, Meghan Duffy, Brian McGill, or as otherwise noted at the top of each post).
The copyright holders have made these posts available on the Dynamic Ecology website at the present time for reading and commenting to benefit the scientific community. Hypertext links to posts which transfer readers to our website are also welcome. However, the authors retain all other rights to the posts including the rights to republish elsewhere and to charge for access. The authors also prohibit other uses including copying or republishing entire or substantial portions of posts without the author's permission, but do allow quoting small sections as allowed by fair use law for purposes of commentary and criticism.