First batch of reviews completed

Astronomers taking part in the first Fast Turnaround (FT) cycle had until February 14th to submit their grades and reviews. The Gemini FT support team met this morning to go over the results. We reviewed the instrument configurations, observing conditions, target RAs, and total time requests of the top-rated proposals, as well as any special considerations such as timing constraints. We then used this information to construct a list of proposals which will fit in the available time and stand a reasonable chance of being observed. This provisional list very closely follows the reviewers’ rankings even though, given that almost all GMOS configurations were requested and many of the proposals requested good weather conditions (see previous news item), we had been expecting that some fairly highly-ranked proposals may have to be excluded. Our next step will be to carry out technical assessments on the proposals that look like they will be accepted, replacing them with alternatives if serious problems are found. PIs will be informed of the outcome by the 21st.

We are planning to extensively analyze the review process as we gather data from this and subsequent cycles. However, the temptation to draw some preliminary conclusions has of course proved irresistible. Figure 1 shows the mean score of each proposal vs proposal rank, with the vertical bars indicating the range of scores received. Clearly, the range of grades for most proposals is considerable, with one even being awarded scores spanning the entire range from “poor” to “excellent”. However, reviewers uniformly thought highly of the two highest-ranked proposals (accounting for ~7 hours between them), and the three lowest-ranked submissions received no “excellent” scores. The review process appears able to identify the best proposals and agree on a handful that should not be accepted, while proposals in the middle of the range receive less definitive evaluations.

Figure 1: Mean proposal grade vs proposal rank. Proposals are graded on a scale of 0 (poor) - 4 (excellent), and the dashed horizontal line indicates the threshold of mean=2.0 that must be achieved for any proposal to be accepted. Vertical bars show the range of grades received by each proposal.

Reading the individual reviews was also instructive. For example, although the need for rapid response is only 4th on the list of assessment criteria, we noticed that most reviewers appeared to give this considerable weight in their grading. Indeed, one reviewer recommended that proposals not strictly requiring Fast Turnaround should only be assessed by the regular TAC. We will therefore need to decide whether to emphasize that this criterion is less important than overall scientific merit, or whether it should indeed be given more weight. If the latter, no changes will be made until the March call for proposals is issued (to avoid changing the rules as people are writing proposals).

Initial, informal feedback about the process has been very constructive, and will help us improve the system (for instance, by building in better safeguards against accidentally assigning too many proposals to one reviewer!). PIs and reviewers will be sent a structured feedback form along with their notifications later this week.

Categories

Meta

Legal Disclaimer
Some of the individuals posting to this site, including the moderators, work for Gemini Observatory. Opinions expressed here and in any corresponding comments may be personal opinions of the original authors, not of Gemini Observatory. The content is provided for informational purposes only and is not meant to be an endorsement or representation by Gemini Observatory or any other party. This site is available to the public. No information you consider confidential should be posted to this site. By posting you agree to be solely responsible for the content of all information you contribute, link to, or otherwise upload to the Website and release Gemini Observatory from any liability related to your use of the Website. You also grant to Gemini Observatory a perpetual and royalty-free right to exercise all copyright, publicity, and moral rights with respect to any original content you provide. The comments are moderated. Comments will appear as soon as they are approved by the moderator.