Grant Reviews, Part Three: An Insider's View of the NIH Review Process

A team of referral officers, scientific review administrators, program officers, reviewers, expert specialists, and NIH staff is responsible for handling grant applications from this point on. There are two rounds of review:

Study Sections: External reviewers determine the scientific merit of applications. They are not affiliated with an NIH institute.

Advisory Councils: Scientists from universities and research institutes, as well as other experts, approve the study section reviews and recommend to the institute director which applications to fund. They are affiliated with an NIH institute.

Together, these officers select review groups, choose primary reviewers, determine which institutes or centers will fund applications, and communicate with the applicant. So a proposal should be understood by many people, not just the reviewer!

Tough Choice #1: Which Study Section?

A dozen or so NIH-employed referral officers (all of whom have advanced degrees) examine applications and decide which NIH initial review group (IRG) should get them. For the majority of applications, these review groups are categorized into three main areas:

Molecular and Cellular Mechanisms

Clinical and Population-based Studies, and

Physiological Systems

Assigning applications can be a considerable undertaking, because the network of study sections is vast. The Division of Molecular and Cellular Mechanisms, for example, is split into seven different IRGs, ranging from "Cell Development and Function" to "Infectious Diseases and Microbiology." Each of these is further divided into numerous study sections--almost 50 in total.

Despite their expertise, referral officers never have enough time to interpret a project's goals in a leisurely fashion. "We encourage people to suggest the appropriate IRG or study section, or the areas of expertise best suited to reviewing a proposal," recommends Fisher. As it turns out, only a quarter or a third of applicants actually do this in their cover letters.

Tough Choice #2: Which Institute?

Referral officers also must choose, from NIH's 25 institutes and centers, the one most appropriate to fund an application. The decision to grant an award is made by an institute, not by the study section. After second-level approval by an institute's advisory council, funds will be made available from that institute's budget.

"Applicants should request secondary assignments," advises Robin Barr, deputy associate director in the Office of Extramural Affairs at the National Institute on Aging. This "lets other institutes get full access right from the start to an applicant's proposal," which may increase its chances of being funded (although in reality less than about 2% of applications are actually shuffled around institutes this way).

Contact!

Usually within 6 weeks after an application's arrival, notification letters are sent to the applicant and their grant administrators telling them which study section has been assigned to review the application, who the scientific review administrator is, and how to contact these officials. Applicants also learn which institutes or centers have been selected to possibly fund the application.

Receiving this notice is a central step in the review process, which means your complete mailing address is a necessity! Call the Division of Receipt and Referral (301-435-0715) if you have not heard anything after 6 weeks. Unfortunately, there is a lack of feedback from candidates. "It still surprises me that applicants treat this like a black box," exclaims Fisher. "When funding does not occur after the 9-month review process, then they contact us," she says, urging applicants to be aware of important review stages and contact NIH officers when they have questions or concerns.

But do your homework first! If you feel your application has been inappropriately assigned, find out more information before you pick up the phone. Check these Internet sites, for example:

The scientific review administrator (SRA), who heads the study section, decides which reviewers are most suitable to review an application and whether there are (or ever have been) conflicts of interest between reviewers and applicants, such as collaborations, co-authorship on research papers, or other interactions. If so, those reviewers are not sent the relevant application.

Applications are mailed to the reviewers 6 to 8 weeks before they meet. By then, the SRA has assigned to every application one primary reviewer, one or more secondary reviewers, and one or more readers. Outside opinions from individuals who do not come to the meeting, but who send in written comments, may also be sought. Each reviewer evaluates usually no more than six applications, but this depends on the number submitted and the size of the study section.

All reviewers must identify and critique research grant applications "believed not to rank in the top half for scientific merit." This step considerably reduces the workload and allows reviewers to focus on applications that stand a chance of being funded. This list of "streamlined" applications is discussed further at the meeting.

Review criteria include:

Significance. Does the study address an important problem?

Approach. Are the methods appropriate to the aims of the project?

Innovation. Does the project employ novel concepts or methods?

Investigator. Is the investigator well trained to do the work?

Environment. Does the environment contribute to success?

The Study Section

Study sections meet 5 to 6 months after application submission (which occurs three times a year), usually in a hotel near NIH in Bethesda. The panels vary in size but are typically made up of no more than two dozen reviewers selected by the SRA. Reviewers serve a 4-year term, receive an honorarium of $200 per day, and are reimbursed for travel and hotel expenses.

The Priority Score

Applications are usually scored between 1.0 (excellent) and 5.0 (very poor). Because streamlining removes the bottom half of applications, the scores given are usually between 1.0 and 3.0. However, it only takes one vote or request from a reviewer to bring a streamlined application to discussion.

The chair, who is also a reviewer, asks the primary and secondary reviewers to tell the study section how enthusiastic they feel about an application. They then proceed to summarize their reviews; after discussion, which potentially involves the entire study section, they may change their rating (for better or worse) and state their final priority score. The reviewers then analyze the budget request and recommend that it be reduced, left unchanged, or, in some cases, increased.

From either their own analysis or the discussion, the other reviewers privately score the application on their vote sheets. After the meeting, the SRA enters all these ratings into a computer, calculates an average priority score for each application, and ranks them ("percentiling"). All applicants (both priority-scored and unscored) are notified 2 weeks later.

For the sake of confidentiality, nothing is allowed to leave the building. So, after review of an application (which can last anywhere from 5 minutes to an hour), the reviewers perform one of their favorite stress-reducing rituals: In one synchronized moment, they fling their copies into huge cardboard boxes in the middle of the room. All applications, whether they are deemed outstanding or terrible, are then shredded together.

The SRA writes a "summary statement"--the official review record. It includes the reviewers' comments and a summary of the discussion for each application (unscored applications include only the reviewers' critiques). The statements are mailed to candidates 6 to 8 weeks after review.

NIH sends this information to applicants for their benefit. Unsuccessful applicants beware! Study sections do not look favorably upon proposals that have been resubmitted without addressing the previous reviewers' comments.

To Fund or Not to Fund?

Members of the institute's advisory council, selected by the referral officer, meet in January or February to decide which of the June applications to fund. They convene three times a year, coordinating with different deadlines.

Each institution receives applications reviewed by different study sections, which can pose a problem: Mediocre applications reviewed by one study section may end up with the same priority score as applications considered to be fairly good by another. Percentile ranking solves such predicaments, because it allows a council to identify outstanding applications, regardless of the study section under which they were reviewed. Summary statements also help distinguish between proposals with similar priority scores.

Based upon such materials, the advisory council members make recommendations to their institute. The institute's director and other staff members reach their final decisions after considering both the opinions of its advisory council and the study section review statements. Then the awards are made.

"Applicants need to avail themselves of all of the information possible, both in preparing applications for submission and in monitoring the process afterward," summarizes Fisher. And successful applicants are not completely out of the woods. They're about to enter yet another administrative phase of their career: accounting and management of all those federal dollars.

The pace is relentless. But despite the torrent of applications that surge into NIH three times a year, it isn't a lottery; the staff at the Division of Receipt and Referral prepare themselves as yet another cluster of FedEx vans round the corner and pull up to the temporarily bare loading docks ...