Post navigation

Scoring Analysis with Funding and Investigator Status

My previous post generated interest in seeing the results coded to identify new investigators and early stage investigators. Recall that new investigators are defined as individuals who have not previously competed successfully as program director/principal investigator for a substantial NIH independent research award. Early stage investigators are defined as new investigators who are within 10 years of completing the terminal research degree or medical residency (or the equivalent).

Below is a plot for 655 NIGMS R01 applications reviewed during the January 2010 Council round.

A plot of the overall impact score versus the percentile for 655 NIGMS R01 applications reviewed during the January 2010 Council round. Solid symbols show applications for which awards have been made and open symbols show applications for which awards have not been made. Red circles indicate early stage investigators, blue squares indicate new investigators who are not early stage investigators and black diamonds indicate established investigators.

This plot reveals that many of the awards made for applications with less favorable percentile scores go to early stage and new investigators. This is consistent with recent NIH policies.

The plot also partially reveals the distribution of applications from different classes of applicants. This distribution is more readily seen in the plot below.

A plot of the cumulative fraction of applications for four classes of applications with a pool of 655 NIGMS R01 applications reviewed during the January 2010 Council round. The classes are applications from early stage investigators (red squares), applications from new investigators (blue circles), new (Type 1) applications from established investigators (black diamonds) and competing renewal (Type 2) applications from established investigators (black triangles). N indicates the number in each class of applications within the pool.

This plot shows that competing renewal (Type 2) applications from established investigators represent the largest class in the pool and receive more favorable percentile scores than do applications from other classes of investigators. The plot also shows that applications from early stage investigators have a score distribution that is quite similar to that for established investigators submitting new applications. The curve for new investigators who are not early stage investigators is similar as well, although the new investigator curve is shifted somewhat toward less favorable percentile scores.

It is not appropriate for me to comment on particular applications. Recall, however, that new investigators are defined as those who have not received substantial NIH funding. This pool includes investigators at many career stages and is quite diverse with regard to past and current funding situations from other agencies.

is there a difference between the pools in the fraction of applications that are ND?

The explicit study section rules (IME) to not make NI/ESI investigators disproportionally suffer the triage burden means that there is an artificial distortion in the dataset for this measure.

I am not so sure it would be an informative analysis.

Director Berg, I am surprised that your Institute receives more Type 2 than Type 1 applications. Is this common across ICs? I suppose I usually see numbers for Type 1s that include all of the NI/ESI apps so perhaps that is the reason for my perception.

With regard to the second question, the percentage of each class of application that was discussed was as follows:
Early Stage Investigators: 66%
New Investigators (not ESI) 49%
Established Investigator (Type 1): 50%
Established Investigator (Type 2): 77%

Jeremy, it would be great to see similar data for pre-doc and post-doc individual NRSAs: the distribution of impact scores, and color coding of awarded/non-awarded. In the regard, could you explain why it is that NIGMS does not percentile NRSAs?

These data are extremely helpful. I am an ESI and trying to learn about the funding system.

I have a perhaps naive question. Do the data include R01’s submitted to all RFAs/PAs/PARs, or only those to PA-07-070? Or in other words, do other RFA/PA/PARs consider ESI/NI status in funding decisions?

I wondered if your IC would be willing to break down these results into new and revised applications (A1s and A2s since A2s were likely accepted for this). I am curious how many new investigators get awards on their first submission compared to more established investigators.

Are there any graphics available that sort for established investigators receiving awards by institution (e.g. state funded academic, private academic, research institutes and commercial)? Are there any graphics available that sort between currently funded established investigators receiving awards versus unfunded established investigators receiving awards?