Announcer:
From the National Institutes of Health in Bethesda, Maryland, this is All About
Grants.

Megan:
Welcome to another edition of All About Grants. I’m your host, Megan Columbus,
from the NIH Office of Extramural Research. Today we’ll be talking about
understanding your summary statement. To get a couple of different perspectives
I have with me Dr. David Armstrong, the Chief of the Review Branch from NIH’s
National Institute of Mental Health and Dr. Mike Sesma a Program Officer, also
from NIMH. Both Mike and David have been on the other side of the table as
investigators competing for and receiving NIH grants. So, before we get into
the summary statement, let’s begin talking about your respective roles in the
review process. David, how about we start with you.

David:
We have the Scientific Review Officer who serves a variety of roles. One,
actually at the review meeting itself, he or she is the designated federal
official who must be present for the review meeting to go forward. This
individual is the expert there to answer any of the questions that the review
committee may have regarding procedural issues or issues related to the review
itself. Prior to the meeting it’s the responsibility of the Scientific Review
Officer to ensure that the applications we receive are complete and most
importantly, though, to understand the science in the applications that are
being reviewed and to select and recruit appropriate extramural scientists to
serve as reviewers to evaluate the scientific merit of that application.

Mike:
The role of the PO in all of this is a little bit less direct. Our role is to
represent our institute at the review, and we track those applications that are
assigned to our institute as they go through the review. Often a number of
those applications that will be reviewed by this scientific review group or
study section are applications that may actually be assigned to our individual
grant portfolios that we manage at the institute. Basically, we’re there to
take notes on the individual applications, noting the particular aspects of the
review criteria and strengths and weaknesses of the application and how they
respond to those review criteria.

Megan:
So how about we turn to the topic of summary statements. What exactly is a summary
statement used for?

Mike:
I’m usually the second one to read it after the applicant, and basically, summary
statements are used to obtain an independent and objective, expert opinion of
the scientific and technical merit of an application. So we count on the
reviewers and the panel to provide us with their best assessment of both the
significance, the importance of the science that’s being proposed, but also the
way it’s going to be done and who’s going to be doing it. All of those things
should give us a sense of the likelihood if that will be a successful project,
but also one that helps advance the field.

Megan:
So, David, what should I expect a summary statement to look like and how are
they constructed?

David:
In general, your application will be assigned to three independent reviewers. The
reviewers are asked to comment upon the strengths and weaknesses of the five
primary review criteria, as well as other criteria that may affect, or not, the
overall impact scores. As an applicant, when you receive your summary statement,
you will see these individual critiques collated together along also with the
various scores on a 1-9 associated with each of those five criteria.
Importantly, though, and this is the role of the Scientific Review Officer, if
you are one of those applications that is discussed during the meeting, you’ll
have a resume and summary of the discussion.

Megan:
So that summary is written by that Scientific Review Officer that is running
that meeting?

David:
Yes, that summary and resume of the discussion is written by the Scientific
Review Officer, and it is meant to capture the essence of that discussion,
not merely to repeat the highlights that are in the critique. And where it is
particularly useful to you as the applicant, when there may be disagreement
among the committee members as to the strengths and weaknesses, as well as to
the overall impact of a particular application. It’s the Scientific Review
Officer who will capture that minority voice or a majority voice, a dissenting
voice, is stating one view or another. So it’s in fact a very important role of
the SROs to capture, I really do stress, the essence of that conversation.

Megan:
The reviewer’s critiques are written before the meeting. Do the reviewers ever
go back and adjust their critiques based on the discussion?

David:
The reviewers have the opportunity to go back in to revise their critiques, The
SRO will certainly stress repeatedly and encourage the reviewers to please
change your written words and your criterion scores to match that which is at
the end of the discussion. So if you have a change of heart, let it reflect
that final thought that you have, both in written words and in your criterion
scores.

Megan:
But that may or may not happen. However, you can also contact your Program Official
who was listening to the discussion.

Mike:
Exactly. The role of Program at a review meeting is to basically sit in the room,
observe and listen. What we try to observe and listen, as each application is
reviewed during the discussion, is to get a feel for how the reviewers convince
each other of their particular assessment of the strengths and weaknesses of a
particular application. And often, as David said, since the individual
critiques are written by each reviewer and submitted prior to the meeting, often
the emphasis of their evaluation of the individual strengths and weaknesses,
using those review criteria, can shift as a function of that discussion or as a
function of questions asked of the reviewers by each other or by other panel
members who weren’t assigned to the review. And what’s important for us is to
be able to understand how the reviewers arrived at the position that they
ultimately end up with in terms of assessing the overall impact of the application
and arriving at a score that reflects both the scientific and technical merit.

Megan:
I assume that means that after I receive my score and summary statement, I
should call my program officer to find out if they can provide any additional
details about that discussion?

Mike:
Absolutely. Usually I get a call from an applicant as soon as they get their
score; they want to know what that means. And I always tell them that what you
have to do is wait for the summary statement and read it several times before
you call me and then we’ll have that conversation. Often my first question to
the applicant will be, “What did you read in your summary statement, what did
you think that the reviewers said about your application? Then with the notes
that I have I can basically either confirm or explain further and extend that
discussion a little bit to help the applicant understand what it was the
reviewers found compelling or unexciting or problematic or innovative in their
application.

David:
I think another thing to stress again, and it can’t be stressed too many times,
is that when the applicant receives his or her summary statement, what he or
she is receiving is the critiques of the three, plus or minus, assigned
reviewers. The overall impact score is of course the composite of all voting
members around the table, which could be 30, 40 individuals. We hope that the
critiques are somewhat comprehensive and do not stray too much. It certainly is
a practice of SROs, if there is a voice that may be non-conforming, to ask that
individual to please write a comment to be included in the critique, as well.
So those are common practices so that everybody has the full picture of what
actually occurred.

Megan:
Recently, NIH has been encouraging the use of a paragraph written by each of
the assigned reviewers to describe how they came to the overall score of that
application. What information is that intended to provide?

David:
So this is just starting with this current round of applications in terms of
providing, rather than a bulleted explanation of the strengths and weaknesses
that went into your consideration of the overall impact score, rather a narrative.
What the intent of this is again to capture your gestalt, so to speak, of those
elements that really contributed to your overall impact scores, not meant to be
a reiteration of your bullets that may be listed below under the various
criterion. We want a distinct narrative from the reviewer that says, “I think
this application is strong because of this, this and this. I think it may be
weak because of this and that.” So again, a clear statement as to what
components came to drive your final score. Once again, the overall impact score
is the synthesis of the five criterion scores, so it’s important that we know
to some extent what contributed most in terms of arriving at your overall
impact score. We hope it will be useful, and we think it certainly will be.

Megan:
Thank you both for joining us. We’ll talk to you soon when we talk about resubmissions.
For OER and NIH this is Megan Columbus.