Panel Urges Tighter Review of Research-Grant Proposals

If the Department of Education is going to raise the credibility of
the research it supports, a new report concludes, it must bolster its
system for outside review of grant proposals.

For More
Information

Free copies of "Strengthening the
Standards: Recommendations for OERI Peer Review" will be available
after April 1 from the National Educational Research Policy and
Priorities Board at (202) 208-0692 or by e-mail at eve_bither@ed.gov.

At the direction of Congress, the department four years ago
tightened guidelines governing the peer-review panels that evaluate
proposals for agency funding. But the report, commissioned by an
advisory board to the department, suggests that the changes didn't go
far enough.

Some of the problems cited in the report include:

Instances of evaluation panels whose members had almost no
expertise in research methodology;

At least one reviewer who had not read the proposals before a
review meeting;

Complaints from applicants that reviewers had misunderstood or
misstated their proposals; and

Cursory written reviews of proposals that gave researchers
little or no substantive feedback.

The lack of experienced researchers on some of the panels is the
most serious shortcoming in the department's peer-review practices,
concludes the study released last month. For example, in a 1997
competition for "field initiated" grants--which are essentially funding
for researchers' pet projects--12 of the 35 reviewers who sat on peer
panels had no research experience or publications on their
r‚sum‚s.

Expertise Pays Off

"They're not peers if they're not engaged in research," said Carl F.
Kaestle, a prominent Brown University researcher who sat on the panel
that directed the study. Two private research firms--August and
Associates in Bethesda, Md., and Lana D. Muraskin of
Washington--conducted the study.

"The standards say all peer reviewers should have knowledge of
research as well as what's going on in the field," Mr. Kaestle
said.

Technical expertise is key, the report says, because it appears to
lead to better-quality reviews. The three panels whose members had no
research training produced no reviews that were rated "good" by the
study team. The opposite was true for panels that included researchers
working in the same fields as the applicants. Most of their reviews
were rated "good," and none was deemed "poor."

Part of the problem has been that reviewers with the know-how called
for in the guidelines are hard to find, the report points out. Besides
knowing something about the subject area of the proposed research,
reviewers need to have in-depth knowledge of research methods and of
educational policy and practice. And department staff members,
sometimes given only three weeks to recruit reviewers, have to mix and
match panel experts.

One remedy to the problem, the panel concludes, might be to
establish standing review panels for each of the department's five
national research institutes.

Changes Considered

"If you have sort of a one-shot, ad hoc meeting of people with all
those pieces of knowledge, you run the risk of having people who are
not really experts in any of those areas," said Kenji Hakuta, the
chairman of the National Educational Research Policy and Priorities
Board, which commissioned the report. The congressionally mandated
board endorsed the report at its January meeting. "With a standing
panel, you have to meet at least a minimal level of expertise for
everyone," Mr. Hakuta added.

Proposed three-year terms for panel members, recommended in the
report, would also offer more time for training reviewers.

Department officials are studying that idea, which may require
congressional approval, as well as the report's other
recommendations.

But some of the changes called for are already being tested in the
current competition for field-initiated studies. This year, study
proposals will be screened first through a panel of researchers, who
will assess their technical quality. The research designs making that
cut will then go before a broader panel made up of practitioners and
researchers.

"It's something we would've done whether or not we had the
peer-review report," said C. Kent McGuire, the assistant secretary in
charge of the department's office of educational research and
improvement. "But this report gives further credence to the need to do
something about peer review in this place."

Notice: We recently upgraded our comments. (Learn more here.) If you are logged in as a subscriber or registered user and already have a Display Name on edweek.org, you can post comments. If you do not already have a Display Name, please create one here.

Ground Rules for Posting
We encourage lively debate, but please be respectful of others. Profanity and personal attacks are prohibited. By commenting, you are agreeing to abide by our user agreement.
All comments are public.