3. evaluate the costs of implementing a formal peer review of search strategies programme as a part of the review process.

The investigators undertook a randomized controlled trial of peer review using the PRESS instrument versus “free-form” review of search strategies, at the publication of the research protocol phase of AHRQ reports, by predominantly qualified and experienced information specialists. Of those participants randomized to the PRESS intervention (20/25) only 11 went on to complete the study and use the PRESS instrument. Of these, most took less than 2 hours to complete the review using PRESS (91%), some thought it could be incorporated into their workflow (yes: 46%; maybe 46%); some would have been willing to take on peer review duties on a more permanent basis (yes: 37%; maybe 37%); most found the instrument helpful (82%); 54% indicated that they preferred having the checklist but not being required to use it and 27% said that they preferred the PRESS instrument and being required to use it. The PRESS instrument reviews contained more recommendations overall and had more comments that could be termed error detection.

Limitations stated by the author(s):

There was significant dropout in the study (about a third of the peer reviewers failed to complete all reviews, in an already small sample). The sample of original searchers was also small. Some of the wording in the questionnaire was ambiguous. The project focused on pharmacological treatment topics, as they were the more common type of effectiveness review within the AHRQ, and it was unclear whether these results would be generalizable to more diffuse topics.

Limitations stated by the reviewer(s):

In the Results section the Authors state that “In 97% of cases, the original searcher indicated that the comments did not cause them to alter their search strategies. However, it should be noted that only one original searcher indicated that this was because he or she disagreed with the review. In nearly all other cases, the reason given was simply that the report had already gone forward, and it would be too late to incorporate any changes suggested.” This is not elaborated upon further in the Limitations section of the report.