Readers' guide for review articles: Why worry about methods?

Readers of medical journals would be astounded to find an original article that did
not include a description of the research methods. On the other hand, they are not
surprised to find review articles that do not report the methods used to prepare the
review. In fact, until recently, almost no reviews included methods sections. Why,
then, do the criteria for selecting review articles for inclusion in ACP Journal Club articles insist on a methods section?

Indeed, some of our readers might ask why we include review articles at all. We do
so for 2 reasons. First, it is sometimes not feasible to test an innovation definitively.
Second, the completed studies on a given topic may not provide a clear answer; reviews
of inconclusive or conflicting studies can sometimes help sort out these situations.
In fact, because few definitive studies exist on any clinical topic, scientifically
sound reviews (or “overviews”) are needed for most clinical questions.

The reason for insisting on a methods section for review articles is the same as that
for the report of an original investigation: Without knowing how a conclusion was
reached, the reader has difficulty deciding whether to believe it (1). Readers of reviews that do not report the methods of review must accept the conclusions
of the authors on faith.

Many types of review articles exist, but those addressing practical clinical questions
should be, in fact, scientific surveys of the literature. The same principles that
apply to epidemiologic surveys apply to reviews: A question must be posed, a target
population of information sources must be identified and accessed, a reproducible
method must be used to select articles for detailed review, appropriate information
must be extracted from those articles in an unbiased way, and conclusions must be
derived that are supported by the evidence presented. In meta-analyses (quantitative
overviews), formal statistical analysis should be used to help derive conclusions.

As with other research, several methodologic decisions must be made when preparing
a review, and each one involves potential threats to the validity of the ultimate
conclusion. These threats include how studies are identified and selected for inclusion,
how the quality of studies is assessed, how much weight is given each study, and how
results are combined and interpreted. If authors do not describe how these decisions
were made, readers cannot assess the credibility of the authors' conclusions.

Unfortunately, reviews of the same topic often disagree, and, if the authors and peer
reviewers do not pay enough attention to review methodology, neither group can guarantee
the validity of a review. Experts preparing reviews risk being unduly influenced by
their own research, and their understandably strong opinions about their area can
lead them to judge other evidence more on its support for their convictions than on
its methodologic strengths.

ACP Journal Club's criteria for selecting review articles begin by insisting that they have a methods
section. This section enables readers to decide for themselves whether the review
has been conducted using reproducible methods and with safeguards against bias in
the selecting and interpreting of studies. Readers can then make informed decisions
about using the review's conclusions as the basis for clinical decisions that will
do more good than harm.