The CEBM criteria are the most prominently used among the core group of
EBMers. They are very detailed.
The SORT criteria are more recently introduced (February 2004) and have the
advantages (and disadvantages) of simplicity. For practicing clinicians,
the SORT criteria seem easier to interpret.
To facilitate interpretation of level of evidence grading by practicing
clinicians who may not take the time to read about the underlying rules,
the DynaMed Editors chose to use the SORT criteria and add brief phrasing:
level 1 (likely reliable) evidence
level 2 (mid-level) evidence
level 3 (lacking direct) evidence
grade A recommendation (consistent high-quality evidence)
grade B recommendation (inconsistent or limited evidence)
grade C recommendation (lacking direct evidence)
We have not formally studied the result, but it appears to be going
well. From the perspective of processing information for a clinical
reference, distinguishing evidence as level 1 (likely reliable) vs. level 2
(mid-level) appears more useful than purely distinguishing evidence by
study type.
In this model, level 1 labels require the best study type within a category
(e.g. randomized trial for treatment, inception cohort study for prognosis)
PLUS meeting a set of quality criteria for that study type. Other rating
systems typically require quality criteria for randomized trials to get the
level 1 rating, but the SORT criteria provide more details for this than
some other systems.
The Delfini system is an excellent system as well. We chose SORT in part
because of the potential for wide acceptance, as it was created by mutiple
leading journals in family medicine in the US working together and agreeing
to use it.
There are other approaches (such as the efforts of the GRADE working group)
trying to collaboratively develop the "standard" for a level of evidence
system for many to use, but these efforts have to deal with the tensions
between using a small number of levels vs. a large number of levels, and
exactness/detail-level vs. simplicity. In addition, different frames for
what is being measured (studies vs. collections of studies vs.
recommendations) and different target audiences (practicing clinicians vs.
researchers vs. guideline developers) complicate the decision-making for
choosing an ideal level of evidence rating system.
Getting back to the original question for this post, regarding how to rate
a systematic review with one randomized trial and many non-randomized
studies, here are some additional considerations:
The rules may vary with different labeling systems.
A system could allow a systematic review which includes a randomized trial
to get a level 1 rating (or whatever the highest rating is in that system),
but this could be misleading if applied indiscriminately.
The level of evidence would most accurately be applied if based on the
outcome and the data for that outcome---this could results in different
levels of evidence being reported for different outcomes mention in the
same systematic review.
A systematic review that covers high-quality and low-quality evidence, and
finds consistent high-quality evidence to support an outcome, could
appropriately give that outcome the highest level of evidence rating.
If the support for an outcome is completely based on low-quality evidence,
then the level of evidence should not be the highest rating, regardless of
the quantity of studies involvved.
If the support for an outcome is completely based on low-quality evidence,
then the level of evidence should not be the highest rating, regardless of
the quality of the systematic review. A systematic review could be very
high quality but find limited evidence. The quality of the systematic
review cannot change the quality of the underlying evidence.
Brian S. Alper MD, MSPH
Editor-in-Chief, DynaMed (http://www.DynamicMedical.com)
Founder and Medical Director, Dynamic Medical Information Systems, LLC
3610 Buttonwood Drive, Suite 200
Columbia, MO 65201
(573) 886-8907
fax (573) 886-8901
home (573) 447-0705
"It only takes a pebble to start an avalanche."