Evidencebased medicine

Evidence-based medicine (EBM) has been described as the 'conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients' [1]. Because there are so many biomedical journals (perhaps as many as 30000), the chance of any practitioner being aware of all the developments of interest is vanishingly small. The philosophy of EBM, therefore, extends into ways of summarizing information to make it understandable and useful. The key tool is the systematic review, and most work on systematic reviews, and indeed on EBM, has concentrated on treating disease.

Reviews are called systematic when they include a thorough search for all published (and sometimes unpublished) information on a topic. Empirical observation in systematic reviews of treatment efficacy demonstrates several sources of bias occurring because of the architecture of study design. The ones we know of are:

Systematic review up to 40%, or even change the conclusions of a review. Including only randomized studies is likely to be sensible for reviews of the effectiveness of treatments.

Blinding

Open (nonblinded) studies overestimate treatment effects by about 17%.

Quality

Studies of lower reporting quality overestimate treatment effects.

Quantity

Small studies can overestimate treatment effects.

Duplication Trials may be reported more than once. This may be legitimate, but is often incorrect and without cross-referencing. Unrecognized duplicate publications can lead to an overestima-tion in treatment effects of 20%.

Now, not all of these sources of bias will occur in each circumstance, but some will, and there may be others that are yet to be identified. What the systematic review process teaches us about trials of effectiveness is that there are many sources of potential bias, and we may not know all of them. It is notable is that every one we know of tends to overestimate the effects of treatment. There are other factors that may be important as potential sources of bias, particularly issues relating to the validity of experimental design in specific clinical situations.

Since systematic reviews concentrate on all the worthwhile published material on a topic, they provide the basis for a fresh look at where we are. One of their main results is to refresh the research agenda. A particular example is the increasing concentration on outcomes - the change in a disease state that is worthwhile for patients, their carers or the healthcare system. All too often, research concentrates on what is measurable, rather than what is meaningful. The large, simple, clinical trial with patient-defined outcomes may be one of the most important developments of EBM.

Size

Clinical trials are performed in order to tell whether one treatment is better than another. The statistical power of the trial is calculated on the basis of being able to say with confidence that there is a difference. It is the direction of the effect that is being measured. However, most of the time what we really want to know is the magnitude of the effect of treatment. To do this, we need much more information - perhaps 10 times as many patients need to be studied.

Figure 1.1 shows the results of 56 meta-analyses of placebo in about 12000 patients in acute pain trials [2]. Overall, 18% of patients given placebo had more than 50% pain relief over 6 hours. All trials in the meta-analyses were randomized, all were double blind and all had the same outcomes measured in the same way. The variability with small samples is huge, from 0% to nearly 60%. Only when the sample is above 1000 patients given placebo is the true rate measured.

This is just one example of how small studies can be affected by random chance. This should not be surprising: calculating confidence intervals around small samples will demonstrate that uncertainty is large with small samples. However, it serves to illustrate the power of random variation with the use of small samples, and why it is dangerous to extrapolate from a single small trial to clinical practice.

Figure 1.1 Per cent of patients with at least 50% pain relief from meta-analyses of acute pain studies. Each symbol represents one meta-analysis; all trials were randomized and double blind and with the same outcome measured over the same time (2). Size of the symbol is proportional to the number of patients included. The vertical line shows the overall average response (18%) from over 12000 placebo patients.

Figure 1.1 Per cent of patients with at least 50% pain relief from meta-analyses of acute pain studies. Each symbol represents one meta-analysis; all trials were randomized and double blind and with the same outcome measured over the same time (2). Size of the symbol is proportional to the number of patients included. The vertical line shows the overall average response (18%) from over 12000 placebo patients.

Expressing results

EBM has a real problem in how to express the results of research so that they can be understood and used. Statistical significance is in itself an unhelpful output, as are odds ratios, risk ratios, relative risks, weighted mean differences or effect sizes. The simple fact is that few people understand them and even fewer can use them.

What catapulted EBM into the real world was the use of the number-needed-to-treat (NNT). This is the inverse of the absolute risk reduction and describes the therapeutic effort required to produce one patient with the required clinical outcome [3]. It has proved particularly useful when there are many different treatments, as in analgesics for pain. By producing tables of NNTs for analgesics, choice can be made in terms of efficacy, harm and cost.

However, better understanding of the requirements of large samples to assess clinical outcomes accurately [4] is likely to lead to even simpler outcomes than the NNT. The future holds the prospect of being able to say, with confidence, that a given treatment in patients with a given disease and severity will lead to a successful outcome in x% - which would be understandable by doctors, patients and policy makers.