Systematic Reviews

The highest level of evidence about medical interventions

How can we know which medical treatments are helpful and which are pointless or potentially damaging? To begin with, we need solid evidence in order to decide how effective and safe a specific treatment is. But what exactly does SOLID evidence mean? According to medical science, there are several levels of evidence and these levels vary in reliability, principally based on THE WAY the evidence is gathered.

From top (most reliable) to bottom (dubious), the commonly accepted hierarchy of evidence includes:

Systematic reviews

Meta-analysis

Clinical trials

Observational studies

Personal experience

The highest level of evidence, systematic reviews, is the topic of this article and what CureFacts' rating and recommendations are based upon. But before we elaborate on systematic reviews, let us cover the other, lower and less reliable, levels of evidence.

A private observation by a patient, caregiver and even a doctor, otherwise known as "I say so, based on my experience!" is regarded as the lowest level of evidence. The personal experience of the observer may be based on insufficient data (not enough individuals that tried the treatment), and is certainly prone to bias, since the subjective beliefs of the observer about the treatment may lead to a self-fulfilling prophecy. For this reason, personal observations and deductions are, in fact, nothing but uncertain assumptions – that should be tested rigorously and objectively in order to draw valid and reliable conclusions. Observations and assumptions have a very important role as the first steps in gaining knowledge. Nevertheless, these are only the first steps in a long road towards solid evidence.

Observational studies provide the next level in the evidence hierarchy. These studies attempt to check an assumption (for example – that champagne reduces the chances of a heart failure), by comparing a population that takes the suggested treatment (drink champagne every day) to a population that does not take this treatment (never drinks champagne), in an attempt to find assumed differences in their health condition (differences in the frequency of heart failures, in this case).

Observational studies are more reliable than personal observations: They check very large populations (often millions of people), and they compare populations in a systematic and objective manner. However, they still include significant flaws.

One problem with observational studies is that the collected data is often based on what people declare ("How frequently do you drink champagne?"). Unfortunately, people tend to over or under estimate their own behavior ("Well, I only drink champagne after dinner, and that does not count, does it?").

The second, and much more crucial flaw in observational studies is that you do not really know whether the assumed treatment (Champagne) is the actual cause of the benefit (less heart failures) or is merely associated with the benefit due to an entirely different reason. In our case, it is very likely, for example, that people who can afford drinking champagne every day enjoy better nutrition and less stress (at least financial stress) than the non-champagne drinkers.

In some cases, observational studies make sense, particularly if there is no practical or ethical way to conduct controlled studies (for example – clinical trials). However, the level of confidence that we can gain from observational studies is quite limited.

This brings us to the next level in the evidence hierarchy – clinical trials (also known as RCT – Randomized Controlled Trials). These trials provide better evidence, since the testing is done in a controlled environment (the treatment is provided and the results are measured by the researcher). RCTs also include further measures to avoid bias. The comparison is between two groups of participants that were randomly selected (to assure that the groups are similar in all aspects other than the treatment they receive). In addition, one group receives the assumed treatment (for example – a new type of painkillers) while the second group receives a "control" – a similar looking treatment with a known effect (for example a white pill with no active ingredient in it – an ineffective Placebo, or an older, well researched type of painkiller). Moreover, both the patient and the doctor are "blinded" – they do not know if any specific patient receives the assumed treatment or the "control".

With such rigorous measures, it sure seems that clinical trials provide absolute certainty about the tested treatment. Unfortunately, this is not necessarily so. There are many ways to manipulate the results of clinical trials, and there are many incentives to do so. Researchers prefer to be recognized for finding effective new treatments (no one wins a Nobel prize, or ever gets their study published, for finding that a never-tried-before active ingredient does NOT kill pain). And pharmaceutical companies prefer to fund clinical trials that result in painkillers that they can actually sell. So, results are presented in a favorable way using sophisticated statistical manipulations, presumed benefits are re-defined to bring some good news, sub-populations are omitted from the results to make the results favorable, and some trials are simply unpublished because they do not add value to the assumed treatment ("Well, we hope other studies will add value to our brand new painkiller, and show it is effective, and then we will publish it, wouldn't we?")

So, if we cannot really trust a clinical trial as is, what should we do? To overcome this problem, meta-analysis was invented. It uses statistical methods that calculate an outcome (for example – the effectiveness and safety of a new painkiller) by combining together the results of several clinical trials that tested the same treatment. This way, the combined populations are bigger, and hopefully the results and conclusions are more significant and reliable.

Meta-analysis can be conducted only when the clinical trials are similar enough (for example – with similar populations, similar tested and "control" treatment, and similar measurements of outcome). And even when a meta-analysis can be performed, the results can be easily manipulated by "cherry picking" - including only clinical trials that show that an assumed treatment is helpful and excluding less "supportive" trials.

Which brings us to the highest level of evidence – Systematic Reviews – that summarize the information found in all valid clinical trials that were ever published. The advantage here is clear: no "cherry picking". All relevant clinical trials are included, including those that do not support the assumed treatment.

Systematic reviews are usually done by two volunteering doctors, for a non-profit independent organization (for example Cochrane organization, or a university), and are not easy to conduct. The doctors work according to a very strict protocol, for about a year, in the following steps:

Defining the question for their research ("Is treatment A for medical condition B in population C effective and safe?")

Searching in databases that include all the scientific articles about clinical trials (RCTs) that were ever published for all the clinical trials that were done about the defined treatment, condition and population. This can lead, for example, to a list of 15 clinical trials about treating common cold with Vitamin C in adults.

Checking each and every clinical trial on the list for its scientific validity and strength: Was there an assumed treatment and a "control" group? Were the patients in each group selected and assigned randomly? Were the patients and the treatment providers in each group "blinded" (not aware who gets the assumed treatment and who gets the "control")? Was the outcome of the treatment measured in a proper and relevant way? Were the side effects of the treatments explored and reported? Were all participants included in the final results? Was the research funded by a pharmaceutical company, or was it an independent study?

The validation step in creating a systematic review takes a long time, and attempts to rule out all possible biases in each study. Following this step, invalid RCTs, that contain significant flaws and biases, are excluded, and only valid clinical trials are left for the following steps (for example – 7 valid clinical trials about Vitamin C for treating Common Cold in adults, out of 15 RCTs in the initial list).

A meta-analysis is performed on all these valid RCTs, if possible, in order to combine all the valid data about the assumed treatment. As usual with meta-analysis, in some cases it is not possible to statistically combine the data, due to significant differences between the trials.

During the creation of a Systematic Review, two doctors typically follow a protocol to validate each RCT separately, and then compare their conclusions with each other, and discuss and overcome any discrepancies. In addition, their work is reviewed by a group of content experts (for example – Cochrane's Acute Respiratory Infections Group), to assure accuracy and quality.

Following all these steps, a systematic review is published, with a thorough description of the entire process: the question investigated, the data sources that were searched, the RCTs found and those that were proven valid, the accumulated results, the conclusions and the recommendations.

Systematic Reviews provide the highest level of evidence since they summarize all valid scientific evidence at the time they are published. However, even systematic reviews do not provide 100% accuracy, since they rely mainly on published articles, and usually do not include unpublished data that may change the overall results and conclusions. In addition, not all systematic reviews are alike: some were created using a more reliable and rigorous protocol such as Cochrane's, while others are created in a less formal way. Nevertheless, we believe that systematic reviews offer best available evidence about the effectiveness and safety of medical treatments, and this is why CureFacts' data relies on systematic reviews.

Currently, over 10,000 systematic reviews already cover a large part of medical treatments of all types – from surgeries and prescription drugs, to non-prescription drugs and food supplements and alternative medicine treatment. And while old reviews are being updated every few year (with newer RCTs included), hundreds of entirely new systematic reviews are created every year. Moreover, automated processes of creating systematic reviews (called Rapid Reviews) have been devised and calibrated recently. These Rapid Reviews enable the creation of reviews within minutes, instead of the rigorous manual work taking about one year and requiring two dedicated doctors. As a result, CureFacts expects to have all medical treatments covered by systematic reviews in the near future.

These are exciting times to live in, when more and more treatments can be avoided or used based on solid evidence. At CureFacts we work hard in order to bring this information to the public, in a user-friendly and intuitive manner, so that everybody will be able to make educated decisions about their healthcare, and enjoy longer life and better life quality. If you have actually read this entire article, your education has already improved, and you are headed in the right direction. We wish you good health, and helpful treatments when needed.