The fallacy of time-to-intervention studies

We are barraged by time-to-intervention studies (door-to-balloon time, time-to-antibiotics, door-to-needle, etc.). However, it must be kept in mind that these studies are purely correlational in design. Such studies cannot prove causation and, at best, are only hypothesis generating.Frequently they are downright misleading.

0

Time-to-intervention is an extremely complex variable which reflects properties of the patient, the health-care system, and their mutual interactions.This complexity yields multiple sources of confounding. Below are some examples of confounding variables which typically plague these studies.

0

(1) Treatment is often delayed in more complicated patients, including older patients, confused patients, or patients with multiple medical problems.Complicated patients present a greater challenge to diagnosis, which naturally delays treatment.

0

(2) Timing of one intervention will often correlate with timing of other interventions. For example, a patient in septic shock who receives antibiotics during the first 30 minutes is probably being managed aggressively and is more likely to receive prompt fluid, vasopressor support, imaging, and source control as well. Time to initiation of antibiotics may simply reflect how aggressively the patient is being treated overall, rather than the benefit of antibiotics themselves.

(4) Lower time-to-treatment time may correlate with arrival at the hospital during regular weekday hours (Saver 2013).Patients arriving at these times may fair better due to greater availability of physicians and immediate access to more diagnostic and therapeutic modalities.

0

(5) Over the last few decades, emergency medicine has become more timely and organized. Therefore, in any retrospective study evaluating care over a long period of time, prompt treatment will correlate with patients treated more recently with more modern therapies.

0

(6) Patients with more severe disease may receive attention and therapy sooner.Note that most confounders tend to amplify the apparent benefit of prompt intervention.This is the only confounder listed here which causes prompt intervention to correlate with worse outcomes.

0

Why we believe time-to-intervention studies

0

Despite understanding the difference between correlation and causation, we continually make the same mistakes when interpreting time-to-intervention studies. One of the reasons is confirmation bias.

0

0

As described above, most confounders tend to amplify the apparent benefit of prompt intervention.Therefore, time-to-intervention studies describe a fairytale in which medicine is all-powerful and our interventions are incredibly successful. You know that 95-year-old man with chronic renal failure, diabetes, and cardiomyopathy presenting with multiorgan failure and sepsis? According to Kumar, if you can get antibiotics in within the first half-hour his survival will increase to 80%.Did you get called about a 70-year-old woman with dementia and a massive stroke? Well, if you can just get a dose of TPA in right away, the stroke will go away and maybe her dementia will get better too. And if you can get her transferred out of the ED to the ICU quickly, her mortality will improve (Chalfin 2007). These studies usually reinforce our faith in the therapies that we provide, confirming how we practice medicine to be right. We believe them because we want to.

0

0

0

The unfortunate reality is that our therapies are not this effective. Consider, for example, the famous study by Kumar which related time-to-antibiotics in septic shock to mortality (figure below). According to this study, a woman with urosepsis who is finally dragged to the hospital by her family at 9:00 AM will have an 80% survival if she receives antibiotics before 9:30 AM, compared to a 60% mortality if she receives antibiotics at 11:00 AM.However, if her family gets stuck in a traffic jamfor two hours and presents to the hospital at 11:00 AM, then her mortality will be 80% as long as antibiotics are administered before 11:30AM.This 8% per hour reduction in mortality is implausibly enormous (e.g., Ferrer 2014 found a roughly 1% per hour reduction).Nonetheless, the Kumar figure is frequently shown at conferences because it is inspirational and dramatic.

0

0

Excessive time pressure: Community Acquired Pneumonia

0

One way time-to-intervention studies cause harm is by encouraging speed over accuracy.For example, in 1997 Meehan published a retrospective study in JAMA relating door-to-antibiotic time under eight hours to mortality for elderly patients with pneumonia (figure below).This study also found a relationship between measurement of oxygenation and increased mortality, which should have been a reminder that they were studying correlationsand not causation (sicker patients are more likely to have their oxygenation measured).Nonetheless, this study led the Centers for Medicare and Medicaid Services (CMS) to recommend that all pneumonia patients be treated within eight hours.In 2004, Houckpublished a similar study concluding that door-to-antibiotic time under four hours correlated with reduced mortality.This study was criticized because it actually showed an increased mortality for patients treated either within 0-2 hours or after four hours.Nonetheless, the Infectious Disease Society of America (IDSA), CMS, Hospital Quality Alliance, National Quality Forum, and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) adopted a quality-assessment benchmark requiring all pneumonia patients be treated within four hours.

0

0

Pressuring physicians to initiate therapy within four hours was subsequently shown to increase misdiagnosis and inappropriate antibiotic use, without significantly improving the timeliness of antibiotic initiation (Welker 2008, Kanwar 2007). Patients misdiagnosed with pneumonia received inappropriate treatment and suffered delays in reaching the correct diagnosis.In one outbreak of Clostridium difficile, 80% of affected patients were receiving antibiotics for a diagnosis of pneumonia, of whom only half truly had a pneumonia.This outbreak occurred following introduction of a “pneumonia care plan” to expedite early antibiotic initiation (Polgreen 2007).

0

Waterer 2006 demonstrated that delayed antibiotics related to altered mental status, absence of fever, absence of hypoxemia, and increasing age.After adjusting for these confounders, the relationship between time-to-antibiotics and mortality disappeared.This study revealed how flimsy the evidence was that initially supported the four-hour cutoff.Ultimately the four-hour benchmark was discarded.

0

Door-to-needle time in stroke thrombolysis

0

Currently there is fervent debate about the utility of thrombolysis in acute ischemic stroke (e.g., see EmCrit).Regardless of your opinions about this topic, it should be clear that the best way to resolve this issue is well-designed prospective randomized trials, not retrospective time-to-intervention studies.

0

Saver 2013and Gumbinger 2014 published retrospective observational studies correlating door-to-needle time with outcome in the US from 2003-2012 and in Germany from 2008-2012, respectively.Early thrombolysis correlated with better neurologic outcomes.Both papers concluded that earlier thrombolysis improved outcome, supporting efforts to reduce door-to-needle time.Saver reported a mortality reduction with early thrombolysis, which is surprising given that the 2014 Cochrane Review found that thrombolysis increases all-cause mortality within randomized controlled trials.The explanation of this disparity is that although Saver and Gumbinger seem to imply a benefit of thrombolysis versus no thrombolysis, in fact there is no control group and they are merely comparing early versus late thrombolysis.Thus, a more realistic conclusion from Saver's data may be that delayedthrombolysis is associated with increasedmortality.Ultimately, retrospective observational time-to-intervention studies spanning long time periods involve innumerable confounders, making them difficult to interpret.

0

Emberson just electronically published in Lancet a meta-analysis of nine randomized controlled trials relating time-to-thrombolysis to outcome.This study is profoundly flawed due to combination of heterogeneous studies.For example, all patients in the NINDS trials received treatment in under 3 hours, whereas no patients in ECASS-III or EPITHET were treated in this time interval.Attempting to compare patient outcome based on time, therefore, degenerates into a comparison between different study populations.Nonetheless, the authors forced all data into a regression equation (figure below), which seems to imply that thrombolysis is beneficial up to five hours after stroke.

0

0

This time-to-treatment analysis is invalid because there is no evidence that time functions as a continuous variable to affect the benefit of thrombolysis.For example, consider IST-3, the largest randomized controlled trial of thrombolysis in stroke (IST-3 contributed 45% of the raw data for this meta-analysis).IST-3 found no consistent relationship between time-to-treatment and outcome (Forest plot below).Similarly, the 2014 Cochrane Review found little impact of time on outcomes up to six hours, stating that “this should not be interpreted to mean that time to treatment is unimportant, but rather that other factor(s) like stroke severity may have confounded the association.”Nonetheless, Emberson's study finds a relationship, likely due to confounding factors (for more see SmartEM).

0

0

What is the ideal time-to-intervention?

0

The ideal time-to-intervention is a known unknown. It is extremely difficult to study this rigorously.For example, a Cochrane review of early versus late antibiotics for sepsis concluded that there was not a single high-quality study on the topic.It is unlikely that we will ever know the true increase in mortality per hour delay in antibiotics, assuming this phenomenon is real (1).

0

Certainly we all agree that prompt treatment is important. However, blindly focusing on the speed of therapy will push treatment to be fast and sloppy. A balanced approach is needed which emphasizes both speed and accuracy.

0

Conclusions

0Time-to-intervention studies are correlative, and therefore hypothesis-generating only.Unfortunately they are widely misconstrued to imply causation.These correlations are subject to many confounders which often increase the apparent effectiveness of therapy. Confirmation bias leads us to accept the findings of these studies, because we already believe in the effectiveness of our therapies. In the past, time-to-antibiotic studies have driven health policy changes regarding pneumonia treatment which were subsequently shown to be harmful.Its doubtful that time-to-intervention studies add meaningful information to the scientific literature, but rather may merely be a source of noise.Unless of course they support my beliefs, in which case they are pure gold.

0

Notes

(1) In some infections, antibiotics release bacterial toxins thereby worsening systemic inflammation.One could imagine that delaying antibiotic therapy for a brief period of time to allow hemodynamic stabilization and possibly initiation of anti-inflammatory therapy could be beneficial (similar to administering steroids prior to antibiotics in bacterial meningitis).Thus it is possible that time-to-antibiotic studies in sepsis are misleading us and blinding us to potential therapies.

Share this:

9

Comment Here

Role

5Comment threads

4Thread replies

2Followers

Most reacted comment

Hottest comment thread

6Comment authors

Recent comment authors

Role

newestoldestmost voted

Guest

Anand Kumar, MD

Josh, re “implausibly enormous”…look at the animal data e.g. Kumar et al. JID 2006. from 10% to 80% mortality in a 3 hr window in septic mice following hypotension onset. Stats is good…so is a knowledge of physiology. And yes, your example is apt. that is why for ER cases, you have to excluse those hypotensive at presentation…you don’t know how long they’ve been hypotensive.

Vote Up1Vote Down Reply

2 years ago

Guest

Robert L Schmidt

Josh, very much like the idea you presented in your note. The endotoxin released by the bacteria in sepsis cases is the source of shock. If more is released as the result of use of antibiotics then perhaps the order of doing things DOES need to change.

Steroid first to strongly but unsustainably deal with the shock caused by the endotoxin. Followed by Vitamin C and Thiamine to sustainably deal with the shock caused by the endotoxin, as well as fortify the immune system. Then initiate antibiotics as indicated.

yep. We might eventually get there. The ADRENAL trial seems to justify early steroid for patients on any dose of vasopressor – not just as a salvage for mostly dead patients.

What's Your Job?

intensivist

Vote Up0Vote Down Reply

11 months ago

Guest

Shih-Chin Chou

Hi! The 2006 Kumar study sets Time Zero at recurrent/persistent hypotension.
Kumar’s Time Zero is after the patient receives an adequate fluid bolus, and on reassessment, is still hypotensive, then the clock starts. This cohort of patients have definite hypotensive septic shock.
Do other studies select this definite hypotensive septic shock cohort and this time zero?
Other studies seem to lump sepsis/severe sepsis/septic shock, or use ED time as the time zero, which are different patient populations.

As the Kumar study is still widely distributed by public sepsis campaigns.

Kumar 2006:
OBJECTIVE:
To determine the prevalence and impact on mortality of delays in initiation of effective antimicrobial therapy from initial onset of recurrent/persistent hypotension of septic shock.

I don’t think it matters, all of these studies are fatally flawed based on poor methodology.

What's Your Job?

intensivist

Vote Up0Vote Down Reply

7 months ago

Guest

Anonymous

The above represent just a few of the multitude of studies that mis-lead physicians and interfere with quality health-care. If you have at least 3 semester hours of statistics (research statistics, not just 'plug-in a formula' statistics) you would be able to see thru the 'noise' presented in so many articles and make quality decisions for your patients and workplace.

an excellent essay, too many of us do not grasp the logical aspects of EBM. May be it requires too much time to read and reflect on for doctors who like their knowledge in soundbite chunks on rounds or on Twitter?