You must be logged in to access this feature.

I take issue with Memtsoudis and Liu’s editorial acceptance1 of the study by Mashour et al.2 being referred to as comparative effectiveness research (CER), or the actual study being labeled as a practical clinical trial. It is neither. Although the study did indeed enroll a large, “real-world” patient population undergoing various procedures, the hospital setting was specific, and even more importantly, the interventions being compared could hardly represent what the Institute of Medicine expects of alternative treatment methods used in typical daily practice to be evaluated with CER.3 Put simply, the study results are not generalizable.

CER is the direct comparison of existing (that is, commonly used) interventions, aiming to determine which treatment works best for whom and under what circumstances.3 An important ingredient of CER is pragmatic (or practical), effectiveness trials that reflect everyday clinical practice.3–6 The explicit purpose of pragmatic trials is to inform decision makers about study interventions commonly used in practice and to measure clinically important outcomes in typical populations.4,7–11 The design and interpretation of large pragmatic trials has been a major interest of mine for more than 15 yr.9,10,12 Pragmatic trials test commonly used interventions in a wide variety of healthcare settings—that is, they should represent “real-world” practice.

The study by Mashour et al.2 occurred in a single university hospital system (albeit with three hospitals) in one city in the United States which used an electronic perioperative information system in all of its operating rooms. Furthermore, they used automated real-time analysis of bispectral index (BIS) values or minimum alveolar concentration every 5 min and provider-specific electronic alphanumeric paging alerts sent in under 1 min. This does not represent real-world practice in most of the western world (United States included). The most widely used approaches anesthesiologists use to avoid awareness include delivery of an appropriate concentration of hypnotic and other drugs, typically titrated to autonomic signs focusing on blood pressure and heart rate, and to look for patient movement—this is real-world monitoring of anesthetic depth. It is hard to see how the results of the study by Mashour et al.2 can be applied outside of their specific setting. In contrast, the B-Unaware trial at least tested two broadly accessible, competing interventions.13

Memtsoudis and Liu go on to describe the B-Aware trial,14 for which I was the principal investigator, as “a traditional randomized trial” despite it clearly being a pragmatic, effectiveness trial. Unlike the study by Mashour et al.,2 the B-Aware trial was conducted in a much broader range of hospital settings, involving 21 hospitals in 5 countries. The decision to use a high-risk population in the B-Aware trial should not be confused with the bias inherent in the strict selection criteria seen in some trials, which are intended to exclude high-risk patients. Enrolling an at-risk cohort does not infer a lack of generalizability, for it has been repeatedly shown that study results (relative effect) are nearly always in the same direction and of similar magnitude in unselected settings8,15; in fact, a high-risk study population may underestimate the treatment effect.16,17 The decision to use a high-risk group adds to study power because of the increased event rate and so it is more efficient; no more, no less. Finally, the view that we did not compare BIS with an alternative strategy to decrease the risk of intraoperative awareness is blatantly wrong. As stated above, the near-universal approach to avoiding intraoperative awareness—in other words, a relevant “competing intervention”3—is traditional monitoring of (predominantly) patient movement, hypnotic agent delivery, and autonomic signs. It is hardly a “no intervention” group. The B-Aware trial, therefore, was a pragmatic trial, and it remains highly relevant to inform CER.

Finally, CER is not only the generation of relevant evidence but also the synthesis of that evidence, with the latter best achieved using systematic review.3,6 I therefore provide results of broadly applicable CER evaluating the evidence for BIS monitoring in anesthesia, using an updated systematic review and meta-analysis of the pertinent randomized trials using Revman 5.1 software (Cochrane IMS team; fig. 1). This analysis includes input of the most robust data from the study by Mashour et al.,2 based on their as-treated data, because the presumptions of an intention-to-treat analysis cannot be claimed because of the gross failure of the intended intervention in 36% of the BIS group.

Fig 1.

Risk of awareness with bispectral index (BIS) monitoring combined with traditional monitoring versus traditional monitoring alone (“Control”), in which an end-tidal (ET) minimum alveolar concentration can be selected, with or without alarms being set to avoid low levels of volatile agent delivery during anesthesia. The BIS group can include patients undergoing total intravenous anesthesia. Both groups could include an electronic record system and immediate automated paging to alert the anesthesiologist, as used by Mashour et al.2 The “routine care” subgroup studies did not insist on the settings of ET alarms for a minimum alveolar concentration.

Risk of awareness with bispectral index (BIS) monitoring combined with traditional monitoring versus traditional monitoring alone (“Control”), in which an end-tidal (ET) minimum alveolar concentration can be selected, with or without alarms being set to avoid low levels of volatile agent delivery during anesthesia. The BIS group can include patients undergoing total intravenous anesthesia. Both groups could include an electronic record system and immediate automated paging to alert the anesthesiologist, as used by Mashour et al.2 The “routine care” subgroup studies did not insist on the settings of ET alarms for a minimum alveolar concentration.

There are six pertinent trials2,13,14,18–20 and one large observational study.21 The results of this CER approach depend on the local setting or scenario being considered:

The most clinically relevant question is whether the inclusion of BIS monitoring reduces the risk of awareness compared with traditional monitoring that usually includes end-tidal agent monitoring—for which the pooled data show that this significantly reduces the risk of awareness by 79% (95% CI: 21–95%), P = 0.02. The number needed to treat in a high-risk setting (incidence 1%) is about 130, and the number needed to treat in a low-risk setting (incidence 0.1%) is about 1,300. These conclusions are consistent with a large observational study in everyday practice21 and the post hoc findings from Mashour et al.,2 adding further weight to this finding.

If considering the question of whether BIS monitoring provides additional benefit compared with any type of “traditional” monitoring that usually includes end-tidal volatile agent monitoring, and could include having the agent monitoring alarms activated, it is unclear whether BIS monitoring reduces the risk of awareness, pooled risk reduction 39% (95% CI: −16 to 68%), P = 0.13. But, in my view, this is an ill-focused and unhelpful clinical question because the non-BIS options of monitoring (that is, the alternative treatment options) will be known to the decision-maker at the time. The inconsistency statistic, I2, highlights the heterogeneity when pooling disparate studies.

When comparing end-tidal agent monitoring with the alarms activated to avoid delivery of less than 0.7 minimum alveolar concentration,13,20 it is unclear whether BIS monitoring modifies the risk of awareness, pooled risk increase 225% (95% CI: −31 to 731%), P = 0.18. The range of possible risk reduction afforded by BIS monitoring extends from a 31% risk reduction to a 731% risk increase. This nonsignificant finding might lead decision-makers to regard the two monitoring options as equivalent, but such a conclusion is fraught with error.22

In settings where an electronic record system incorporating automated alerts is used, it is unclear whether the addition of BIS monitoring reduces the risk of awareness compared with end-tidal agent monitoring, pooled risk reduction 58% (95% CI: −151 to 88%), P = 0.18. The range of possible risk reduction afforded by BIS monitoring extends from an 88% risk reduction to a 51% risk increase.

These up-to-date findings can be used by all anesthesia providers to determine whether or not to rely only on traditional monitoring, avoid total intravenous anesthesia because of the inability to use end-tidal volatile agent monitoring, with or without activation of the volatile agent monitoring alarms, use BIS monitoring, or purchase/configure an electronic monitoring system with automated paging alerts. Scenario (a) would be most relevant to most anesthesia providers in the western world.

Another important component of CER is to consider efficiency: is the problem clinically important and is the intervention worthwhile and cost-effective?3,6 Regard for patient values and expectations, and consideration of benefits and harms, acquisition costs, and the number needed to treat should assist in such clinical and policy decision-making.

There have been dozens of high-quality large clinical trials guided by principles of CER in perioperative medicine over the past decade, including in the field of anesthesiology. The shame is that these are often overlooked or otherwise not incorporated in clinical practice more quickly. That is what our patients expect.

Risk of awareness with bispectral index (BIS) monitoring combined with traditional monitoring versus traditional monitoring alone (“Control”), in which an end-tidal (ET) minimum alveolar concentration can be selected, with or without alarms being set to avoid low levels of volatile agent delivery during anesthesia. The BIS group can include patients undergoing total intravenous anesthesia. Both groups could include an electronic record system and immediate automated paging to alert the anesthesiologist, as used by Mashour et al.2 The “routine care” subgroup studies did not insist on the settings of ET alarms for a minimum alveolar concentration.

Risk of awareness with bispectral index (BIS) monitoring combined with traditional monitoring versus traditional monitoring alone (“Control”), in which an end-tidal (ET) minimum alveolar concentration can be selected, with or without alarms being set to avoid low levels of volatile agent delivery during anesthesia. The BIS group can include patients undergoing total intravenous anesthesia. Both groups could include an electronic record system and immediate automated paging to alert the anesthesiologist, as used by Mashour et al.2 The “routine care” subgroup studies did not insist on the settings of ET alarms for a minimum alveolar concentration.