From Brown University, Providence, Rhode Island; Mount Holyoke College, South Hadley, Massachusetts; Ohio State University College of Health, Columbus, Ohio; Albert Einstein College of Medicine, New York, New York; and National Institutes of Health, Bethesda, Maryland.

John Rapoport, PhD

From Brown University, Providence, Rhode Island; Mount Holyoke College, South Hadley, Massachusetts; Ohio State University College of Health, Columbus, Ohio; Albert Einstein College of Medicine, New York, New York; and National Institutes of Health, Bethesda, Maryland.

Stanley Lemeshow, PhD

From Brown University, Providence, Rhode Island; Mount Holyoke College, South Hadley, Massachusetts; Ohio State University College of Health, Columbus, Ohio; Albert Einstein College of Medicine, New York, New York; and National Institutes of Health, Bethesda, Maryland.

Donald B. Chalfin, MD, MS

From Brown University, Providence, Rhode Island; Mount Holyoke College, South Hadley, Massachusetts; Ohio State University College of Health, Columbus, Ohio; Albert Einstein College of Medicine, New York, New York; and National Institutes of Health, Bethesda, Maryland.

Gary Phillips, MAS

From Brown University, Providence, Rhode Island; Mount Holyoke College, South Hadley, Massachusetts; Ohio State University College of Health, Columbus, Ohio; Albert Einstein College of Medicine, New York, New York; and National Institutes of Health, Bethesda, Maryland.

Marion Danis, MD

From Brown University, Providence, Rhode Island; Mount Holyoke College, South Hadley, Massachusetts; Ohio State University College of Health, Columbus, Ohio; Albert Einstein College of Medicine, New York, New York; and National Institutes of Health, Bethesda, Maryland.

From Brown University, Providence, Rhode Island; Mount Holyoke College, South Hadley, Massachusetts; Ohio State University College of Health, Columbus, Ohio; Albert Einstein College of Medicine, New York, New York; and National Institutes of Health, Bethesda, Maryland.

Disclaimer: The opinions expressed in this paper are those of the authors and do not reflect policies of the National Institutes of Health or the U.S. Department of Health and Human Services.

Through use of a random-effects logistic regression, investigators compared hospital mortality between patients cared for entirely by critical care physicians and patients cared for entirely by non–critical care physicians. An expanded Simplified Acute Physiology Score was used to adjust for severity of illness, and a propensity score was used to adjust for differences in the probability of selective referral of patients to critical care physicians.

Results:

Patients who received critical care management (CCM) were generally sicker, received more procedures, and had higher hospital mortality rates than those who did not receive CCM. After adjustment for severity of illness and propensity score, hospital mortality rates were higher for patients who received CCM than for those who did not. The difference in adjusted hospital mortality rates was less for patients who were sicker and who were predicted by propensity score to receive CCM.

Limitation:

Residual confounders for illness severity and selection biases for CCM might exist that were inadequately assessed or recognized.

Conclusion:

In a large sample of ICU patients in the United States, the odds of hospital mortality were higher for patients managed by critical care physicians than those who were not. Additional studies are needed to further evaluate these results and clarify the mechanisms by which they might occur.

Editors’ Notes

Context

Contribution

This study described 101 832 patients in 123 intensive care units in the United States. Patients managed by critical care physicians were sicker, had more procedures, and had higher hospital mortality rates than those managed by other physicians. Analyses that adjusted for severity of illness and the tendency for sicker patients to be managed by critical care specialists still showed higher mortality among patients managed by the specialists.

Caution

The extent of involvement and supervision by critical care physicians varies somewhat in U.S. intensive care units (ICUs) (1–6). Some ICUs are organized as strictly closed services, in which critical care physicians, or intensivists, assume control and decision-making ability over all aspects of patient care, whereas in some “hybrid” ICUs, mandated consultation and management by critical care physicians is the primary administrative model. Most ICUs, however, are structured as completely open units, in which the admitting physicians retain full clinical and decisional responsibility and thus have the option to care for their patients with or without input from critical care physicians.

Evidence from several settings suggests improved outcomes when critical care physicians assume substantial responsibility over the care and triage of ICU patients (1, 7–22). These studies, however, have methodological limitations and limited generalizability. Most are small, use historical controls or before–after study designs, and are limited to specific ICUs (for example, medical or surgical) in 1 or 2 centers. They have the usual risks for confounding by illness severity commonly seen in cross-sectional studies (7, 8, 14–21) and retrospective analyses of administrative databases that were limited to certain diagnostic categories (12, 13).

Recognizing the limitations of previously published studies and considerable variability in critical care management (CCM) in the United States, we examined data from 123 ICUs across the United States to assess the relationship between management by critical care physicians and hospital mortality rates of critically ill patients. These data were derived from a large national project that examined resource use in intensive care (2). At the beginning of our analysis, we hypothesized that CCM would be associated with improved outcomes in critically ill patients.

Methods

Patients

Patients were identified through Project IMPACT (Cerner, Bel Air, Maryland), a national database of ICU patients. The Project IMPACT database is a large administrative database originally developed by the Society of Critical Care Medicine in 1996. Participation is voluntary. All data are collected at each institution by on-site data collectors who are certified in advance by Project IMPACT to assure standardization and uniformity in data definitions and database definitions and entry. The database for 2000 to 2004 included 142 392 patients admitted to 123 ICUs in 100 U.S. hospitals. We excluded patients with missing data for variables of interest from our analysis, leaving 111 907 patients. We included only the first ICU admission, reducing the number of patients to 106 623, and then excluded patients who were managed only part time during their ICU stay, reducing the total observations to 101 832.

Variables

Our primary outcome variable was hospital mortality. Our key exposure or “risk factor” was the same regardless of whether a patient was managed by a critical care physician during his or her ICU stay. This was ascertained in Project IMPACT by using the survey question, “Was the patient managed by a critical care physician/team?” Trained data entry personnel for Project IMPACT define CCM as treatment occurring when the physician is asked to take responsibility for the overall management of a patient in the critical care unit without having to first provide expertise about a single organ system. A physician should meet 1 or more of the following criteria to be considered a critical care physician: 1) be recognized by the institution as a critical care specialist within a specialty unit, even without a specialty board certification (such as burn or neurointensivist), and must treat the total patient and not a single organ system; 2) have passed critical care medicine board examinations or be qualified to take the examination; and 3) be trained in an accredited critical care fellowship.

When a patient received CCM, it was documented, regardless of whether the treatment was for all or part of the ICU stay. Covariates included patient characteristics, such as demographic characteristics, diagnosis, and clinical condition at ICU admission. We also controlled for ICU and hospital characteristics. Severity of illness was measured by the Simplified Acute Physiology Score (SAPS) II. Through use of recently published work on SAPS (23), we added additional variables to SAPS II and modified coefficients in the logit model to derive a better fit. These included the patient's age (<40 years, 40 to 59 years, 60 to 69 years, 70 to 79 years, and >79 years), sex, duration of hospital stay before ICU admission (<24 hours, 1 day, 2 days, 3 to 9 days, >9 days), patient's location before ICU (transfer from outside emergency department, rehabilitation or skilled nursing facility, wards, or another hospital), clinical category (medical patient or other), and intoxication (yes or no). For this expanded SAPS II, the Hosmer–Lemeshow goodness-of-fit P value was 0.38. (The Appendix provides more detail on the expanded SAPS II.)

Statistical Analysis

We divided ICUs into 3 groups based on the percentage of patients receiving CCM for the entire stay: 95% of patients or more, 5% to 95% of patients, and 5% of patients or fewer.

We excluded 4793 patients who received CCM for only part of the ICU stay from the analysis, leaving 2 patient management types: CCM for the entire stay and no CCM. For each of the 6 categories defined by the combination of patient management type and ICU group, we computed expected and actual mortality rates. Expected mortality was the mean SAPS II probability of mortality. Actual mortality was the percentage of patients who did not survive the hospital stay. We computed the standardized mortality ratio and its 95% CI, based on an exact Poisson distribution, as the ratio of actual to expected mortality.

We developed a score to measure the propensity that a patient would be selected for CCM. We derived our score from a logistic regression model, with CCM as the dependent variable. The model was estimated on patients only from ICUs not mandating CCM. We screened all available patient characteristics known at the time of ICU admission and ICU characteristics for inclusion in the model. A propensity score was then estimated for each patient. Variables used to create the propensity score were age, Glasgow Coma Score, number of licensed hospital beds, insurance (commercial, Medicaid or Medicare, or self-pay), ventilation at ICU admission, tracheostomy at ICU admission, gastrointestinal bleeding, noninvasive ventilation at ICU admission, cerebrovascular event, chronic immunosuppression, chronic respiratory disease, acute renal failure, hospital location (rural, suburban, or urban), continuous sedation, and admission source (emergency department, another hospital, invasive procedures, or other non-ICU location). Figure 1 shows the proportion of patients managed by critical care physicians. Hospital mortality rates tend to increase from the first decile to the last decile of propensity and SAPS II. More details of the score and the sensitivity of results to changes in the propensity score are shown in the Appendix.

Figure 1.

Critical care management (CCM) and mortality.

SAPS = Simplified Acute Physiology Score.

We performed random-effects logistic regressions on the entire sample, using hospital death as the dependent variable. This method uses the within- and between-ICU variability inherent in the nesting of the patients into 123 ICUs. The crude model included only the risk factor “CCM for the entire stay” versus no CCM. Severity of illness (as measured by the expanded SAPS II score) and likelihood of selection for CCM (as measured by the propensity score) were then added to the model as control variables, along with all interactions of the control variables and risk factor. Where a statistically significant interaction term indicated that a control variable was an effect modifier, the regression was estimated within each quartile of the control variable.

We repeated random-effects logistic regression analysis of mortality on several subsamples. The “no-choice” subsample included 2 groups of patients: those from ICUs in which 95% or more or 5% or fewer patients received CCM. In addition, the following subsamples were examined: patients not transferred from another hospital, patients with a respiratory diagnosis with ventilator support at ICU admission, patients with respiratory diagnosis without ventilator support at ICU admission, patients with ventilator support at ICU admission, patients with a diagnosis other than respiratory and no ventilator at ICU admission, patients with a circulatory diagnosis, patients with a diagnosis of infection, patients with at least 1 ICU procedure, and patients with no ICU procedures. The Appendix presents additional details of regression analyses.

Role of the Funding Source

Eli Lilly and the Department of Bioethics at the National Institutes of Health Clinical Center funded the study. The funding services had no role in the design, conduct, and analysis of the study and did not participate in the decision to submit the manuscript for publication.

Results

Table 1 shows that ICUs that manage 95% or more of their patients with critical care physicians for the entire stay were, on average, larger and in larger hospitals than other ICUs. A greater percentage of ICUs had academic affiliation or activity and were only medical, surgical, or trauma, as opposed to a mixed model. A smaller percentage had staffing policies that permitted either licensed practical nurses or registered nurses.

Table 2 shows patient characteristics by ICU category and CCM status. Among the 123 ICUs, 23 (18 618 patients) had at least 95% of patients managed for the entire stay by critical care physicians, whereas 21 (22 870 patients) had 5% or fewer managed by critical care physicians. These 2 groups together make up the “no-choice” group. The remaining 60 344 patients were treated in the 79 ICUs in which 5% to 95% of patients received CCM for the entire stay (the “choice” group).

Table 2. Patient Characteristics

Table 2. Patient Characteristics

Comparison of patients managed for the entire stay by critical care physicians versus those managed by other physicians shows that more patients treated by critical care physicians received interventions, such as ICU procedures, intravenous drugs, mechanical ventilation, and continuous sedation. They were less likely to be postoperative patients or to receive surgery while in the ICU. Respiratory system disease, infections, and trauma occurred more among patients treated by critical care physicians. Intensive care units managing 95% or more of patients with critical care physicians had somewhat more admissions from other hospitals and from invasive procedures and somewhat fewer admissions from the emergency department than other ICUs did.

Table 3 provides the discharge destination according to CCM status of patients who survived their hospital stay. More than 59% of patients who did not receive CCM were discharged home (including those with home health care and those leaving against medical advice) compared with 58% of patients who received CCM. About 16% of patients were discharged to an extended care facility for CCM and no CCM, whereas 13% and 9% were discharged to a rehabilitation center for CCM and no CCM, respectively. More than 13% of patients who did not receive CCM were discharged to an unknown location compared with about 9% of those who received CCM.

Table 3. Hospital Discharge Destination

Table 3. Hospital Discharge Destination

Table 4 shows that patients managed by critical care physicians for the entire stay had a higher mean severity of illness (SAPS II probability of mortality) than patients who did not receive CCM. These patients also had higher hospital mortality. The standardized mortality ratio for patients who received CCM in ICUs that managed 95% or more patients was 1.09 (95% CI, 1.05 to 1.13) compared with a standardized mortality ratio of 0.91 (CI, 0.88 to 0.94) for patients who did not receive CCM in ICUs in which critical care physicians managed 5% or fewer patients. Among patients who received CCM in ICUs that managed 5% to 95% of patients, the standardized mortality ratio was 1.09 (CI, 1.05 to 1.12) for patients who received CCM for the entire stay compared with 0.91 (CI, 0.88 to 0.94) for patients who did not receive CCM.

Table 4. Expected and Actual Hospital Mortality

Table 4. Expected and Actual Hospital Mortality

A random-effects logistic regression model including only CCM as a predictor of hospital mortality produced a crude odds ratio (OR) of 2.13 (P < 0.001). The addition of SAPS II to this model reduced this OR to 1.42 (P < 0.001). Further inclusion of the propensity score decreased the OR to 1.40 (P < 0.001). For additional regression results, see the Appendix.

Interaction terms were statistically significant, indicating that severity and propensity were acting as effect modifiers. Models were estimated for each quartile of severity and propensity score (Table 5). For 11 of 16 resulting groups, the OR for mortality was statistically significant (P < 0.05). All statistically significant ORs were greater than 1.0, ranging from 2.83 (severity quartile 1 and propensity quartile 1) to 1.18 (severity quartile 4 and propensity quartile 4). Within each severity quartile, ORs tended to decrease as propensity quartiles increased.

Table 6 shows results of subgroup analysis through use of a random-effects logistic regression. When interaction variables were not significant, the ORs reported for CCM are from a model adjusted for SAPS II and propensity score. When 1 or both of these are effect modifiers, we report results by quartiles of the relevant variables. All of the ORs reported for the subgroup analyses are greater than 1.0, with 4 of 22 not significantly greater than 1.0. These analyses are a respiratory diagnosis of patients with no ventilator in place when admitted to the ICU (OR, 1.11; P = 0.121) and the first 2 quartiles of no ICU procedures (ORs, 2.11 and 1.53; P = 0.129 and 0.101).

We conducted 2 sensitivity analyses to determine whether transferring patients to another location (for example, a new hospital, rehabilitation center, hospice care, or extended care) determined the reduced mortality rate seen in the group that did not receive CCM (Table 7). In the first case, the operational definition of mortality included in-hospital mortality, transfer to another hospital, a rehabilitation center, extended care, hospice, or a long-term acute care facility versus home. Patients whose discharge destination was unknown were omitted from the sensitivity analysis. The second sensitivity analysis included only in-hospital mortality versus discharge to home. The crude OR, the OR adjusted for expanded SAPS II, and the OR adjusted for both expanded SAPS II and propensity for patients to receive CCM are similar. The ORs are greater for the group that received CCM than the group that did not for both sensitivity analyses, demonstrating the robustness of our results. The Appendix shows additional sensitivity analyses involving changes to the propensity score. Conditional logistic regression analyses for the 19 largest ICUs generated an OR greater than 1.0 in 18 of 19 ICUs. In 50% of these ICUs, the difference was statistically significant. In the remaining ICUs, the difference was not statistically significant because of the small sample size within the individual ICUs.

Table 7. Sensitivity Analysis on the Robustness of the Results to Changes in Mortality Definition

Table 7. Sensitivity Analysis on the Robustness of the Results to Changes in Mortality Definition

Discussion

By using a database of more than 100 000 patients, we identified 3 types of ICUs: ICUs in which all patients are required to receive management by critical care physicians, ICUs in which no patients are managed by critical care physicians, and ICUs in which patients may or may not be managed by critical care physicians. Despite adjustment for severity of illness, we cannot demonstrate any survival benefit with management by critical care physicians. In fact, patients managed by critical care physicians had higher odds of mortality than patients managed by physicians not trained in critical care medicine.

Our results are surprising and completely contrary to previously published findings (7–21). Almost all published studies on the impact of critical care physicians have demonstrated decreased morbidity or mortality with management by critical care specialists (24–28).

To control for potential confounders by severity of illness and the tendency for sicker patients to be transferred to physicians trained in critical care, we used an expanded SAPS II (23) and developed a propensity score. The expanded SAPS II was designed to better estimate the probability of mortality of patients admitted to ICUs than was possible with the older SAPS II system. To explore the possibility that some subgroups of patients might benefit from CCM more than others, we conducted several subgroup analyses. For almost all of the subgroups analyzed, risk for mortality associated with management by critical care physicians statistically significant increased.

What could account for these unexpected results? Several possible explanations must be considered. First, there may be residual confounders of severity not covered by either the expanded SAPS II or the propensity score. Our data indicate that patients cared for by pulmonary or critical care physicians for their entire ICU stay were sicker, as evidenced by higher median SAPS II scores. Our results are based on the ability to adjust the increased severity in patients managed by pulmonary or critical care physicians. Despite our attempts to adjust for severity to match patients in both groups for the purposes of comparison, no severity adjustment is perfect, and thus, there may be substantial unrecognized markers of severity in patients cared for by critical care physicians that remain unaccounted for. Some examples of residual unrecognized confounding include comorbid conditions and additional diagnoses not reported in the Project IMPACT database; responses to therapy; presence of protocols in some ICUs; presence and responsibilities of nonintensivist physicians, nurses, and other clinicians; and the influence of where and how long the patient received treatment before ICU admission (lead-time bias).

Second, we must consider the possibility that, for the patients in the Project IMPACT database, management by critical care physicians was associated with worse outcomes. Despite compelling evidence in the literature that care provided by trained critical care physicians leads to better outcomes, our data raise an important point: Although we believe that critical care physicians are trained and expertly skilled in the management of critically ill patients, perhaps some routine critical care practices and procedures may not be beneficial or cumulative use of more interventions may take a negative toll. Although further analyses and studies are needed to understand the possibility that care from critical care physicians is associated with higher hospital mortality, we speculate that there may be several plausible explanations. First, critical care physicians may use their own judgment to manage patients instead of using standardized protocols that may be associated with better outcomes. Second, because of their familiarity and expertise with procedures, they may use more procedures that subsequently lead to more complications. Their use of more procedures, such as placement of catheters and other invasive devices, may make critically ill patients more susceptible to life-threatening infections. Third, patients who receive care from a critical care physician may be transferred to different, unfamiliar physicians, whereas patients who receive care from non–critical care physicians may be more likely to receive ongoing care from physicians already familiar with them. Transfers may, be associated with greater chances of disruption in management and medical orders and create a greater likelihood of miscommunication and errors, all of which can have adverse consequences. This last possible explanation would be more noticeable in patients whose illnesses require less critical care expertise.

We do not claim that this list is exhaustive, but each speculation could be explored by future studies that examine the rates of protocol use, procedures, drug-resistant infections, and care for large groups of patients among physicians who are trained in critical care and those who are not.

Our study has several limitations. First, hospital mortality, rather than 30-day mortality, is the end point. Project IMPACT measures only ICU and hospital mortality. No information on the patients was collected after they left the hospital. Thus, the database contains no information on 30-day mortality. This allows for the possibility that the outcome between the 2 groups may be different at 30 days compared with hospital discharge. If more patients managed by non–critical care physicians died between hospital discharge and 30 days, our results might be very different. For this to be the case, non–critical care physicians would have to routinely discharge patients when they are sicker and at higher risk for death. The fact that more patients were discharged home by non–critical care physicians, rather than to extended care facilities, would seem to argue against this possibility.

Second, the process for identifying the management of patients has limitations. Data collectors at each institution decided, on the basis of training and instructions from Project IMPACT staff, whether to classify patients as managed by critical care physicians. Ultimately, this is a subjective process and may have led to unrecognized bias in the classification of patients.

Third, data elements for analysis are limited to those available in the Project IMPACT database. Limited information is available about the internal structure of each ICU in the database. For example, the presence of protocols, order sets, the length of experience of the nursing staff, the nurse–patient ratio on any particular day, and how many different groups of critical care physicians function within each ICU remain unknown. These and other factors may have had a strong, unrecognized influence on the outcomes of patients in a given ICU. In addition, the Project IMPACT database was not established to address the impact of critical care physician management on patient outcome.

Finally, the percentage of patients managed by full-time intensivists cannot be identified in the Project IMPACT database, and we therefore cannot assess the benefit of full-time, on-site management by ICU physicians. Treatment designated as “management entire stay by critical care physicians” includes all models of management in the ICU by board-certified or board-eligible critical care physicians, including full-time intensivists, office-based pulmonary critical care physicians seeing patients on rounds in the ICU once or twice a day, and private consulting groups with responsibility for critical care patients. Therefore, our study does not identify 1 particular model of critical care practice but rather a broad array of practice management styles provided by trained, board-certified or board-eligible critical care physicians. In the Project IMPACT database, we know little about the non–critical care physicians who manage patients in the ICU or the ICUs in which no patients are managed by critical care.

Future prospective studies should be designed to better answer the questions raised by our study, including characteristics that identify high-performing critical care units.

In conclusion, our study, which to our knowledge is based on the largest cohort ever analyzed to examine the relationship of CCM to survival of critically ill patients, found some unexpected results. Patients managed by critical care physicians for the entire ICU stay had a higher risk for death than patients managed by non–critical care physicians. Although all of the possible explanatory mechanisms we have mentioned may seem to portend badly for the practice of critical care medicine, we suggest that, if true, they are amenable to correction or mitigation through such efforts as guideline development and adherence, quality improvement, and systematic efforts to reduce errors. Given the complexity of critical illness, the need for dedicated critical care physicians seems inevitable, and strategies to assure best practices will help them to guarantee the best outcomes possible. Further research is needed to explain these findings and determine whether these results may be explained by unrecognized residual confounders of illness severity.

Committee on Manpower for Pulmonary and Critical Care Societies (COMPACCS)

Caring for the critically ill patient. Current and projected workforce requirements for care of the critically ill and patients with pulmonary disease: can we meet the requirements of an aging population?

Table 7. Sensitivity Analysis on the Robustness of the Results to Changes in Mortality Definition

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

This feature is available only to Registered Users

21 Comments

Marya D Zilberberg

University of Massachusetts, Amherst

June 6, 2008

Not time to leap

In their study Levy et al. have reported that, based on the data from Project IMPACT, patients managed by intensivists have a higher risk of hospital death than those managed by a non-critical care specialist [1]. While the authors tried earnestly to adjust away some of the confounding and heterogeneity by applying adjustments for propensity scores and other variables, they have rightfully acknowledged the possibility of residual confounding. However, there are other limitations, some acknowledged and others not, that impair our interpretation of the results.

First, the high number of excluded cases due to missing data predisposes the study to a selection bias. Second, it is our understanding that the authors were unable to ascertain which individual patients were cared for by an intensivist, but rather were forced to divide patients into groups based on the likelihood of being cared for by one. This classification was quite imprecise, resulting in over 50% of patients in each group being lumped into the 5%-95% group, an admittedly vast range of probabilities, predisposing the study to an ecological fallacy. Third, the authors did not stratify their results by "do not resuscitate" status, a factor likely to produce effect heterogeneity. This is an important consideration, since intensivists may be more attentive to end-of-life care practices than non-critical care doctors. Fourth, the endpoint of hospital mortality was not adjusted for exposure time (i.e., length of stay), and thus leaves open the possibility that this outcome was influenced by differing discharge practices between the groups of physicians compared [2].

The current data are at odds with much literature that generally relates outcomes of care to volume, and specifically demonstrates the value of being cared for by an ICU specialist in the setting of critical illness [3]. Because of the vast limitations of the data and the analyses, no firm conclusions can or should be drawn and no "leaps" towards policy change taken based on these findings.

Conflict of Interest:

None declared

Kenneth J. Gorelick

Enzybiotics, Inc.

June 6, 2008

Who follows guidelines?

Levy et al suggest that one reason for their unexpected finding of increased mortality in critically ill patients treated by intensivists rather than non- intensivists may be that "... critical care physicians may use their own judgment to manage patients instead of using standardized protocols that may be associated with better outcomes." This statement exposes the authors' bias rather than sheds light on their observation.

When a patient is treated by his or her own physician, that doctor is more likely to customize treatment, based on a deep understanding of the individual, than is a shift-working intensivist who has never laid eyes on the patient.

I was unable to find a report of a trial where patients were randomized to care by their own doctors using medical judgment vs. care by intensivists whose treatment was dictated by guidelines. As Sherlock Holmes said, "It is a capital mistake to theorize in advance of the facts." (Doyle, AC. Adventure of the Second Stain)

We read the paper by Levy and colleagues with much interest.[1] We believe that the Project Impact (PI) database was not designed (and is not capable) of answering the question posed and as such it is most likely that the results are erroneous The PI database demonstrates that sicker patients are more likely to be managed in larger academic hospitals by critical care physicians and are more likely to die (because they are sicker!). It is likely that the statistical manipulation of the data could not correct for this association. The critical reader should be extremely suspicious of any observational study using conglomerate propensity scores that assert associations, let alone causality with outcomes. The Connor's paper, which suggested that pulmonary artery catheters were associated with an increased mortality, provides a key example of this point.[2] The study by Levy and colleagues suggest that critical care physicians are harmful, however, an analysis of their data would equally suggest that academic hospitals as well as hospitals with primary fellowship training programs are harmful.

Current evidence suggests that the organizational structure of an intensive care unit (ICU) has an enormous impact on the quality of care delivered and patient outcome.[3,4] Open units are those in which admission of patients to the ICU is uncontrolled and management of the patients is at the discretion of each attending physician. Admissions are based on a first-come, first-served basis. As the attending of record, usually not a critical care physician, frequently does not have the time nor skills to provide comprehensive critical care he/she "portions off" the patients' care to a number of organ-specific sub-specialists. Whilst this may have some merit in a small number of complex patients, management by proxy or by "committee" frequently results in conflicting treatment strategies. Furthermore, both accountability and responsibility are also apportioned-off, with no physician assuming ultimate responsibility for the patients' care. Such a system is highly cost inefficient and not conducive to achieving optimal patient care. This is the typical organizational structure of most ICUs in the USA. It is likely that the 21 "no choice, non-intensivist" ICUs which participated in PI are not representative of the typical open ICU in the USA. The mere fact that these ICUs participated in PI suggest that they were well organized, evidence-based units with a desire to measure (and improve) their outcomes.

It is also important to recognize that not all "intensivists" (in the USA at least) are created equal. Whilst in Australia, New Zealand and many European counties critical care physicians are solely dedicated to the practice of critical care medicine, this is not so in the USA where many "intensivists" devote less than 50% of their clinical time to the management of patients in the ICU. In addition, in many academic medical centers, particularly Ivy League Institutions, research activities take the "intensivist" out of the ICU for much of their time. It is therefore not uncommon for many "intensivists" to spend as little as 2-4 weeks/year in the ICU. Clearly, unless your professional time is devoted to your specialty you cannot be good at what you do. The American College of Critical Care Medicine defines an "intensivist" as a clinician who has undergone specialty training in critical care medicine, is accredited by the appropriate governing agency and devotes greater than 50% of his/her professional time to the practice of critical care medicine.[5] It is therefore likely that (in the US at least) the type of "intensivist" may influence outcome. This observation is not restricted to critical care medicine. Surgeons and interventional cardiologists who have high volumes have better outcomes than those with low volumes.[6-8] Unfortunately, this study did not provide any information on "full time" intensivists providing "full-time" care.

We assert that the preponderance of evidence supports the concept that critically ill patients are best managed by dedicated critical care physicians who have undergone specialized multidisciplinary training which provides them with the necessary knowledge, skills and attitudes to achieve the best outcomes for critically ill patients. In critical care, it is the intensive attention at the bedside by dedicated physicians and nurses working together in a compassionate and caring environment that will achieve the best outcome for our patients.

Conflict of Interest:

None declared

Constantine Manthous

Yale Scool of Medicine

June 11, 2008

Critical Care, Science and Ideology

The study by Levy and colleagues (1) suggests that critically patients cared for by critical care physicians' (CCP) have a higher mortality rate than those cared for by non-specialists. This observation challenges the "science" (2,3) that has driven public policy for the past decade. Despite application of statistical methods to adjust for sicker patients in the treatment group, it is quite possible that residual confounding accounts for the observed result. While the growing momentum of the critical care movement (3) may entice us to this conclusion, Levy's findings deserve very careful consideration. The implications for public health are significant.

Let's assume the observed result is not due to error. The "key exposure was whether the patient was managed by a critical care physician/team." (1) Critical care outcomes research has suffered from varying, and sometimes arbitrary, definitions. The crux of the problem is the distinction between "intensivist" and critical care trained/certified physician. Notice that even the accompanying editorial, written by world leaders in critical care, conflates the two (4). An intensivist is a physician, usually with training and/or certification in critical care, who supervises or conducts, in person, care of critically ill patients. In 1997, we reported a 26% reduction in mortality (from excessive to acuity-appropriate) associated with adding an intensivist to our hospital (5). The intensivist supervised resident-trainees, often at the bedside during weekdays, 6 months/year, and actuated evidence-based protocols and educational programs. We stress the specificity of this intervention because it is not the same as others "“ so combining study results in this area may be perilous (2,3). Caregivers in the pre-intensivist period had Board certification, but were not intensivists. They all conducted simultaneously busy outpatient and wards practices. If this (and other) observations (2) were not spurious, it is not the training and certification that impacts outcomes. Rather it is the rapid bedside (and/or telephonic) response of a knowledgeable clinician that likely impacts outcome. It makes perfect sense that even the most knowledgeable critical care physician cannot resuscitate an extremely ill patient as well from the office as she could at the bedside. Patients so close to death cannot afford even small errors. "Being there," the independent variable at the foundation of intensive care, was not measured in Levy's study. Another potentially powerful confounder, not considered in statistical adjustments, was the role of trainees in the critical care specialist group. Table 1 suggests that patients receiving CCP care were far more likely to be at teaching hospitals (1). At most such hospitals, trainees provide initial care, much of the day, albeit with CCP guidance. Previous data suggest an association of trainee-provided-critical care and negative outcomes (6,7).

Finally, mortality and disposition are not the only meaningful outcome measures of critical care. To the extent that intensivists are best trained and situated to detect when ongoing care is unlikely to prolong life that is consistent with the values of the patient, and actuate palliative care when appropriate, facilitating death with dignity is an equally important goal of intensivists. Higher mortality is not necessarily a worse outcome.

We urge readers not to discard Levy's observations because they do not conform with current ideology (3). There are certainly lessons for day-to-day patient care and implications for past and future critical care scientific inquiry.

Conflict of Interest:

None declared

Hadi Hussain

Department Of internal medicine,combined military hospital,lahore,pakistan

June 11, 2008

Association between Critical Care Physician Management and Patient Mortality in the ICU

This article by Levy et al suggests , but does not confirm, that intensive care provided by critical care specialists does not improve outcomes and may be detrimental to patients.(1) The authors have examined data from Project Impact, including 101,832 patients from 123 ICUs in 100 US hospitals. The primary results show that patients cared for by critical care physicians were sicker than other patients and underwent more procedures. Notably, hospital mortality was higher for patients managed by critical care physicians, even after adjustment for severity of illness and other factors that may have influenced the probability of being cared for by a critical care physician. This study is observational in nature, making cause-and-effect conclusions impossible. However, the authors conducted elegant analyses to adjust for other factors that may have influenced survival differently between the critical care and non-critical care groups. Unfortunately, even the most powerful statistical analyses are incapable of dissecting and adjusting for all relevant factors that may influence the outcome of interest. Thus, while observational data such as these may suggest a relationship between intensivist-led patient care and outcomes, ascertaining the real relationship (i.e. cause and effect) requires the conduct of a prospective, randomized trial.

Conflict of Interest:

None declared

Stephen R Workman

QEII HSC

June 13, 2008

The trouble with normal

While selection bias clealry must be considered, in 1996 Connors et al performed a well designed observational study showing that Swan Ganz catheters, previously considered clearly beneficial, were not, and that their use may was harmful. (1) Denials of the study results were immediate and sutained but randomized trials have since resolved the dispute.(2)

Since that time other previously held and intuitively obvious beliefs about normalizing values in critically ill patients have been disproven (3,4).

It would be useful to know how often disproven or unproven therapies are utlized in the different groups. It seems possible that the difference in survival could arise because trained intensivists take greater efforts to normalize parameters in critically ill patients, efforts that could ultimately be causing harm.

Conflict of Interest:

None declared

William T. McGee

Baystate Medical Center & Tufts Medical School

June 13, 2008

In Response

To the Editor:

Both the accompanying editorial and the discussion of their findings provide multiple possible explanations to elucidate the association between intensivists' care and mortality. Other organizational aspects of the various ICU models and healthcare systems or hospitals studied may not be captured by Project IMPACT and can influence outcome(1).

Regardless, considering the experience of working solely in the Intensive Care Unit, although hospital mortality is an important outcome parameter for some, this is not true for all ICU patients. Taken to its logical extreme, 0% mortality by this analysis under any staffing model would represent the highest performing Intensive Care Unit. Although many would like to believe otherwise, for patients who are going to die, medical intervention prolongs suffering which may be greatest in the Intensive Care Unit(2,3), and adds cost to a healthcare system that is already floundering under the weight of 40 million uninsured Americans, many of whom cannot receive basic healthcare that would clearly improve the quality of their lives. Recognizing that patients can, and actually will, die at the end of their lives, and facilitating this process with a minimum of suffering on the part of both patients and families, while minimizing the cost to the healthcare system is far more important than discharge from the hospital alive. The prolonged suffering and costs incumbent upon some ICU survivors, many of whom never regain health, independent function, or return home could be mitigated by focusing on more important goals for patients and our healthcare system than hospital mortality(4).

Neonatal ICUs in the United States have at least vocalized concern around this issue(5). Perhaps it is time for adult intensivists to do the same.

Conflict of Interest:

I was struck by the very unusual major disease categories described under Patient Characteristics in Table 2 of the article.

For example, infections were present in 0-8.1% of the patients in this article. This seems very low.

On the other hand, 19.4-52.9% of patients had a circulatory system disorder, while 12-17.6% had a digestive system and 9.2-19.1% a nervous system disorder. These latter three disease categories were not often the reasons for ICU admission in the mixed medical-surgical ICUs where I worked in the last 6- 7 years.

Particularly the very high proportion of patients with circulatory system disorders and the low frequency of infections suggests to me that perhaps many of the ICUs in this article are in fact coronary care units, or that many of the patients who presented with septic shock, but had no obvious source of infection on admission to the ICU, were misclassified as having a "circulatory system disorder" rather than an infection as the main disease category.

Conflict of Interest:

I am a board-certified pulmonary/critical care physician who worked >6 years as an intensivist in mixed medical-surgical ICUs.

Patients admitted to the ICU are a heterogeneous group of patients, with a wide range of ages, underlying disorders and acute conditions. Given the diversity of this patient population, it is difficult to envision that a single treatment or exposure would show a consistent signal across an entire group of critically ill patients.

The findings of Levy and colleagues showing increased in-hospital mortality rates for those patients treated by critical care physicians compared with non-critical care physicians are therefore surprising(1). Even after adjusting for severity of illness and a propensity score for exposure to critical care, treatment by critical care physicians appears to be associated with increased in-hospital mortality(1).

While the authors made many efforts to collect potential confounders that might affect in-hospital mortality, given the heterogeneous patient population, it is difficult to imagine that one might be able to identify all of the treatment and patient related factors that might be associated with mortality. Of note, a previous attempt to control for severity of illness by using propensity scores in a large heterogeneous cohort of critically ill patients (2) led to results that were discordant with subsequent randomized clinical trials (3-5). Perhaps the most important role of this study by Levy and colleagues will be to stimulate further research into which practices will improve outcomes in critically ill patients.

Conflict of Interest:

Dr.Sevransky is a practicing intensivist

Petey Laohaburanakit

Rogue Valley Medical Center

June 22, 2008

Measure of an intensivist : Is mortality rate all that matters?

Dr. Levy et al published a commendable retrospective analysis in a recent issue of the Annals. Dr. Levy found that the hospital mortality rates were higher for patients who received critical care management (CCM). While the authors insightfully declared several limitations of the study, they did suggest a possibility that CCM might have been associated with poorer outcomes, in particular higher mortality rates. As an intensivist, I would like to interject an opinion on mortality in the intensive care unit (ICU).

Mortality rates have long been the key measures of medical interventions. Unfortunately, as it often happens in the ICU, mortality is an inevitable outcome. When a skilled intensivist realizes that death is imminent, change in direction of care to comfort measures is frequently recommended to the surrogate. This results in decrease ICU and hospital lengths of stay and usually improves the family's satisfaction.

It would have been more meaningful had Dr. Levy and his colleagues enlightened us with comparisons of certain outcome measures, e.g. proportion of patients whose direction of care was changed to comfort measures, ICU and hospital lengths of stay, and family satisfaction with end-of-life care. The value of the intensivist is not only focused on saving lives, but also ensuring that ICU resources are well utilized and our patients are well-cared at the end of their lives.

Respectfully, Petey Laohaburanakit, M.D., F.C.C.P.

Conflict of Interest:

None declared

Joris J. Arends

Leiden University Medical Center

June 23, 2008

The limitations of standard risk-adjustment methods.

To the editor:

The conclusion that management of critically ill patients by critical care physicians (CCM) is dangerous is premature. The authors state that the results are unexpected and completely contrary to previous findings. The best explanation of this result is the study's design: the study allows clustering of confounders of mortality which cannot be adjusted by any standard risk-adjustment method (1). Because the study uses few in- and exclusion criteria, not only the patient characteristics in the "˜no-choice' (CCM) and "˜choice' group (no CCM) differ significantly in important prognostic factors (e.g., artificial ventilation), but also the "˜treatment environment' differs (e.g., academic vs. other institutions ). The authors assume that inclusion of SAPS II and propensity-score in the random-effects logistic regression model suffices to correct for these lopsided influences on mortality, which are introduced by treatment choices that lead to severe allocation bias. However, mortality predictions based on scoring systems are known to fall short of a clinician's overall judgment (2). The estimation of prognosis of a severely ill patient and the decision to place this patient under CCM care, are influenced by more than the sum of variables in the SAPS II and propensity score alone. Thus, an excess of unrecognized influences on mortality is eventually attributed to CCM. To eliminate such treatment allocation bias, some mechanism that approaches randomization is needed (3). For example, Diringer et al., using an early version of the same database, demonstrated that admittance in neurologic/ neurosurgical intensive care units (ICU) improved outcomes in patients with intracerebral hemorrhage (4). By choosing a patient population whose urgency leads to admissions to both specialty and general ICUs, part of the usual allocation bias can be eliminated. In addition, the focus of the analysis by Diringer et al. was not the care giver, but the structure of care. In the study by Levy et al., the argument that subgroup analysis and expansions of the propensity score did not affect the findings of the primary analysis, does not prove robustness, but proves again the limitations of the risk-adjustment method to correct for intractable confounding due to treatment allocation bias (1).

Conflict of Interest:

None declared

Michael Blumhardt

Mercy Regional Medical Center, Durango, CO

June 23, 2008

Challenges Facing Critical Care

The June 2008 study in Annals by Levy and colleagues found an association between critical care management and increased mortality. This follows the recent publication of the Dartmouth Health Atlas 2008(1) and associated commentary in the New England Journal of Medicine.(2) The Dartmouth study reviewed the records of several million Medicare patients hospitalized during the last two years of life at nearly 3000 hospitals. Both the Levy and Dartmouth studies offer unfavorable appraisals of critical and end-of-life care as practiced today.

Citing wide disparities in end-of-life-care expenditures, the authors of the Dartmouth study argue such differences arise from a lack of unified scientific standards. For example, Medicare patients in their last two years of life at UCLA were cared for by an average of 16.9 physicians with an average cost of $93,842 dollars. This contrasts with similar patients at the Mayo Clinic who had 8.9 physicians with an average cost of $53,432. The authors of the Dartmouth study argue higher intensity end-of- life-care, much of which occurs in the ICU, has increased costs without clearly improving outcomes and that wide disparity in expenditures amongst institutions reflects a lack of evidenced based medicine rather than disease severity or patient preference. Similarly, the authors of the Levy study suggest that the increased mortality with critical care consultation reported in their study may stem from a lack of evidence based protocols and over-reliance on potentially risky procedures.

Like any research, these studies should not be accepted without careful and skeptical analysis. But regardless of their merits, the field of critical care is likely to remain a focus of policy makers and researchers when you consider that critical care expenditures are estimated at $55 billion annually.(3) Widespread application of evidence- based and cost-effective care is our best hope for preserving and enhancing the status of critical care medicine as a field, yet challenges clearly remain.(4-5)

Most ICU severity models, including SAPS II and MPM0-II, were derived from populations with higher mortality than Project Impact's. Without recalibration, these models exaggerate risk for low acuity patients. Thus, MPM0-III, developed from the same Project Impact database, required a novel "zero factor" term connoting negative risk to accommodate the inclusion of so many patients without discernible mortality risk[2]. Intensivists were less likely to manage low risk patients so any overestimation of risk for "zero factor" patients would favor non- intensivist management.

A second, crucial problem arises from an inability to distinguish patients improving with initial care from those not. Most patients were in units with "optional" intensivist management where transfer of deteriorating patients from non-intensivists to intensivists is common practice. SAPS II utilizes the worst physiologic derangements during the first 24 hours of ICU care, allowing patients worsening after ICU admission to have identical scores to those who improve rapidly. The problem is substantive: "intensivist managed" patients may have received up to 24 hours of initial non-intensivist directed ICU therapy, as Project Impact recorded dates but not times of intensivist management. The propensity score, determined by pre-ICU factors, is similarly blind to clinical trends. Since deterioration is both associated with mortality and motivates transfer to intensivists, the absence of data permitting adjustment for early ICU courses may be insurmountable.

This would not be the first study of decision-making for ICU patients where overlooking trend data led to suspect conclusions. Connors, et al. reported excess mortality with pulmonary artery catheterization[3], a finding unsupported by subsequent randomized trials[4,5]. As with the current study, adjustments ignored post-admission clinical trends, which doubtless impacted catheterization decisions. Though Levy et al. argue only large, undetected confounders could negate their findings, the attribution of mortality to pulmonary artery catheters missed confounders of such magnitude, in a similar population, under similar circumstances.

The distribution and SAPS II calibration for "zero factor" patients is addressable. Unfortunately, Project Impact does not contain the critical data needed to understand the larger issue of why intensivist management was chosen when optional. Its absence renders the study's conclusions doubtful.

Conflict of Interest:

None declared

Thomas L. Higgins

Baystate Medical Center/Tufts University School of Medicine

June 24, 2008

In Response

We were intrigued by the results of Levy and colleagues(1) demonstrating higher odds for hospital mortality in patients managed by critical care physicians. We recently employed the Project IMPACT database to update the Mortality Probability Model (MPM) . Although our sample of 124,885 patients at 98 hospitals between October 2001 and March 2004 is not identical, the overlap is substantial. We analyzed the variable: "Percentage of patient stay managed by critical care physician/team". In our sample, 68179 patients (54.6%) were coded as 0% critical care management (CCM), 50694 (40.6%) were coded as 100% CCM, and 6012 (4.6%) had a value between 0 and 100%. Mortality was universally higher in the CCM group at every decile of risk. However, the majority of patients (64% of non-CCM and 54% of CCM) fell into the lowest decile of MPM-III mortality risk where risk-adjusted outcome was exactly as expected (Standardized Mortality Rate (SMR) 1.0; 95% confidence interval 0.95-1.05 for both non-CCM and CCM managed patients). There were also no differences in SMRs in patients when the MPM0-III risk prediction was >50%. SMR differences occurred at the 2nd and 3rd deciles of MPM- predicted risk, exactly where the calibration curve for this model deviates from "perfect" prediction(2). Although our analysis utilizes MPM, we suspect similar calibration considerations may apply to SAPS used by Levy et al, and a calibration curve of their observed vs. predicted mortality, along with measures of model fit would be informative.

We agree with the authors that unmeasured confounders are likely. The pattern of raw mortality rates across severity-adjusted deciles of risk suggests case-mix differences between CCM and non-CCM units. The risks of emergency surgery are incompletely captured by risk adjustment models(3), and our Project IMPACT sample shows the proportion of emergency surgical patients to be 58% higher in CCM units. We suspect that other differences (urban vs. rural, teaching vs. non-teaching, alcohol and tobacco use, socioeconomic status) also play roles; more time is needed to evaluate these concerns.

As payment for medical care becomes linked to performance metrics, it is essential for these metrics to be accurate. Levy and colleagues have opened an important research question with their seemingly counter- intuitive results. More work is needed to find and correct for potential confounders before concluding that existing severity-of-illness models adequately correct for patient population differences in retrospective studies where assignment to CCM and non-CCM units is not random.

The paper by Levy et al looking at relations between critical care and patient mortality (1) has a fundamental methodological weakness. The method they followed was that they first collected the data, second, they noted as association between critical care physician management & increased patient mortality, thirdly, they conducted further detailed & exceedingly complicated analysis with statistical experts on the same data-set & they thereby "verified" this adverse association. Nobel prize winning physicist Richard Feynman has lucidly pointed out the fallacy of this practice. In his lecture "This Unscientific Age" (2) he states "I now turn to another kind of principle or idea, and that is that there is no sense in calculating the probabilÂ¬ity or the chance that something happens after it hapÂ¬pens. A lot of scientists don't even appreciate this. In fact, the first time I got into an argument over this was when I was a graduate student at Princeton, and there was a guy in the psychology department who was runÂ¬ning rat races. I mean, he has a T- shaped thing, and the rats go, and they go to the right, and the left, and so on..............................This man had designed an experiment which would show something which I do not remember, if the rats always went to the right, let's say, .....................And then he noticed, most remarkably, that they alternated, first right, then left, then right, then left. And then he ran to me, and he said, "Calculate the probability for me that they should alternate, so that I can see if it is less than one in twenty." I said, "It probaÂ¬bly is less than one in twenty, but it doesn't count." He said, "Why?" I said, "Because it doesn't make any sense to calculate after the event. You see, you found the pecuÂ¬liarity, and so you selected the peculiar case."

Levy et al, based on observed data, generated a hypothesis; There is an association between critical care physician management & increased patient mortality. On expert review with sophisticated statistics they essentially regenerated the same hypothesis. Instead of testing this hypothesis in another prospectively collected data-set, they interpreted the re-stated hypothesis as being a conclusion.

Considering the potential implications of this association between critical care physician management & increased patient mortality, I wonder if the editorial board was hastily inappropriate in accepting for publication instead of asking for prospective verification on a new data- set.

Conflict of Interest:

None declared

Christopher R. Dale

Skagit Valley Medical Center

June 26, 2008

Hospitalist effect?

To the editor:

I read with interest Levy et al's analysis of of the effects of critical care medicine in a recently-collected large retrospective database. (1) Much of the previous work in this area had been before the advent of the hospitalist movement and with before-and-after study designs. (2)

Is it possible that much of the previously observed difference in mortality seen with intensivist staffing might not be present in situations where critical care is provided by hospitalists who do so regularly?

An added on-the-job critical care certification for hospitalists would be one way to meet the upcoming shortfall of critical care physicians.

Conflict of Interest:

None declared

Bernard A FoÃ«x

Emergency Department, Manchester Royal Infirmary, Manchester, UK

July 9, 2008

Benjamin Franklin on ICU mortality

The study by Levy et al makes interesting reading(1). Hospital mortality may be higher for those managed by intensivists, but mortality in the community is still 100%. As Benjamin Franklin put it, "In this world nothing is certain but death and taxes".

Conflict of Interest:

None declared

Mitchell M Levy

Brown University/Rhode Island Hospital

July 28, 2008

The Authors Response

We agree with and appreciate several of the thoughtful comments offered by authors of these letters. Most of the comments by these authors have been mentioned in the discussion section in our manuscript. We would like to make some important points for clarification: 1) Recalibration was necessary for the severity model. For this reason, an expanded SAPS II was utilized for severity adjustment in our study. We believe that Figure 1, along with the reported value for the Hosmer-Lemeshow goodness of fit statistic, in our published manuscript demonstrates the usefulness of the expanded SAPS II for predicting actual mortality in the population used in our study. 2) Critical care-managed patients did not receive any care from non critical care physicians. The category "managed entire stay by critical care physicians" exclude any patients managed by non-critical care physicians in the first 24 hours. Because Project Impact records dates but not times of critical care physician management, and in order to eliminate potential confounding by this uncertainty, all patients managed partially by critical care physicians were excluded from analysis in the study. We agree that the ability to detect unrecognized confounding by severity is limited by the information contained in the Project Impact database. 3) As stated in our discussion, it is true that the Project Impact database does not allow for the separation of "full time intensivistt" management from management by board certified critical care physicians who are not necessary "on-site." Therefore, the "being there" factor was not evaluated in our study, nor was the impact or role of management by trainees. 4) Although the published literature suggest that full-time, on-site, intensivists are more likely to write for Do-Not-Resuscitate orders for critically ill patients, there was no difference in DNR orders amongst the groups evaluated in our study. 5) Patients in coronary care units were excluded from analysis in our study. 6) The study by Diringer et al. included patients from institutions outside of the Project Impact database and so does not have relevance to the current study. 7) The Project Impact database does not describe the "culture" of the ICUs included in the database. It is possible that the ICUs in which there is no management by critical care physicians are not representative of the typical open ICU in the United States. We welcome all the comments included in these letters as part of a new, ongoing dialogue aimed at improving the care provided to critically ill patients. Conflict of Interest:

None declared

Howard Belzberg

Los Angeles County+University of Southern California Medical Center

August 12, 2008

Assault on reason

It was with great enthusiasm that we turned our attention to the article in the recent Annals of Internal Medicine (Levy MM, Rapoport J, Lemeshow S, Chalfin DB, Phillips G, Danis M. Association between critical care physician management and patient mortality in the intensive care unit (Ann Intern Med. 2008 Jun 3;148(11):801-9.) regarding the association between critical care physician management and patient mortality in the ICU. We were immediately struck by the similarity to the methodology used in the Connors study (1) which condemned the use of the pulmonary artery catheter, and which is largely responsible for many practitioners reducing their use of this technique. If indeed we are to accept this "propensity score" as a legitimate tool for evaluating the worthiness of a therapeutic approach, then we must condemn with equal enthusiasm the use of critical care physicians in critical care medicine. This obviously flies in the face of reason, and requires that we rethink the validity of the use of propensity scores in medicine. (1) The effectiveness of right heart catheterization in the initial care of critically ill patients. SUPPORT Investigators.Connors AF Jr, Speroff T, Dawson NV, Thomas C, Harrell FE Jr, Wagner D, Desbiens N, Goldman L, Wu AW, Califf RM, Fulkerson WJ Jr, Vidaillet H, Broste S, Bellamy P, Lynn J, Knaus WA. JAMA. 1996 Sep 18;276(11):889-97. Conflict of Interest:

None declared

Andrew A. Quartin

University of Miami

August 26, 2008

In Response

We are concerned that the authors may have misconstrued the meaning of key data fields from the Project IMPACT database. Both their article and response letter indicate they used these fields to assign patients into the cohorts they compared. Far from being an arcane issue, such misclassification would prevent any meaningful interpretation of the data.

In our first Letter to the Editor, we pointed out that patients classified as "managed entire stay by critical care physicians" may have in fact received up to 24 hours of non-intensivist directed care in the ICU prior to critical care physician management, depending upon what time of day ICU admission took place. This is very important. When transfer to critical care physician management occurs some time after ICU admission in "open" units, it is often motivated by poor response to initial therapies administered in the ICU, a harbinger of increased mortality. Inadvertent inclusion of these patients in the "managed entire stay by critical care physicians" cohort would constitute an obvious bias favoring management by non-intensivists.

In their response posted electronically on July 28, 2008, the authors do not dispute that the inclusion of such patients would bias their analyses. Rather, they incorrectly claim "the category "˜managed entire stay by critical care physicians' exclude (sic) any patients managed by non-critical care physicians in the first 24 hours." Their manuscript and response letter are not explicit with regard to which data elements from Project IMPACT were used to accomplish this. However, as we explain below, the data fields alluded to by the authors do not permit such distinctions, nor are we aware of others that would.

We note (and the authors concur in their response) that Project IMPACT records the dates but not the times critical care physician management starts and stops. Apart from individual patient data, Project IMPACT also records certain policy information for each participating ICU. Project IMPACT classifies patients as having "Critical Care Management During the Entire ICU Stay" under two circumstances:

1. The ICU the patient was admitted to has a policy dictating that all patients receive critical care physician management throughout their ICU stays.

2. The starting and ending dates of critical care physician management recorded in the database coincided with the dates of ICU admission and discharge.

All patients that Levy, et al. classified as having been cared for in a "choice" unit and "managed entire stay by critical care physicians" must have met this second criterion, since patients meeting the first criterion were cared for in ICUs that should have been classified as "no-choice". Note that the second criterion is satisfied if critical care physician management starts on the same calendar day as ICU admission, even if there was a period of non-intensivist guided ICU therapy preceding critical care physician involvement on the ICU admission day.

Levy, et al. report that there were 23,324 patients "managed entire stay by critical care physicians" in "choice" units, over half the intensivist managed population. It is impossible to determine how many of these patients were transferred to critical care physician management after ICU admission but within the first calendar day, rather than admitted directly to the ICU under critical care physician management. However, as preferential transfer of difficult ICU patients to intensivists is common practice in many medical centers, we suspect this was not rare. In fact, the authors themselves excluded from analysis 17% of the patients managed by intensivists in "choice" units because of transfers that occurred after the first ICU day, which Project IMPACT does capture.

Accuracy in cohort assignment is very important for the analysis the authors conducted. Absolute differences between predicted and observed mortality rates in all cohorts were well under 2%. Shifting a few "sick" patients from one group to another can easily determine the outcome of analysis. Simulations have shown that performance relative to standardized mortality ratios, as the authors reported, are very sensitive to the addition or removal of just a few patients (e.g. Kahn, et al., Chest 2007; 131:68-75). (Moreover, the exclusion from analysis of patients who required transfer to critical care management after the first ICU day effectively removes high risk patients from the non-intensivist cohort, reinforcing this bias.)

Unfortunately, neither of the parameters used to adjust for confounding factors, a modified SAPS II score and a propensity score, can correct for misclassification. Both are insensitive to trends following ICU admission, the very factors often motivating transfer of management responsibility. SAPS II, based on worst physiologic values over the first 24 hours of intensive care, can yield identical scores for patients who are deteriorating, improving with initial therapy, or remaining static. For example, SAPS II assigns the same score for blood pressure derangement (five points) to both a patient arriving to the ICU with a systolic blood pressure of 85 mm Hg that than rises to 105 mm Hg after one liter of fluid, and a patient whose blood pressure falls from 105 mm Hg at admission to 85 mm Hg despite administration of several liters of fluid and institution of a norepinephrine infusion. Despite identical scores, the latter patient, with volume refractory shock, is both more likely to die and to be referred to a critical care physician. The propensity score was based entirely on factors apparent before ICU admission, and thus cannot capture or account for trends following ICU admission that might stimulate transfer to critical care physician management.

This paper has been a source of great interest and controversy, both within and beyond the Critical Care Medicine community. It is essential that the methodology employed be both appropriate and clearly delineated. If the authors used additional techniques to avoid inaccurate cohort assignment of the type we have described above, they have not indicated so in either their manuscript or in their initial response to our comments. Could the authors either more fully explain their methodology, or, alternatively, acknowledge a fundamental difficulty of their data?

Conflict of Interest:

None declared

Mitchell Levy

Brown University

September 26, 2008

In Response

In response to Dr. Quartin and colleagues' concerns regarding misclassification of cases in our paper, we offer the following comments and further analysis.

The Project IMPACT database includes the date and time of ICU admission and discharge, but only the date for starting and stopping CCM. Thus Dr. Quartin et al are correct that the PI database does not contain precise information about the initiation of critical care management during the first calendar day. This is a limitation of the study which often occurs when using an administrative database. We have acknowledged this in the discussion section of our paper (Third limitation of the study).

We would argue however that this limitation does not represent a misclassification of patients. We classified patients precisely according to the definition of CCM used by PI:

"The starting and ending dates of critical care physician management recorded in the database coincided with the dates of ICU admission and discharge."

Consider for example, a patient who was admitted to the ICU at noon and was subsequently managed by a critical care trained physician from 6 pm on the initial day until 7 days later when he was discharged. We would classify this patient as belonging in the CCM group. It certainly would not be appropriate to classify him in the non-CCM group.

Next, consider the issue for the aggregate of patients classified as having CCM their entire stay. If we assume that patients are admitted around the clock, the average time of admission would be half way through, or 12 hours, into the 24 hour period of a calendar day; and if some of these patients are transferred to CCM physicians, this transfer takes place on average half way through the remaining 12 hours of the day. Thus, on average, those patients transferred to a critical care trained physician on the first calendar were likely to receive an average of 6 hours of care under the supervision of a doctor not trained in critical care.

The question then becomes, how to interpret our results given the definition we use. How much does it matter if we do not know precisely when in the first 24 hours in the ICU management by critical care physicians begins? We ought to and do acknowledge that sicker patients are likely to be transferred to critical care trained physicians. The modified SAPS II score and propensity score are designed to adjust for the increased likelihood that critical care trained physicians care for sicker patients.

In order to address the possibility that some of the sickest patients are more likely than others to get transferred to CCM some time after admission on the first calendar day, we ran an analysis eliminating those admissions that were in the top decile of expanded SAPS II which would represent the top 10% of the sickest patients. The overall results do not change (Table 1). In addition, we ran our primary analysis without 10% of the shortest ICU LOS's (Table 2) and again without 25% of the shortest stays (Table 3). Deleting the 10% of patients with the shortest ICU stay, is equivalent to dropping those patients whose ICU stay is less than 0.66 days. Deleting the 25% with the shortest ICU stay, is equivalent to dropping out those whose ICU stay is less than 0.98 days, which should therefore delete all patients cared for by another physician. The results without the 10% short stays are almost identical to the results in the original manuscript. For the 25% shortest stays, the results are similar; however, the p-values are slightly attenuated. We believe that the results of these additional analyses demonstrate that even without including patients who may have been cared for by non-CCM physicians for a brief period of time, the results of the study are unchanged.

Thus despite the limitation of the dataset which does not allow us to know which patients received delayed critical care management on the first calendar day, this additional analysis that excludes the sickest patients who might be most likely to be transferred or patients with a very short length of stay does not change our major finding. We believe this addresses the concern about bias due to the transfer of such patients.

We do agree with the point that the modified SAPS II score does not offer any information about trends in a patient's condition during the 24 hour period of the first day. In eliminating the patients with the worst scores from this latest analysis, patients who are sickest at either the beginning or the end, or for that matter at any point of the day, are excluded. Since the score is based on the worst values over the 24 period, it seems to us that it suffices to capture information regarding who is most likely to be transferred to the care of a critical care trained physician at any point in the first calendar day.

We comment on one last point made by Quartin et al, who mention that ".. the exclusion from analysis of patients who required transfer to critical care management after the first ICU day effectively removes high risk patients from the non-intensivist cohort." We would suggest that by excluding these patients we reduce the likelihood of these sicker patients being assigned to either the CCM or non-CCM group. Hence this exclusion does not seem to bias in favor of the non-CCM group.

We appreciate the comments by Quartin et al and the opportunity for further the dialogue on the important issue.