Abstract

Objectives The goal of this study was to examine the calibration of a validated risk-adjustment model in very high-risk percutaneous coronary intervention (PCI) cases and assess whether sites' case mix affects their performance ratings.

Background There are concerns that treating PCI patients with particularly high-risk features such as cardiogenic shock or prior cardiac arrest may adversely impact hospital performance ratings. However, there is little investigation on the validity of these concerns.

Methods We examined 624,286 PCI procedures from 1,168 sites that participated in the CathPCI Registry in 2010. Procedural risk was estimated using the recently published Version 4 National Cardiovascular Data Registry (NCDR) PCI risk-adjusted mortality (RAM) model. We calculated observed/expected mortality using several risk classification methods, and simulated hospital performance after combining their highest risk cases over 2 years into a single year.

Results In 2010, crude in-hospital PCI mortality was 1.4%. The NCDR model was generally well calibrated among high risk, however there was slight overprediction of risk in extreme cases. Hospitals treating the highest overall expected risk PCI patients or those treating the top 20% of high-risk cases had lower (better) RAM ratings than centers treating lower-risk cases (1.25% vs. 1.51%). The observed/expected ratio for top-risk quintile versus low-risk quintile was 0.91 (0.87 to 0.96) versus 1.10 (1.03 to 1.17). Combining all the high-risk patients over a 2-year period into a single year also did not negatively impact the site's RAM ratings.

Conclusions Evaluation of a contemporary sample of PCI cases across the United States showed no evidence that treating high-risk PCI cases adversely affects hospital RAM rates.

Hospital quality is now judged by several metrics. A common measure for hospital outcomes is risk-adjusted mortality (RAM) (1,2), which is calculated for many different clinical conditions, including percutaneous coronary intervention (PCI). Historically, these risk-adjustment models have their basis in the belief that mortality, if appropriately adjusted for case mix, is a measure of overall hospital quality (3). Hospitals and providers have voiced concerns that risk-adjustment models employed to account for case mix may not adequately account for particularly high-risk clinical features (4,5), and that clinicians and hospitals treating a greater number of high-risk patients may have a worse rating (6). At worst, these concerns might lead clinicians to avoid very high-risk, but appropriate-to-treat, patients in order to protect their RAM ratings (7,8).

PCI is a commonly performed procedure, occurring at approximately 1,700 medical centers in the United States (9). These centers have a wide variation in hospital volume and case mix (10), hence RAM ratings are a commonly used, though controversial, quality measure. Cardiac arrest and cardiogenic shock patients represent the highest-risk patients potentially treated by PCI. These 2 groups of patients have the highest in-hospital mortality, but may also have the highest potential benefit from urgent percutaneous revascularization (11–15). PCI practice patterns in the United States show significant variability for patients with cardiac arrest and cardiogenic shock, and recent data indicate that public reporting of performance measures may partially drive the differences seen in these practice patterns (16). A possible explanation for these data is that PCI practitioners may change their behavior (become more risk avoidant) to avoid a negative impact on their RAM ratings. There has been significant documentation of these concerns (17) and even observational evidence of shifts in PCI and surgical practice patterns (8,18).

Our study aimed to evaluate the accuracy of a validated and widely used risk adjustment model from the National Cardiovascular Data Registry (NCDR) CathPCI registry, the current NCDR PCI RAM model, to estimate mortality in moderate- and high-risk subsets. We then aimed to assess whether sites treating more high-risk cases have worse observed versus expected mortality ratios and worse RAM ratings than sites treating lower-risk patients.

Methods

The CathPCI Registry is a collaborative effort of the American College of Cardiology Foundation and the Society for Cardiovascular Angiography and Interventions, and remains the largest ongoing registry of PCI in the United States. Descriptions of the registry have been previously published (9). The registry collects data on patient and hospital characteristics, clinical presentation, procedural characteristics, and in-hospital outcomes for PCI procedures from >1,200 sites across the United States. Data are entered into NCDR-certified software at participating institutions, and exported in a standard format to the American College of Cardiology. There is a comprehensive data quality program, including both data quality report specifications for data capture and transmission, as well as an auditing program (19).

Study population

All PCI procedures in the NCDR CathPCI registry in calendar year 2010 were initially included. We excluded those procedures that were repeat procedures in a patient, that were transferred to another facility after the procedure, those with unknown vital status, and those hospitals whose annual volume was <50 PCIs. This resulted in a final study population of 624,286 PCI procedures at 1,168 hospitals (Figure 1).

Outcomes and current risk model

The primary outcome evaluated in this study was in-hospital mortality. The current, Version 4 NCDR PCI RAM model has been recently described in detail (20). Briefly, the current model was based upon the findings in over 1 million PCI patients seen between mid-2009 and mid-2011, and includes 18 clinical variables. It was developed in 60% of the total population, and validated in the remaining 40% of the study population. The model showed excellent discriminative ability, with a c-index of 0.93. It also showed good calibration with a Hosmer-Lemeshow goodness-of-fit statistics p value of <0.05.

Statistical methods

As previously described, the predicted in-hospital mortality risk was calculated for all patient visits in the sample, and the distribution of patient risk was plotted. High-risk patients were prospectively defined as those with an expected mortality rate of ≥10%. Model calibration was evaluated by plotting the expected versus the observed mortality in deciles of expected mortality. Further evaluation of model calibration was performed by plotting the expected versus observed mortality for the subset of cardiogenic shock and/or salvage PCI patients, defined as cardiogenic shock at the start of the procedure, requiring manual compression for cardiac arrest within 10 min of the procedure start, or the requirement of extracorporeal hemodynamic support at the beginning of the procedure.

The hospital-level predicted RAM was then calculated by combining the patient predicted mortality risk for all patients treated at a hospital site. The hospital sites were grouped into risk quintiles by overall hospital expected mortality; patient characteristics and hospital characteristics were reported for each quintile. The ratio of observed versus expected mortality was calculated with 95% confidence intervals (CIs) (based on binomial distribution) for each quintile of risk. Sensitivity analyses were also performed by repeating this approach with hospitals ranked in quintiles by the percent of cardiogenic shock/cardiac arrest patients treated, and by the percentage of high-risk (≥10% predicted mortality) patients treated.

As a third method of analysis, sites’ highest-risk cases from a 2-year period (2009 quarter 2 through 2011 quarter 2) were combined into a single year’s volume to simulate a “concentrated risk year,” and the hospital sites were used as their own control. This was done by rank ordering the patients according to their predicted risk of mortality, selecting the hospitals’ highest-risk cases, and assuming that these patients were seen in a single year, keeping annual case volume constant. Observed versus expected mortality (O/E) ratios were again calculated with 95% CIs for each quintile based on expected risk. The O/E ratio in the concentrated risk year scenario was then plotted against the average O/E ratio during that time period, with 95% CIs. Centers whose 95% CI for observed mortality was significantly better or worse than for the predicted mortality were flagged as outliers. The frequency of outlier status was then compared between the concentrated risk year and the overall 2-year period. All statistical analyses were performed by the Duke Clinical Research Institute using SAS software (version 9.2, SAS Institute, Cary, North Carolina). The study was reviewed by the Duke Institutional Review Board and was determined to meet the definition of research not requiring informed consent, given that patient information is collected anonymously without unique patient identifiers and only aggregate data are presented and published.

Results

From January 2010 to December 2010, 647,210 PCI procedures during 630,377 admissions in 1,208 sites were registered in the CathPCI registry. After application of the aforementioned exclusion criteria, 624,286 PCI procedures in 1,168 sites were included in the analysis (Figure 1). The CathPCI mortality model (V4) demonstrated good performance over the spectrum of risk (from 0.01% to 100%) (Figure 2A). The model performance seemed equally good when the same O/E ratios were calculated in the extremely high-risk subgroups of patients with cardiogenic shock and/or salvage PCI and in patients at extreme mortality risk (>10%) (Figure 2B).

Calibration plots for the full spectrum of mortality risk (A) and in high-risk subgroups of cardiac arrest and salvage percutaneous coronary intervention (PCI) patients (B) are shown. The observed versus expected mortality plots show good calibration of the CathPCI V4 model for the full spectrum of risk in PCI patients. Carshock = cardiogenic shock.

The baseline clinical characteristics of the patient population stratified by the hospital risk quintiles are shown in Table 1. Patients at higher-risk hospitals were of similar age, race, and sex, and had similar frequency of insulin-dependent diabetes, though were less likely to have peripheral arterial disease, cerebrovascular disease, prior congestive heart failure compared with those in lower-risk hospitals. However, patients at the highest-risk hospitals were more likely to have prior myocardial infarction and to present with ST-segment elevation myocardial infarction, New York Heart Association functional class IV heart failure symptoms, shock, or cardiac arrest. Also, the proportion of PCI procedures at high-risk hospitals was less likely to be elective, more likely emergent, and more often involved the proximal left anterior descending coronary artery when compared with lower-risk hospitals. The hospital-level characteristics, grouped by hospital risk quintile, are shown in Table 2. High-risk hospitals, compared with low-risk hospitals, were larger (by bed number), had a lower annual PCI volume, and were more likely to be rural or academic centers. There was substantial variation present in the geographic distribution of hospitals by risk quintile. A greater proportion of hospitals in the West are in the highest-risk quintile group, whereas a greater proportion of Southern hospitals are in the lowest-risk quintile group, when compared with the Northeast and Midwest regions (Table 3).

On comparison, the O/E ratios for each risk quintile of hospitals were similar and approximately 1.0, except for the highest-risk quintile of hospitals, for which the O/E ratio was 0.91 (95% CI: 0.87 to 0.96), indicating these hospitals performed better than expected (Table 4). In addition, when comparing hospital-level RAM, the highest-risk hospitals had slightly lower RAM than the lowest-risk hospitals. A sensitivity analysis, in which hospital sites were regrouped by frequency of patients with an expected mortality risk of ≥10% (Online Table S1) and by the frequency of cardiogenic shock/ salvage patients (Online Table S2), found similar results, with the hospitals’ RAM rates being slightly lower in the highest-risk hospitals.

As a final analysis, we compared the overall performance of the hospitals in an extreme scenario, the concentrated risk year (Table 5). For all risk quintiles, the O/E ratio was approximately 1.0 with CIs that crossed the line of unity, and the agreement of concentrated risk year O/E with average year O/E was consistent across nearly all centers (Figure 3). In addition, during the overall 2009 and 2011 period, 65 (5.4%) centers were flagged as lower-performing outliers, whereas 48 (4.0%) were classified as higher-performing outliers. During the concentrated risk year, 60 (5.0%) were flagged as lower performing, and 45 (3.7%) as higher-performing outliers, indicating no increase in the identification of outlier hospitals when creating a higher-risk annual case mix.

Observed/Expected Mortality by Hospital Risk Quintile in a “Concentrated Risk Year” Versus an Average Year

Observed versus expected mortality (O/E) ratios when the highest-risk cases or concentrated into a single year compared with overall O/E ratio in an average year show there is still a 1:1 relationship at all levels of risk. The 95% confidence intervals are shown with dotted lines, and a regression line fit to the hospital data shows consistent agreement between concentrated risk year and average year performance.

Concentrated Risk Year Versus Average Year O/E Ratios by Predicted Mortality Risk Quintile

Discussion

There is a growing emphasis to publically report hospitals’ performance and outcomes from common diseases. Our study showed the current CathPCI V4 mortality model performed well with good discriminative ability and calibration across the spectrum of risk. Using this model, we also found that hospitals treating more high-risk PCI patients did not have worse outcomes after risk adjustment by this means. We tested this by comparing observed/expected mortality in hospitals grouped by 3 different measures of risk, and concentrating the highest-risk cases from a 2-year period into 1 year, and subsequently comparing observed/expected mortality. In both instances, we found no evidence that RAM was worse when higher-risk patients were included.

High-risk and extreme-risk cases

Previous investigation raised concerns that even a well-calibrated model may lack certain variables that are known to signal the “compassionate use” of PCI and thus a very high-risk group that could affect RAM ratings. Resnic et al. (21) studied this question in the MassDAC (Massachusetts Data Analysis Center) registry, defining compassionate use as either coma on presentation for emergent PCI, ventricular assist device or extracorporeal bypass, or cardiopulmonary resuscitation at the initiation of the procedure. The combination and inclusion of these variables into the standard NCDR CathPCI mortality model did improve the c-statistic for prediction of mortality (0.87 to 0.90; p < 0.001). However, the number of cases that qualified under these compassionate use criteria was very small, and the authors included a further analysis of hospital RAM classification showing that inclusion of compassionate use criteria did not significantly change any hospital’s RAM rating.

Physician attitudes and practices

RAM as a measure of hospital quality has been criticized in many ways, but remains a widely reported metric. Physician survey data reveal concern over the effect of high-risk cases on hospital and provider RAM estimates. Narins et al. (17) published data from a survey of physicians in New York that indicates 85% of respondents believed that the RAM model was insufficient to account for case mix and that nearly 80% of physicians felt that their decision making in the case of critically ill patients in need of angiography or PCI might be affected by the presence of RAM ratings. These sentiments are borne out in recently published data from New York and Massachusetts. Moscucci et al. (8) showed that in comparison to Michigan, where there was no system of public reporting in place, New York patients undergoing PCI were less likely to have heart failure or cardiogenic shock and had lower unadjusted mortality rates. The authors proposed one explanation for this phenomenon might have been physician reluctance to perform high-risk cases because of public reporting. McCabe et al. (18) demonstrated a similar pattern of practice in Massachusetts soon after statewide mandate of public reporting of PCI RAM rates. At previously designated outlier institutions in Massachusetts, patient risk characteristics were significantly lower after initiation of public reporting than before, again indicating a possible change to risk-averse practices because of public reporting.

Our study provides no evidence to support these attitudes and practices. In hospitals across the United States, those with the highest-risk cases showed no adverse effect on their RAM. In fact, there was a signal in our analysis that hospitals that perform the highest-risk cases, by predicted mortality, may be outperforming expectations, thus have a lower than expected RAM. Our study’s results are based on over 600,000 cases from more than 1,150 sites around the United States, and provided similar results when hospital’s case mix was risk stratified using 3 different metrics. In each case, there was no adverse effect on observed/expected mortality, and RAM was reduced in the highest-risk hospitals as compared with the lowest-risk hospitals. Finally, when artificially creating a year with a very high-risk case mix, there was consistent site performance and no increase in the number of outlier hospitals identified in this population.

Clinical implications

With the expansion of public reporting in the United States, and the potential changes in practice patterns because of these reports, our results have important implications. Dissemination of these data could potentially reassure practitioners, emphasizing that treating the highest-risk patients will not adversely affect RAM ratings. These high-risk patients may be those that benefit the most from intervention (11,12). Our results also show significant geographic variation in hospital practice patterns with regard to PCI risk. Programs of surveillance should be established with the goal of preventing a further shift in practice towards inappropriate avoidance of high-risk patients, and providing the highest-quality care across all geographic regions (4).

Study limitations

Certain factors should be considered in the interpretation of our study. First, the CathPCI V4 mortality model was constructed using data from this time period, and thus would have validity for our analysis, but may not be generalizable to all patients undergoing cardiac catheterization. Also, previous CathPCI-based mortality models may not have included variables that allowed for accurate prediction in the highest-risk cases (21). The current mortality model seems to have addressed these concerns (20). The NCDR CathPCI is a large national registry that includes community and academic centers, but may not fully represent practice in all areas of the United States. However, it is estimated that the NCDR collects data from 85% to 90% of all U.S. centers and >90% of all PCIs performed (9,19). Finally, CathPCI is a voluntary program, and thus hospitals that choose to participate may be more attuned to quality measures than other hospitals. This also presents the possibility that hospitals might “up-code” patient risk to improve quality measures such as O/E ratios. There is no evidence that this occurred in our sample population.

Conclusions

We found that the current CathPCI V4 RAM model accurately predicted mortality in high-risk patient subgroups and, if anything, overestimated the hazard in very high-risk patients. We found nothing that would suggest that altering the current decision making for treating high-risk patients would result in worse demonstrated hospital RAM ratings. These findings should reassure physicians and could help to prevent inappropriate risk-avoidant behavior in PCI practice.

Appendix

Appendix

For supplemental tables, please see the online version of this paper.

Footnotes

Dr. Sherwood is funded by National Institutes of Health T-32 training grant #5 T32 HL 7101-37. Dr. Masoudi is senior medical officer and chair of the Science and Quality Oversight Committee of the National Cardiovascular Data Registry. Dr. Messenger is chair of the CathPCI Research and Publications Subcommittee of the National Cardiovascular Data Registry. Dr. Weaver is a member of the Management Board of the National Cardiovascular Data Registry. Dr. Peterson has received research grants (to the Duke Clinical Research Institute) from the American College of Cardiology, American Heart Association, Eli Lilly and Company, Janssen Pharmaceuticals, and the Society of Thoracic Surgeons; and has received consulting/honoraria from Boehringer Ingelheim, Genentech, Janssen, Merck & Co., AstraZeneca, and Sanofi. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.

(2008) Impact of the New York State Cardiac Surgery and Percutaneous Coronary Intervention Reporting System on the management of patients with acute myocardial infarction complicated by cardiogenic shock. Am Heart J155:267–273.

(2011) Improvement in mortality risk prediction after percutaneous coronary intervention through the addition of a “compassionate use” variable to the National Cardiovascular Data Registry CathPCI dataset: a study from the Massachusetts Angioplasty Registry. J Am Coll Cardiol57:904–911.