Author Affiliations: Institute for Social Research (Dr Nicholas) and Department of Surgery, University of Michigan (Drs Osborne, Birkmeyer, and Dimick), Michigan Surgical Collaborative for Outcomes Research and Evaluation, Ann Arbor.

ABSTRACT

ObjectivesTo determine whether high rates of compliance with perioperative processes of care used for public reporting and pay-for-performance are associated with lower rates of risk-adjusted mortality and high-risk surgical complications.

ConclusionsCurrently available information on the Hospital Compare Web site will not help patients identify hospitals with better outcomes for high-risk surgery. The Centers for Medicare and Medicaid Services needs to identify higher leverage process measures and devote greater attention to profiling hospitals based on outcomes to improve public reporting and pay-for-performance efforts.

Figures in this Article

As variations in surgical quality are increasingly observed, payers are escalating efforts to reduce them.1- 3 The Centers for Medicare and Medicaid Services (CMS), the largest public payer, now mandates public reporting of 2 sets of the Surgical Care Improvement Project (SCIP) measures covering infection and venous thromboembolism. Hospitals are required to submit data quarterly, which are posted on the Hospital Compare Web site (http://www.hospitalcompare.hhs.gov/), to receive annual Medicare payment updates.4 This reporting is believed to aid patients and payers in choosing high-quality hospitals and to stimulate quality improvement among reporting hospitals.5- 7

It is unclear whether these efforts will translate into better outcomes for surgical patients. Although the SCIP measures were selected because of strong evidence linking them to certain outcomes, there is reason to be skeptical that improved compliance will result in significant improvements in the most important outcome, risk-adjusted mortality rate. Namely, SCIP processes are associated with outcomes that are rare (eg, deep venous thrombosis and pulmonary embolism) or considered secondary (eg, superficial surgical site infections).8,9 It is unknown whether measured processes of care are important determinants of surgical outcomes. If there is a weak link between process compliance and surgical outcomes, CMS public reporting and pay-for-performance efforts will be unlikely to stimulate important improvements or to help patients find the safest hospitals.10,11

In this context, we sought to determine whether hospital compliance rates for targeted surgical processes of care reported on the Hospital Compare Web site are related to risk-adjusted mortality rate, venous thromboembolism, and surgical site infection. We used national Medicare claims data to focus on patient outcomes after 6 high-risk surgical procedures.

METHODS

HOSPITAL COMPARE DATA

The Hospital Compare Web site reports of medical care process compliance have been described previously.12 Hospitals began reporting 2 SCIP measures in 2005: the rate of prophylactic antibiotic receipt within 2 hours of surgery and the rate of prophylactic antibiotic therapy discontinuation within 24 hours of surgery. Three additional measures were added in the 2006 reports: rate of correct antibiotic administration to prevent infection, recommended venous thrombosis prophylaxis ordered, and recommended venous thrombosis prophylaxis ordered within 24 hours of surgery.13 Hospitals report the number of patients eligible for each process and the percentage receiving each process. The SCIP measures are collected across a broad range of procedures, including cardiac, orthopedic, vascular, general, and gynecologic surgery.14

We obtained archived Hospital Compare data covering the period from January 1, 2005, through December 31, 2006. Data are posted with a 9-month lag.15 To assess hospital compliance, we calculated an opportunity score based on the number of times a hospital complies with recommended measures for each eligible patient on as many as 5 SCIP measures for each year of Hospital Compare data (2 infection measures introduced in 2005 and 3 infection and 2 venous thromboembolism measures collected in 2006). Hospitals are classified into quintiles of composite process compliance score. In sensitivity analyses, we examined infection and venous thromboembolism scores separately.

Of the total hospitals, 39.8% (performing 43.3% of operations) reported SCIP compliance in 2005. Nonreporting hospitals had significantly lower annual procedure volume (78 vs 90) and were more likely to be government owned or nonprofit. No difference was found in nonsurgical composite compliance rates across reporters and nonreporters (97.0% of SCIP nonreporters reported other measures). By 2006, virtually all hospitals performing study procedures reported SCIP compliance. Only 15 hospitals performing 142 procedures did not report.

MEDICARE INPATIENT DATA

We identified all Medicare fee-for-service hospitalizations for 6 high-risk surgical procedures in the 100% Medicare Provider Analysis and Review data set from January 1, 2005, through December 31, 2006. Eligible admissions include abdominal aortic aneurysm repair, aortic valve repair, coronary artery bypass graft, esophageal resection, mitral valve repair, and pancreatic resection. These hospitalizations are well suited to our study because they are sufficiently common and high risk to reveal variation in surgical mortality and complication rates across hospitals.

STATISTICAL ANALYSIS

We estimate risk-adjusted surgical outcomes overall and for each procedure using hierarchical linear models, including hospital-level indicators of quality known to relate to patient outcomes (procedure volume and indicators for highest SCIP compliance quintile and lowest SCIP compliance quintile) and patient characteristics (age, race, sex, severity of comorbid conditions classified using the Charlson comorbidity index,16 patient zip code median income from the 2000 US Census, whether the admission was scheduled (relative to emergent and urgent admissions), and year of admission).16,17 Hospital random effects are included to account for clustering of patients in hospitals.

The base analysis examines the relationship between contemporaneous surgical process compliance and mortality rate. We also examine lagged SCIP compliance and mortality rate, essentially testing whether the historical information posted on the Hospital Compare Web site effectively provides patients with information about their risk of an adverse surgical event.

We conducted additional analyses using only the 2006 data, for which a more comprehensive set of SCIP measures was available. In these data, we examine whether the collected measures provide information about patient risks of experiencing targeted outcomes associated with the SCIP measures of venous thrombosis and postoperative surgical site infection.

RESULTS

Mean surgical compliance rates varied considerably, ranging from 53.7% in hospitals in the lowest compliance quintile to 91.4% in hospitals in the highest compliance quintile (Figure 1). Hospitals in the lowest quintile of process compliance were less likely to be accredited or to have an emergency department (Table 1). These hospitals also had lower rates of nonsurgical process compliance and lower surgical volume.

We found little evidence of a consistent relationship between hospital compliance with processes of care and operative mortality rate (Figure 2). In univariate analysis, mortality rates in the lowest compliance hospitals were statistically indistinguishable from those in the highest quintile of compliance for all procedures studied except aortic valve replacement, in which the highest compliance hospitals had lower mortality rates. Hospitals that did not report SCIP compliance had rates of risk-adjusted mortality similar to those in the highest quintile of SCIP compliance.

Unadjusted complication rates are lower among hospitals in the lowest quintile of compliance with SCIP measures than those in the highest compliance quintile for deep venous thrombosis and pulmonary embolism (low compliance, 0.43%; high compliance, 0.59%) and infection (low compliance, 1.1%; high compliance, 1.9%).

We conducted several additional analyses to test the robustness of these findings. Results were unchanged when we replaced our SCIP compliance composites with outcome-specific measures; risk of infection did not vary with hospital compliance with SCIP infection compliance, and risk of venous thromboembolism did not significantly vary with use of venous thromboembolism prophylaxis. We also eliminated hospitals in the middle 3 quintiles of compliance and directly compared the highest compliance hospitals with the lowest compliance hospitals. There is no difference in risk-adjusted mortality (OR, 0.88; 95% CI, 0.78-1.01), venous thromboembolism (1.12; 0.85-1.48), or surgical site infection (1.04; 0.85-1.28) at highest compliance hospitals compared with lowest compliance ones.

We also consider extended length of stay, which could result from a number of postoperative complications. Extended hospital stays, in the highest quintile of procedure-specific inpatient days, are less likely at more compliant hospitals. Patients at the highest compliance hospitals are 12.0% less likely to experience an extended stay relative to middle compliance hospitals (OR, 0.88; 95% CI, 0.81-0.94), although no difference was found between lowest and middle compliance hospitals (1.05; 0.95-1.17).

COMMENT

The risk of patient death and the higher costs of Medicare associated with adverse surgical events emphasize the importance of providing beneficiaries with information that facilitates the choice of a high-performing hospital. There is a clear business case for increased use of high-quality hospitals for surgical patients. Although compliance with surgical process measures varies widely, we find little evidence that SCIP measures reliably correlate with risk-adjusted surgical outcomes. As a result, patients who choose their hospital based on high rates of process compliance will not improve their chance of survival or complications. However, patients choosing high compliance hospitals reduce their risk of experiencing an extended stay.

In contrast to our findings for surgery, previous research by Werner and colleagues18,19 have found mixed evidence of a relationship between process compliance and mortality rate. These authors have shown that medical process compliance rates reported on the Hospital Compare Web site, particularly those for acute myocardial infarction care, are related to inpatient mortality rate and reflect other dimensions of quality. Similar to our own results, however, medical process compliance explains little of the variation in mortality rate.

Our study has several important limitations primarily related to our use of administrative Medicare data. Hospitals are required to report process measures for all eligible admissions, not just Medicare admissions. Using all-payer data, we have generally found that hospitals' Medicare surgical mortality rates are highly correlated with their overall surgical mortality rates. However, it is possible that we failed to find a relationship between process compliance and surgical outcomes because of insufficient sample size. Relatively few of the high-risk procedures we examined are performed in the low compliance hospitals. Findings are robust to alternative categorizations of compliance and across multiple measures of adverse outcomes.

Our study also faces the well-known limitation of risk adjustment in administrative data. However, this will only bias our results if unobserved patient acuity is systematically related to hospital process compliance, which seems unlikely, especially because surgical process compliance rates were not generally known before their reporting on the Hospital Compare Web site during our study period. Furthermore, for rates of process compliance to be meaningful as indicators of hospital quality, their relationship to patient outcomes should not be heavily dependent on complex risk adjustment techniques, which will be inaccessible to most Hospital Compare Web site users.

There are several reasons why public reporting based on SCIP measures may be inadequate to differentiate quality of inpatient surgical care. The SCIP measures are low leverage because they relate to secondary and relatively less important outcomes. Even when processes are tied to an important outcome such as pulmonary embolism, these events are rare and offer insufficient variation to differentiate between high- and low-quality hospitals.

Unlike other surgical quality measures, such as procedure volume and inpatient mortality rate, which can be calculated from existing administrative data, reporting process measures imposes additional compliance costs on hospitals. It is important to determine whether tracking these measures provides useful information relative to the cost of data collection. If not, the CMS and other payers and policy makers targeting surgical compliance measures may want to consider other reporting options. Direct reporting of surgical outcomes would be an alternative. The Leapfrog Group, a large coalition of health care purchasers, encourages use of high-quality hospitals based on a composite measure of mortality rate and volume for several high-risk surgical procedures.

Despite the intentions of the CMS to provide patients with information that will facilitate patient choice of high-quality hospitals, currently available information on the Hospital Compare Web site will not help patients identify hospitals with better outcomes for high-risk surgery. The CMS needs to identify higher leverage process measures and devote greater attention to profiling hospitals based on outcomes for improved public reporting and pay-for-performance programs. Future research should ascertain whether process measures become more useful as indicators of surgical quality as public reporting programs mature.

Funding/Support: This study was supported by training grant AG000221-17 from the National Institute on Aging to Dr Nicholas, a training grant from the Robert Wood Johnson Foundation to Dr Osborne, grant R01 CA098481 from the National Cancer Institute to Dr Birkmeyer, and career development award K08 HS017765 from the Agency for Healthcare Research and Quality to Dr Dimick.

Correspondence

The American Medical Association is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.
The AMA designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 CreditTM per course. Physicians should claim only the credit commensurate with
the extent of their participation in the activity. Physicians who complete the CME course and score at least 80% correct on the quiz are eligible for AMA PRA Category 1 CreditTM.

Indicate what change(s) you will implement in your practice, if any, based on this CME course.

Your quiz results:

The filled radio buttons indicate your responses. The preferred responses are highlighted

For CME Course:
A Proposed Model for Initial Assessment and Management of Acute Heart Failure Syndromes

Indicate what changes(s) you will implement in your practice, if any, based on this
CME course.

Instructions

Thank you for submitting a comment on this article. It will be reviewed by JAMA Surgery editors. You will be notified when your comment has been published. Comments should not exceed 500 words of text and 10 references.

Do not submit personal medical questions or information that could identify a specific patient, questions about a particular case, or general inquiries to an author. Only content that has not been published, posted, or submitted elsewhere should be submitted. By submitting this Comment, you and any coauthors transfer copyright to the journal if your Comment is posted.

* = Required Field

Comment Author(s)* (if multiple authors, separate
names by comma)

Example: John Doe

Affiliation & Institution*

Disclosure of Any Conflicts of Interest*
Indicate all relevant conflicts of interest of each author below, including all relevant financial interests, activities, and relationships within the past 3 years including, but not limited to, employment, affiliation, grants or funding, consultancies, honoraria or payment, speakers’ bureaus, stock ownership or options, expert testimony, royalties, donation of medical equipment, or patents planned, pending, or issued. If all authors have none, check "No potential conflicts or relevant financial interests" in the box below. Please also indicate any funding received in support of this work. The information will be posted with your response.

This feature is provided as a courtesy. By using it you agree that that you are requesting the material solely for personal, non-commercial use, and that it is subject to the AMA's Terms of Use. The information provided in order to email this article will not be shared, sold, traded, exchanged, or rented. Please refer to The JAMA Network's Privacy Policy for additional information.

Athens and Shibboleth are access management services that provide single sign-on to protected resources. They replace the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Athens or Shibboleth authentication, please contact your site administrator to receive your user name and password.