Introduction

In recent years, the American College of Rheumatology (ACR) has focused increasingly on the development, review, dissemination, and implementation of quality measures. To help position rheumatology providers as the leading force in defining the quality of rheumatology care, the ACR convened a Quality Measures White Paper Development Workgroup. This diverse workgroup's charge included synthesizing the information most relevant to the ACR membership regarding quality measurement. As described in this manuscript, the workgroup ultimately participated in a consensus development initiative to craft a set of criteria for quality measures submitted to the ACR for approval. The criteria are intended to guide measure developers and discourage adoption of subpar measures.

This White Paper is written for rheumatologists interested in quality measurement, groups undertaking quality measure development, and policy makers. It is organized into the following main sections: a brief introduction provides context and describes the ACR's national role and positions; section 1 outlines quality measure domains and reviews measure attributes; and section 2 describes and reports results of an ACR workgroup–led process to develop specific criteria for ACR approval of quality measures.

The quality landscape for rheumatologists

Over the last decade there has been increasing recognition that the quality of health care in the US is suboptimal. Two sentinel reports by the Institute of Medicine, To Err is Human: Building a Safer Health System (1) and Crossing the Quality Chasm: A New Health System for the 21st Century (2), stimulated a call for change. Consistent with trends seen in the US health care system as a whole, significant deficits in health care quality have been identified in the rheumatic conditions (3–6). These include limited adherence to guidelines for rheumatoid arthritis (RA) (7–10) or minimal standards of care for osteoarthritis (11–13) and gout (14), as well as low rates of receipt of vaccinations for immunocompromised patients (15, 16) and treatment of osteoporosis postfracture (17). Variations in care by geographic region and physician specialty (18, 19) suggest considerable room for improvement in the quality of rheumatic care delivered to the US population and provide a clear call to improve health care for patients with arthritis and allied conditions.

As the US quality of care movement has gained momentum, various stakeholders, including the ACR, have become actively involved in quality measurement and improvement efforts. Table 1 lists some of the key stakeholders and their relevant activities in the health care quality movement. One of the most influential stakeholders is the Federal Government, with the Department of Health and Human Services and the Centers for Medicare and Medicaid Services (CMS) launching quality initiatives in 2001 regarding accountability and public disclosure. A program with special relevance to providers in the ambulatory setting is the CMS Physician Quality Reporting Initiative (PQRI), a voluntary program that provides bonus payments for reporting related to care provided to Medicare patients (online at http://www.cms.hhs.gov/PQRI/). For the PQRI and most of the quality programs employed by the CMS, private payers, and other stakeholders, quality measures are used as the basis for assessment of health care quality. Table 2 includes common definitions of quality of care and quality measures.

Accreditation of practice settings, determination of malpractice rates based on practice performance

Table 2. Common definitions of quality and quality measures

Quality of care

The degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge (2)

Quality measure

A mechanism to quantify the quality of an aspect of care by comparison to a criterion (29)

Clinical performance measure

A subtype of quality measure for assessing the degree to which a provider competently and safely delivers clinical services that are appropriate for the patient in the optimal time period (29)

The ACR's position and role

Delivering the highest-quality patient care is paramount to rheumatologists and rheumatology health professionals, and the ACR is committed to being involved in the national quality movement on behalf of the professional rheumatology community. As such, the ACR engages with national quality groups such as those referenced in Table 1. By further examples, the ACR serves as a participating member in the National Quality Forum (NQF) and American Medical Association Physician Consortium for Performance Improvement, as a collaborator with the National Committee on Quality Assurance, and as an advocate for relevant quality initiatives with private insurers. The ACR's educational and scientific meetings also provide forums for teaching quality improvement methods and presenting the results of successful practice process improvement and quality measurement studies.

The ACR considers quality measurement a means to improve long-term outcomes and safety and enhance access to appropriate treatments. To begin to meet these goals, the ACR quality of care efforts focus on improved clinical data measurement of the process of care and disease outcomes. Routine assessment of rheumatology practices' performance of key processes and outcomes is an essential component of delivering high-quality health care (20–22), and will be further discussed in the following section of this article.

One of the challenges for rheumatologists seeking to engage in performance improvement is the abundance of quality measures proposed by various stakeholders. Evaluating these measures requires time and expertise. To offer assistance, the ACR engages in measure development, and also reviews and approves rheumatology-related measures and measure sets. Many of the quality measures approved by the ACR (23) were developed independently and without ACR support, but were later approved by the ACR Committee on Quality of Care and subsequently the ACR Board of Directors. Some measures reviewed by the ACR originated from national quality organizations, some from insurers, and some from academic investigators. Many of these measures were not approved by the ACR, often because insufficient scientific evidence supports their use, appropriate validation studies were not performed, and/or the data collection burden on practicing rheumatologists is significant. To better delineate the requirements and expected attributes of quality measures submitted for ACR approval, the ACR undertook a consensus development process, as described in section 2.

Another challenge to improving the quality of rheumatologic care is the successful implementation of quality measures. Moving from the creation of quality measures to their incorporation in routine clinical care is essential for ultimately enhancing the quality of rheumatologic care. The ACR created the Rheumatology Clinical Registry (RCR) in part to address this need (online at http://www.rheumatology.org/rcr). The use of quality measures for quality improvement has direct relevance to clinicians, clinical investigators, health care delivery groups, and most of all, the patients we serve.

Section 1. Quality measures: domains, desirable attributes, and uses

Types of quality measures.

Quality measures can be grouped into 3 related domains: structural measures (i.e., innate characteristics of providers and the system), process measures (i.e., what health care providers do in delivering care), and outcome measures (i.e., what happens to patients, particularly with respect to their health) (24, 25). As the field has evolved, quality measures have been further categorized into those related to access (i.e., the provision of timely and appropriate health care) (26), patient experience (i.e., the patient's perception of quality of care) (26), and efficiency (i.e., the relationship between clinical performance and resource use) (27).

Perhaps the most important distinction in rheumatology quality measurement is between process and outcome measures. Like other chronic diseases, quality assessment for rheumatology often focuses on processes of care. Process measures provide actionable targets for assessment and improvement, and have fewer inherent barriers than outcome measures. Health outcomes for rheumatic conditions require years to develop, and for many health conditions, uncertainty remains about the best outcomes or intermediate outcomes to examine. Further uncertainty remains about techniques to adjust for the differential risk of these outcomes due to factors not under doctors' control (e.g., disease severity, comorbidities). Moreover, health outcomes are often influenced by factors unrelated to the quality of clinical care, such as adherence to therapy, socioeconomic status, lifestyle choices, or environmental factors. Despite these measurement challenges, outcomes are often considered the most meaningful quality measures, as they demonstrate the end result of care provided. Ultimately, both process and outcome measures have strengths and weaknesses, and ideal quality measurement programs assess all relevant domains, including structural (to identify macro changes needed to achieve the highest-quality care), process (to offer specific and immediately actionable targets for improvement), and outcome (to assess the end result of the care provided).

Desirable attributes of quality measures.

Several groups, including the NQF, the Institute of Medicine, the National Committee on Quality Assurance (NCQA), and the Agency for Healthcare Research and Quality, have developed frameworks outlining the desirable attributes of quality measures. Three main conceptual areas include: the clinical and public health importance of the measure, the scientific validity of the measure, and the feasibility of implementing the measure. The key components of these measure attributes are briefly described in Table 3.

Table 3. Desirable attributes of quality measures

Measure attribute

Key components

Clinical and public health importance

Measure/results are relevant to stakeholders (clinicians, patients, or purchasers)

Measure addresses a condition with a high prevalence or incidence and/or with significant morbidity or mortality

Measure/results have potential to inform interventions that improve quality

Scientific validity

Measure is valid, i.e., assesses what it purports to measure Measure is reliable, i.e., results are reproducible and not subject to significant random variation

Measure includes risk adjustment when relevant, i.e., for factors outside the control of the provider or the health care practice (e.g., health insurance status, socioeconomic factors, or significant medical comorbidities)

Feasibility

Data elements required for the numerator, denominator, and exclusions are available

Potential inaccuracies or errors of the measure (often due to data source limitations) are identified Issues related to the timing and frequency of data collection, patient confidentiality, and other feasibility or implementation issues are noted

Information related to cost and burden of data collection is available

An important attribute of process measures is the link between the care process and improved clinical outcomes. This link can be demonstrated by data extrapolated from clinical trials and well-designed observational studies. For many chronic conditions, however, relevant high-quality data are lacking. Even when such data are available, experts may disagree on the interpretation of evidence. Therefore, rigorously developed quality measures often rely on validated methods that combine evidence and expert consensus (28). At times, the literature review, interpretation, and synthesis process are performed during the development of clinical practice guidelines, which in turn form the basis for quality measure development.

Uses of quality measures.

Quality measures may be appropriate for several related uses and have various intended audiences. Implementation of quality measures in the clinical practice setting, for quality improvement and/or accountability, was the primary focus of the ACR approval criteria development process described in section 2; however, the use of quality measures in the research setting is another important application. These uses are briefly described below.

Quality improvement.

Quality measurement is integral for both internal quality improvement (e.g., within a practice or hospital system) and external quality improvement (e.g., across many practices or systems) (29). Methodologies for successful implementation of measures for quality improvement in health care are emerging, often based on rapid cycle improvement (30–33), such as the Plan-Do-Study-Act (PDSA). The PDSA consists of beginning a series of rapid-cycle, small-scale tests of measurement and data-driven change, integrated into the process of clinical care. For example, periodic assessment of disease activity and functional status are two of the quality measures (developed by the NCQA in collaboration with the ACR) for the management of RA. Through the use of continuous quality improvement techniques over time, RA disease activity and functional status were efficiently collected from questionnaires in a rheumatology clinical practice (34, 35).

Accountability.

Data collected for accountability are intended to inform purchaser and consumer decision making, accreditation, and quality oversight (29). Performance measurement for accountability purposes is included in pay-for-performance, physician tiering, and other incentive programs. While conceptually appealing (the Institute of Medicine has called for increased compensation for those health care providers who deliver “high-quality care” [36]), implementation of feasible, valid, and widely-accepted performance-based programs has proven difficult (37).

Research.

Quality-focused research often addresses questions about the quality of health care provided in different health systems and populations, or tests the effectiveness of a specific quality improvement intervention. Quality improvement research often consists of multimodal programs based on behavioral theory that target patients, providers, and/or the health care system; involves a control group; and cluster randomizes physicians or groups of physicians (e.g., a clinic or a nursing home). One example of a quality improvement intervention is the use of electronic medical record reminders to order bone mineral density (BMD) testing in postfracture patients to increase the rate of BMD measurement (38). Quality improvement interventions studied within rheumatology have yielded mixed results (39).

The ACR can actively promote consistent, valid, and reliable quality measurement (for all of the uses described in the section above) by undertaking review and approval of quality measures on behalf of the membership. To achieve this goal, we initiated a consensus development process aimed at defining: 1) the priority of clinical conditions requiring quality measures, 2) desirable attributes of quality measures submitted for ACR approval, and 3) future recommended uses of ACR-approved measures.

This effort was intended to ensure a standard and systematic review process to guide measure developers that will seek ACR review, to discourage widespread adoption of subpar measures, and to prevent misapplication of measures. This section outlines the methods used to achieve consensus in these areas.

Consensus methods.

The project leader (KGS) identified 18 individuals to serve on the White Paper Development Workgroup, drawing from rheumatologists serving on the ACR Committee on Quality of Care, Subcommittee on Quality Measures, Board of Directors, and other ACR members at large. The panel was balanced to include both rheumatologists who spend a majority of their time in clinical practice (n = 9) and rheumatologists with expertise in health services research and quality measurement (n = 9). Workgroup members represented diverse geographic locations and practice settings (see Appendix A for a complete list of workgroup members and their affiliations).

We employed a formal consensus process, using a modification of the Delphi method (40–42) that incorporated multiple rounds of surveys as well as a face-to-face meeting, to derive a set of ACR approval criteria for quality measures and to define potential uses of ACR-approved measures. Initially, the project leader identified two broad topic areas that the ACR should consider in evaluating and approving quality measures: 1) a list of priority clinical content areas relevant to the care of rheumatic diseases for future quality measure development (clinical conditions were initially chosen based on their prevalence or severity or because they were thought to be areas in which gaps in health care quality exist), and 2) measure attributes. As discussed in section 1, fundamental to the process of quality measurement is the assumption that the measures themselves are relevant, scientifically sound (e.g., evidence-based, valid, reliable), and feasible to apply. Therefore, the project leader generated a preliminary list of desirable attributes for measure evaluation based on available scientific literature (28). Finally, the project leader drafted a list of future recommended uses of ACR-approved quality measures, based on the attributes that characterize the measures.

After a telephone conference in which the project goals were laid out and the rationale underlying the initial topic areas was discussed, workgroup members participated in round 1 of the modified Delphi procedure, in which they responded to a survey asking them to revise the preliminary list of priority clinical content areas, measure attributes, and recommended future uses of ACR-approved measures.

Round 1 resulted in 4 categories for future quality measure development (diseases, drugs, comorbidities/prevention, and access/experience of care) and 19 relevant clinical content areas. The workgroup also identified 44 desirable attributes of quality measures for consideration, and decided to assess these attributes according to two tiers (tier 1 and tier 2). Tier 1 measures would receive a higher level of ACR approval, requiring more rigorous development and testing methodologies. In contrast, tier 2 measures should meet the minimal acceptable standard of development and testing methodologies required for ACR approval. Finally, the Development Workgroup outlined 6 potential future uses of measures.

In round 2, individuals voted electronically on the revised list of clinical content areas, measure attributes, and recommended measure uses identified in round 1 (using a 1–9 scale, where 1 = “strongly disagree” and 9 = “strongly agree”). Deidentified descriptive statistics were then generated and sent to participants for review prior to round 3. For round 3, workgroup members convened for a face-to-face meeting, moderated by the project leader. During the meeting, each participant received an anonymous summary of the rankings of all members of the group, and a separate report summarizing their own previous rankings. The meeting moderator used these data to guide discussion, focusing on areas with the greatest disagreement. No attempt was made to force the panel to consensus; instead, the discussion attempted to determine whether divergent ratings resulted from real disagreement, or simply reflected different understandings of the meaning of content areas/categories/attributes. After several minor revisions and clarifications, panelists anonymously rerated each item using the same scale described above, including whether quality measure attributes should be categorized as tier 1 or tier 2.

We analyzed data generated from round 3 using a prespecified strategy: items with median scores of ≥7 were included in the final ACR approval criteria for quality measures.

Results

Table 4 lists the clinical content areas that the workgroup considered for future measure development and approval. From the disease category, RA, osteoporosis, juvenile arthritis, gout, ankylosing spondylitis, psoriatic arthritis, and osteoarthritis received the highest rankings. These areas represent high-priority areas for the ACR in quality measurement and improvement initiatives moving forward.

Ordinal scale of 1–9, where 1 = “strongly disagree” and 9 = “strongly agree” that an area should be evaluated by the American College of Rheumatology (ACR).

Disease category

Rheumatoid arthritis

9

8–9

Osteoporosis

9

6–9

Juvenile arthritis

8

2–9

Gout

7

4–9

Ankylosing spondylitis

7

4–9

Psoriatic arthritis

7

4–9

Osteoarthritis

7

3–9

Systemic lupus erythematosus

6

3–9

Reactive arthritis

5

3–9

Myositis

5

2–9

Scleroderma

5

2–9

Septic arthritis

5

2–7

Low back pain

4

2–9

Fibromyalgia

4

1–9

Non-gout crystal-induced arthritis

3

2–9

Vasculitis

3

2–6

Regional musculoskeletal syndromes

3

1–7

Drug category

Biologic disease-modifying antirheumatic drugs

9

7–9

Nonbiologic disease-modifying antirheumatic drugs

9

7–9

Oral glucocorticoids (oral and parenteral)

8

5–9

Nonsteroidal antiinflammatory drugs

8

3–9

Intraarticular glucocorticoids

4

1–8

Arthritis-related comorbidities and prevention category

Functional status/disease activity assessment

8

6–9

Pain assessment

8

6–9

Vaccinations

8

5–9

Infection screening

7

4–9

Cardiovascular disease prevention

6

4–9

Cholesterol management

5

2–8

Cancer screening

4

2–8

Access and experience of care category

Access to arthritis/rheumatology health care

8

3–9

Experience of care (satisfaction, etc.)

7

3–9

Table 5 lists the mandatory and recommended attributes of tier 1– and tier 2–approved ACR quality measures. Six categories (development process and quality measure construction; origin and funding of quality measures; feasibility of measurement; ACR review; approval by outside organizations and reviews; and maintenance, modification, and future review by the ACR) representing 24 specific attributes were included based on the rankings of the workgroup. Workgroup members defined mandatory attributes (double check marks) and recommended attributes (single check marks) that should be fulfilled for a measure to be included in a particular tier. Key findings under the “development” category were that measures considered for approval should be developed through a rigorous process that considers the available scientific evidence and integrates this in a transparent group process involving experts in relevant fields, and that financial and other conflicts of interest must be carefully considered and fully disclosed.

ACR = American College of Rheumatology; √√ = a mandatory attribute for a particular tier; √ = a recommended attribute for a particular tier.

†

Indicator statements with appropriate descriptive formulation to identify numerators and denominators will be considered acceptable (e.g., clearly worded IF…THEN statements). Important denominator exclusions should be noted. Ideally, measures will be fully specified with analytic specifications appropriate for each relevant data source.

‡

This constitutes a preapproval review focused on the development process.

Individual quality measures from a quality measure set can be omitted from ACR approval by an ACR subcommittee§

√

√

Committee on Quality of Care can request a more rapid re-review

√

√

Regarding the origin and funding for the development of quality measures, the ACR will consider measures that originate from a variety of sources. Funding sources of all measures submitted for consideration require full conflict of interest disclosures from all members of the development group. Per the ACR's policy, measures supported solely by pharmaceutical or biotechnology funding will not be considered for approval. For tier 1 designation, the ACR must be involved in the development process (e.g., have representation on the development committees, review the development process and the measures prior to project completion).

A key finding under the “feasibility of measurement” category was that measures should have a low or reasonable administrative burden on field testing, ideally flowing from existing data streams such as clinical registries or electronic health records.

Regarding the ACR's review of measures for approval, a tier 1 designation of measures will require formal input from the wider community of ACR members, either through qualitative surveys or through a publicized formal comment period. Furthermore, tier 1 designation will require a formal review by relevant ACR committees and leadership.

The workgroup recommended that all measures that achieve ACR approval should be published in the peer-reviewed scientific literature. Measures that achieve endorsement by national organizations such as the NQF will be considered for tier 1 designation. Regarding maintenance of measures, the workgroup recommended that all ACR-approved measures should be reviewed and updated at least every 3 years.

Table 6 lists potential uses of ACR-approved quality measures the workgroup considered. The workgroup recommended that all ACR-approved quality measures should be posted on the ACR web site. Approved measures, particularly tier 1 measures, should be considered for inclusion in projects such as the ACR's RCR. The RCR is currently configured as a tool that greatly simplifies reporting to quality improvement initiatives, such as the CMS PQRI and the American Board of Internal Medicine's Practice Improvement Modules for recertification. Tier 1 measures also can be utilized by groups such as the CMS or private health care insurers, either for quality improvement or for accountability. In contrast, tier 2 measures are intended primarily for quality improvement programs, either within health care practices or within health care organizations. The workgroup recommended that tier 2 measures would become eligible for tier 1 status after rigorous field testing to determine various measure attributes and after substantial feedback from ACR members was obtained.

Table 6. Uses of tier 1 and tier 2 quality measures after American College of Rheumatology (ACR) approval

Tier 1

Tier 2

For quality improvement and accountability

√

–

For quality improvement only

–

√

Post on ACR web site

√

√

Consider for inclusion in the Rheumatology Clinical Registry

√

–

Support use in the Physician Quality Reporting Initiative, if appropriate

√

–

Consider for inclusion in ACR practice improvement modules

√

–

Discussion

Rheumatology and other fields of medicine will be affected increasingly by the quality movement. The quality movement has the potential to dramatically increase rheumatologists' ability to conduct practice improvement activities, leading to better care for our patients. At the same time, incentives designed to recognize the documentation of quality care will become more prevalent, and payers' strategies are projected to evolve from modest rewards (carrot approach) to financial penalties (stick approach).

This White Paper defines the essential attributes of quality measures that are used to evaluate the complex clinical care provided by rheumatologists. It provides the framework upon which ACR measure review procedures will be developed (and posted to the ACR web site). It defines uses of quality measures based on their attributes to ensure that measures are applied and promulgated in an appropriate manner. It also informs interactions between the ACR and various groups in the quality field, including third-party payers.

The ACR is helping rheumatologists define and document quality of care through the use of the RCR and other existing tools for quality collection and reporting. These tools also support process redesign for rheumatology practices. This White Paper proposes the foundation upon which the ACR might distinguish members who participate in quality improvement activities and adhere to high-quality standards. A voluntary ACR quality recognition program that is recognized by third-party payers and other relevant stakeholders might facilitate practice improvement and reduce the administrative burden presently imposed on rheumatologists.

With a proactive approach, the ACR will shape future dialog on this topic, rather than being subject to policy decisions imposed by others. Finally, this article defines strategies by which quality measurement will lead to meaningful improvements in the care our patients receive.

AUTHOR CONTRIBUTIONS

All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be submitted for publication. Dr. Saag had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.