Share

How can we know if the performance of the health system overall in the U.S. is good and if it is getting better or worse over time? Can a meaningful and reasonably understandable set of national quality measures be identified, accepted by key constituencies, and tracked in ways that allow system-level performance to have a more prominent and evidence-based part of policy discussions and decisions about healthcare?

There are several reasons that we should care about healthcare quality. One is that healthcare is more vital to people than most other goods or services, and we have a strong collective interest in assuring that the healthcare system works as well as it can. The consequences of poor quality can be dire. Another reason is that we, individually and collectively, spend a lot on healthcare (and health insurance to mitigate the cost uncertainty), and these costs have risen much more rapidly over time than those of other sectors of the economy. Financing the rising cost of healthcare is challenging, requiring difficult tradeoffs for families and governments, and those paying the bills quite legitimately want to know if they are getting good value for their money. Healthcare also is extremely complex and specialized, so most of us do not know what to expect when we need healthcare or whether we were treated appropriately or optimally. Objective metrics about healthcare performance can assist individuals with their own healthcare decisions, provide context for state and national policy discussions about healthcare programs and investments, and point to where and how the system can be improved.

More well-understood is the cost side. There is fairly broad agreement on how to measure the financial resources spent on healthcare at the national level, and consistent measurements are routinely reported by governments, both in the U.S. and in many other nations, as part of efforts to measure economic output. The data are not perfect, but we have a very good sense of how much we are spending in aggregate for hospitals, physicians, prescriptions, and other services; who is spending it; and how it all is changing overtime. And the statistics clearly show that the U.S. spends far more on healthcare than any other country per person and as a share of the economy.

The situation is much less clear when it comes to measuring the effectiveness and quality of the healthcare people receive. The healthcare system is vast, with many products — successful diagnoses and treatments, pain management, and delivery of preventive services — but there is no clear way to tally up the gains. Scores of provider quality measures have been developed by accreditation organizations, regulators, payers, and healthcare providers themselves to measure specific areas of practice and performance. Indeed, several recent reports (by the Institute of Medicine and Bipartisan Policy Center, among others) addressing quality measurement have emphasized the burdens associated with reporting a large number of current measures and the sometimes inconsistent requirements for similar measures. A recent Kaiser Family Foundation/Commonwealth Fund survey found that half of primary care physicians say the proliferation of quality measures to assess their performance has had a negative effect on quality of care.

The focused indicators that have been developed to assess treatment of specific diseases or conditions are useful for encouraging the improvement of practice at the provider level. But, each are too narrow to tell us very much about how the health system overall influences the level of health in the population, even assuming that they were consistently reported and available to be analyzed (which they are not). Measures now used by payers – which often focus on health delivery processes believed to influence health and are typically used to compare quality across providers – may change over time as new quality improvement ideas come into vogue or older ideas get implemented by the vast majority of providers, which complicates their use as metrics for trends in quality over time.

There are also metrics for broader measures of population health, including self-reported survey data about disease, disability, health status, service use, and access to care. These are, on the one hand, appealing in that they relate to outcomes people care about. On the other hand, it is difficult to tie these outcomes directly to the health care people receive versus broader social determinants.

Developing meaningful measures of overall health system quality, and how it is changing, requires a combination of indicators that can reliably show how the system – which in our view includes providers, payers, and public health – is influencing the health of the population. Given the scale of the healthcare sector, and the numerous different quality measures from numerous different sources, settling on a set of metrics that can credibly represent health system quality – and can be measured through existing records or reasonable data collection efforts — is a daunting task.

In this brief, we describe some examples of approaches to measuring quality and discuss the issues associated with quality measurement at the system level. We also provide trend and comparative statistics for key measures, where data are currently available. Although the available data are incomplete and sometimes flawed, on the whole they can be taken to suggest that quality of healthcare in the U.S. is improving, but falling short relative to other large and wealthy countries.

What types of data are needed to measure healthcare quality at the system level?

In assessing the quality of the health system, we are naturally most interested in health as the ultimate outcome. However, how healthy people are and whether they are improving or declining over time can be affected by many factors other than healthcare, so using health as a measure of the quality of healthcare requires recognition of the other factors that contribute to well-being.

Mortality rates have fallen steadily in the U.S. over the past 30 years

In addition to looking at health outcomes, another way to measure healthcare system quality is to focus on what happens in clinical settings: Are patients getting the right care when they need it? By using such process measures, we can better understand the extent to which clinical care is following best practices believed to influence health outcomes. Process measures are also more actionable than outcomes measures in that they allow for a more direct assessment of adherence to clinical guidelines. However, it is often difficult to know whether a given process measure is accurately representing general improvements or declines in clinical practices. Additionally, the key data sources for knowing what services patients receive and their health outcomes following this care (i.e., medical records), are not captured in a standardized way and are not consistently available across providers or payers.

To compensate for these data constraints, we look to other, sometimes imperfect data sources that can provide information about healthcare quality. Public health records, such as death certificates, can be used to assess whether the toll of specific diseases is worsening or abating, and also to understand where best to target medical interventions. Some issues arise when using this data to assess healthcare quality. One is that these data are heavily influenced by socioeconomic and other factors outside of the health system itself. Additionally, there is significant lag time between making an improvement in medical care and seeing a measurable change in death rates. This makes drawing connections between specific health programs and their outcomes challenging.

Population surveys can provide more timely results. These surveys can be used to estimate a variety of healthcare quality topics, such as how often people receive recommended treatments, and may be particularly useful for assessing preventive services or the responsiveness of clinical staff from a patient’s perspective. However, there are limitations to measures based on patient surveys. The questions in these surveys sometimes change over time and make it difficult to analyze changes that have occurred in the population. These surveys are also susceptible to patient bias and health literacy levels.

Claims data, when available, provide information such as admissions for specific conditions, treatment received, and readmission rates. These data are useful, but are limited to what is available: The Centers for Medicare and Medicaid Services publish reports using claims data, but it can be difficult to acquire representative information from private payers.

Collections of clinical records, such as disease registries and The Healthcare Cost and Utilization Project (HCUP), have similar benefits and downsides as administrative data. These records come from administrative data that are initially gathered for non-research purposes. They can be used to assess patterns of care, such as how often patients receive recommended care or are hospitalized for conditions that should not require it. There are often data lag issues and incomplete or inaccurate accounts of healthcare conditions and medications.

Examples of current approaches to measuring system-level quality

There are a number of examples of reports or data collections that aggregate quality measures at the national or state levels. One of the most influential organizations in this arena is the National Quality Forum (NQF), which convenes groups of experts and stakeholders who come to agreement over and endorse new quality measures for healthcare providers. Measures that receive NQF endorsement are used by numerous private payers, as well as the federal government, to assess quality of hospitals and physicians. The group also makes recommendations for improving and standardizing health quality data. Additionally, the Commonwealth Fund analyzes national, state, and local quality data. Commonwealth often conducts original surveys to supplement existing sources, which is particularly useful for understanding how the U.S. compares internationally.

The OECD Health Care Quality Indicators project coordinates efforts of national and international bodies working to develop and report quality measures at the health system level. A framework was released identifying effectiveness, safety, responsiveness, accessibility, equity, and efficiency as domains of healthcare quality that were already being measured or found within quality frameworks of member countries. In 2010, the OECD released “Improving Value in Health Care”. This document highlights the importance of improving the coordination of care, disease prevention, patient-centered care, use of health technology assessment and clinical evaluations, pay-for-performance, and quality led governance.

The 2014 AHRQ report includes information on over 250 quality measures from over 45 data sources, selected with guidance from a working group of representatives from various HHS agencies. The report focuses on a subset of core measures that are reported each time data is available and information from other “supplemental” measures is used to complement the core measures. Presentation of the measures is divided by topic, and each measure within each topic is explained and summarized with trend data, which often is broken out by key characteristics such as age, rate, location, or type of payer. A “Highlights” chapter summarizes the information from the report, including the number of measures that are improving, worsening or staying the same, which measures are changing, and how different states rank across a subset of measures. The report also notes some important areas where specific quality measures are not currently tracked (e.g., diagnostic errors made in ambulatory care and appropriateness of therapeutic choices for angina patients).

A different approach is recommended in a recent report by the Institute of Medicine, Vital Signs: Core Metrics for Health and Health Care Progress. The study committee reviewed current measurement efforts and found them to be fragmented, lacking in focus, and burdensome for providers and others. Current quality improvement efforts were found to have too many measures, many of which are “imperfect, too numerous, and uncoordinated.” The report suggests starting with the priority needs of the healthcare system, recommending a core set of measures that could be used to assess progress towards meeting those needs.

The committee identified 15 core measure foci across four areas (or domains of influence): healthy people, quality care, care cost, engaged people. To start the process, the report selected the best current measure for each core measure foci. An additional 39 priority measures are also selected to supplement and provide texture to the core set. The identified best current measures are intended to be starting places in most instances. For example, the best current measure identified in the area of evidence-based care is the preventable hospitalization rate, but the report notes that a composite index of the proportion of care based on evidence could eventually be developed. Working with stakeholders, the report suggests that the Secretary of Health and Human Services would lead the effort to implement the core measures and improve them over time.

The focus on core measures is intended to guide measurement efforts at all levels, and would require changes in how information is captured and reported throughout the healthcare system, including claims systems, medical records, administrative records, and surveys. To facilitate data standardization and measure improvement, the Secretary would work with stakeholders to develop guidance on how to best use the current best measures, move forward the process of standardizing the current best measures, and coordinating a stakeholder process to develop and ultimately implement more advanced measures (called horizon measures) for each core measure focus.

Important for this discussion, the IOM report looks beyond what might be considered the healthcare system and includes measures that more broadly address societal factors related to health. These measures include rates of high school graduation, addiction, health literacy, and community support. While healthcare providers or plans may be able to influence these results through interactions with their patients or enrollees, the outcomes associated with these measures are also influenced by many factors outside of healthcare delivery or payment. The IOM report advocates shared responsibility for outcomes among the health system’s traditional players like doctors and insurers, as well as schools and other institutions that can shape health outcomes. Understanding these determinants can help explain changes and variation in health system performance over time and across demographic groups.

What measures are already available and what do they suggest about health system quality?

Although there is no established framework for evaluating the healthcare system, a number of different metrics are used to look at health outcomes, quality of care, and access to services. Inconsistent or unavailable data and imperfect metrics make it difficult to firmly judge system-wide health quality in the U.S., but a review of the data we do have suggests that the system is improving across each of these dimensions, though it continues to lag behind comparably wealthy and sizable countries in many respects.

Trends in quality in the U.S.

The following tables summarize where the U.S. health system has shown improvement and where it has shown decline. (Full results and sources are shown in the appendix and chart collection.)

Table 1: Trends in quality of care in the U.S.

Quality measures that have improved over time

Quality measures that are similar over time

Quality measures that have worsened over time

Overall mortality rate

Adults age 50+ who received recommended screening and preventive care

Health Related Quality of Life (self-reported healthy days and days in which activities were interrupted by poor health)

Population Health

Measures of health outcomes can give us the most concrete sense of whether health and general well-being are improving, which could be due in part to improvements in the performance of the health system, but also to advancements in science or improvements in socioeconomic determinants of health. Outcomes measures can generally be grouped into those that assess longevity and those that assess the quality of life lived.

Longevity & Mortality

Both overall mortality rates and potential years of life lost (a measure of premature deaths) have dropped steadily in the U.S. from 1980 to 2010. Mortality rates have also dropped for many of the leading causes of death in the U.S., including certain cancers and circulatory system diseases (heart attack and stroke). Infant mortality has also declined somewhat in recent years, particularly among non-Hispanic blacks.

It is very difficult, however, to tie these changes in mortality and well-being to the healthcare system itself, separate from changes in behavior and medical science. A closer link can be found by examining deaths that conceivably could have been prevented through appropriate medical interventions — known as deaths amenable to healthcare. While the healthcare system might not be expected to prevent death in all of these instances, differences in mortality for these conditions provides information about how effectively care is being delivered. The mortality rate for deaths amenable to healthcare in the US has steadily declined, from 96 per 100,000 in 2004-2005 to 86 per 100,000 in 2009-2010.

Health Status & Quality of Life

Broader measures of the burden of disease have also improved somewhat: age-standardized Disability-Adjusted Life Years (DALYs) in the U.S. dropped 14% from 1990 to 2010. As a measure, DALYs account for both poor health and premature death. Breaking down the measure into its components shows that this improvement in DALYs has largely come from improvements in premature mortality, as opposed to improvements in years lived with disability or poor health (which has held steady after accounting for aging of the population).

At the same time, self-reported measures of health have worsened slightly. The share of adults reporting fair or poor health increased from 13% in 1993 to 18% in 2013, and there was a sharp decline in the percent of adults reporting excellent health over this period (from 25% down to 19%). More adults reported having at least one day of poor physical health in 2013 than did in 1993, and of those who did, the average number of days of poor health increased from 9 to 11 days over the time period. Part of this worsening of self-reported health status could be explained by the population aging, though the changes are still significant after adjusting for age.

The share of adults who are overweight or obese has been rising over time, though there are signs of improvement with the first decrease recorded in the 2011 to 2012 time period.

Quality of Care Provided

With the possible exception of mortality amenable to healthcare, each of the mortality and morbidity measures mentioned above are significantly affected by many factors outside of the health system, making it difficult to connect improvements or declines in any of them directly to the functioning of the health system. Therefore, in evaluating healthcare system performance we can also look to short-term outcomes following treatment or diagnosis and “process” measures aimed at assessing whether the patient received the right care at the right time.

Short-Term Outcomes Following Treatment or Diagnosis

One metric that relates short-term outcomes to health system performance is mortality within 30 days of a hospital admission. Deaths shortly after receiving care are not entirely preventable, but reductions may point to progress in treatment and care delivery. The U.S. has seen improvement in 30-day mortality for heart attack, stroke and pneumonia.

Other short-term outcomes point to whether diseases are being properly managed or prevented. For example, the rate of lower extremity amputations caused by diabetes decreased substantially from 37.5 per 100,000 population in 2006 to 17.1 per 100,000 population in 2010.

Process Measures (“Right Care, Right Time”)

With few metrics that can tie outcomes directly to the performance of the health system, quality can be assessed by examining the activities of clinicians and institutions that are believed to lead to better outcomes. Such measures generally show improvement in recent years.

AHRQ has constructed a metric of how often patients receive recommended care across a range of services. Based on that measure, the proportion of patients receiving recommended care increased from 66% in 2005 to 70% in 2010, though most of that improvement occurred between 2005 and 2007.

Screening and preventive care can lead to early diagnosis or prevention of disease, thus improving health outcomes. Measures of preventive services include the number of people screened for specific diseases, the number of children receiving recommended vaccinations, and the number of people receiving instructional or educational information about disease management and prevention. Only 42% of adults ages 50 and older received the recommended screening and preventive care in 2012, similar to rates in recent years.

A care metric that has received particular attention recently is how often patients have to return to the hospital shortly after treatment, which can indicate inadequate transitional care. Medicare 30-day hospital readmissions dropped from 18.4% in 2007 to 16.9 % in 2013 as financial penalties for hospitals with high readmission rates were implemented. However, CMS’s May 2014 analysis of its program aimed at reducing readmissions for high-risk Medicare beneficiaries found that only eight percent of participating hospitals showed significant reductions in the ratio of readmissions to discharges, suggesting that improvement may not be widespread. One factor potentially influencing improvements in readmission rates is an increase in the percentage of adults who received instructions for a successful recovery at home prior to discharge – from 79% in 2007 to 86% in 2013.

Patient Safety

How safe the delivery of healthcare is can be an important measure of quality as well. All adverse events associated with patient care do not necessarily reflect a health system error, negligence, or poor quality of care, but some do. Recent efforts to reduce preventable adverse events have seen some progress. The number of hospital-acquired conditions – including adverse drug events, pressure ulcers, and infections – declined somewhat from 2010 to 2013. The prevalence of retained surgical item or un-retrieved device fragments has also decreased 14% (from 8.1 per 100,000 hospital discharges in 2006 to 7.0 in 2010).

Access to Care

The cost of care and disparities in the U.S. may impact access to care, and in turn affect outcomes. The percent of adults reporting cost-related access barriers increased from 7% in 2000 to 11% in 2009 as the great recession hit, and then decreased to 9% in 2013 as the economy improved.

The economic downturn particularly affected people in worse health, those with lower incomes, and the uninsured. Among people whose self-reported health status was fair or poor, 22% reported delaying or forgoing medical care due to cost in 2013, but had recently peaked at 25% in 2009. Similarly, poor and low-income people (those with incomes below 200% of poverty) were more likely to report cost-related access barriers during the recession (increasing from 13% in 1998 to 17% in 2009, and then declining to 14% in 2013). The uninsured are most likely to report cost barriers, with 31% delaying or forgoing care in 2013 – a much higher rate than in 1998, when 24% of the uninsured reported cost-related access barriers.

How healthcare quality in the U.S. compares to other countries

Benchmarking U.S. quality measures against those of similarly large and wealthy countries is one way to assess how successful the U.S. has been at improving care for its population, and to learn from systems that often produce better outcomes. The OECD has compiled data on dozens of outcomes and process measures. Across a number of these measures, the U.S. lags behind similarly wealthy countries. In some cases, such as the rates of all-cause mortality, premature death, death amenable to healthcare, and disease burden, the U.S. is also not improving as quickly as other countries, which means the gap is growing.

Table 2: How healthcare quality in the U.S. compares to other countries

Quality measures in which the U.S. outperforms comparable countries

Quality measures in which the U.S. performs similarly to comparable countries

Both the U.S. and comparable countries (which we define as OECD nations with both above median GDP and GDP per capita in at least one of the past 10 years) have made dramatic progress in lowering mortality from some of the leading cause of death. With the exception of cancers, though, the U.S. has higher than average mortality rates for the major causes of death. The U.S. also has higher rates of premature death (potential years of life lost) for all leading causes of death than comparable countries on average.

Population Health

Similarly, in both the U.S. and comparable countries there has been a reduction in disease burden, as measured by DALYs. While the U.S. has improved its age-adjusted DALY rate 14% since 1990, comparable countries have seen average improvement of 18%. For both the U.S. and comparable countries, a large part of this reduction can be attributed to improvements in premature death relating to circulatory conditions like heart attacks. Although the U.S. improved its age-adjusted DALY rate for circulatory conditions by 36% over this time period, comparable countries improved 44% on average.

Quality of Care Provided

Relative to comparable countries, the U.S. has better short-term outcomes following treatment for certain conditions, like 30-day mortality post-heart attack and post-ischemic stroke. And five-year survival rates for specific cancers (like breast and colon cancer) are higher in the U.S. Survival rate is a controversial measure, however, as it can vary due to the timing of diagnosis, and not necessarily due to longevity. Survival rates for cervical cancer are lower in the U.S. than in comparable countries, on average.

The U.S. generally lags behind comparable countries in prevention and other measures of quality. The U.S. has relatively higher hospital admission rates for many preventable conditions, including congestive heart failure, asthma, and diabetes complications. And the U.S. has higher rates of medical, medication, and lab errors than comparable countries.

U.S. performance notably lags on a range of surgical process quality measures — on average, more surgical items are retained or un-retrieved in the U.S., post-operative outcomes such as blood clots are more common, and obstetric trauma during vaginal delivery is also higher.

Access to Care

The OECD collects data on cost-related access barriers to care, but the U.S. does not report these statistics. The Commonwealth Fund’s international surveys, however, find that in 2013 the U.S. had by far the highest rates of cost-related access problems among comparable countries, with 37% of Americans reporting barriers, in contrast to comparable countries that ranged from 4% in the UK to 22% in the Netherlands.

Commonwealth’s surveys also suggest that people in the U.S. have longer wait times to be seen by a doctor or nurse when they are sick than other comparable countries, with the exception of Canada (as of 2013). Canada and the U.S. also have the highest rates of emergency room use for non-emergencies, among comparable countries. However, adults in the U.S. experience shorter wait times for specialist appointments than those in most comparable countries.

Discussion

Reviewing current measures of healthcare system quality suggests that the quality of healthcare in the U.S. is improving in many areas, though it still lags behind comparable countries on a number of key measures, and the gap in health outcomes seems to be growing.

Existing outcomes, process, and access measures, many of which are compiled in AHRQ’s National Quality Report, can tell us a lot about the quality of the healthcare system in the U.S. With additional resources and more participation from outside stakeholders, this effort could no doubt be improved. However, there are reasons to believe that many of the current quality measures are too flawed to form the basis of a set of national quality indicators. In particular, consistent and compatible information about what happens in clinical settings is poor, limiting the ability to measure the extent to which care is evidence-based.

Establishing a new set of national healthcare system quality measures that can be presented in a consistent manner over time would permit more definitive assessments about the status and trends in healthcare system quality and could be used to bring healthcare quality to the forefront of policy discussions and decisions. Better and more systematic quality measurement could help to identify those areas where investments of resources could yield the greatest improvements in health. This may be particularly important amid growing interest in tying payments for services to quality and outcomes.

Envisioning a new, national set of health system quality measures requires being clear about the purpose, which has implications for the breadth of information that needs to be collected.

On the one hand, if the primary goal is to focus the attention of the public and policymakers on key health care quality issues and trends, a smaller set of focused indicators may be sufficient to meet that need. A small set of core indicators may be easier for policymakers and the public to understand, and could help motivate change by focusing attention on aspects of the system that are most important.

A larger set of national indicators, on the other hand, would support more granular analysis of where and how the system can be improved, and in that sense would be more actionable. In particular, a larger set of measures allows one to better match investment (costs) with return (quality) within specific parts of the health care system.

It may be that the right balance is a broader set of indicators to diagnose where quality is deficient and motivate efforts to improve care, with a focused set of metrics to communicate more broadly to the public and policymakers how the healthcare system is doing overall. This would require constant vigilance, however, to ensure that the reporting burden on providers does not grow too large.

There are many organizations, public and private, with a role in designing, reviewing, or collecting quality measures, and many other stakeholders who contribute to or use them. These current stakeholders would need to be heavily involved with the selection and implementation of a set of national measures for both practical purposes and provide legitimacy to the effort.

What is less clear is whether the process would need to be led by a government agency or whether it could be largely private. The recent IOM report recommended that the Secretary of Health and Human Services play a lead role in shepherding the process to create core measures, in part because the Secretary oversees large public programs and incorporating the core measures into them could jump start the process. Other reasons to give a prominent role to a government agency include the added visibility and potential legitimacy that attach to “official” measures, as well as the potential for meaningful and stable funding. A government agency, if legally authorized, also could compel providers, plans and others to provide information in prescribed ways, which could lead to better standardization and ability to make international comparisons.

There are reasons to be concerned about placing too much authority over the process in a government agency, however. One is that official processes are less flexible than private ones. The products of government agencies also are more exposed to the political process (through appropriations and otherwise), leaving more opportunities for decisions to be re-litigated or delayed. Some stakeholders also may be concerned about placing too much authority to collect information into the hands of government, which currently has very little access to information associated with privately financed care.

Regardless of the approach taken, presenting results from national indicators will be challenging. There is likely to be improvement in some areas and but worsening in others, and the pace of change will likely vary across different indicators. It is relatively straightforward to report each year how much health care spending is growing. Even with an improved set of quality metrics – which would take years to develop – the story will always be more nuanced.

You Might Also Be Interested In

The Peterson-Kaiser Health System Tracker is launching a new Health System Dashboard that will track U.S. health system performance over time across four domains: health spending, quality of care, access and affordability, and health and wellbeing. The dashboard examines trends in the U.S. health system, compares the U.S. to a group of other large and […]