The preparation of this Resource Guide would have been impossible
without the cooperation of dozens of people who demonstrated their commitment
to ensuring the quality of care provided to people with disabilities. In
particular, project staff would like to acknowledge the contributions of a
group of advisors, whose names are listed below. The group's make-up mirrored
our intended audiences, and therefore included staff in State Medicaid agencies
and others who work with State Medicaid agencies; staff from managed care
organizations; health care providers; and families and other advocates for
people with disabilities. A subset of the group provided critical input at the
inception of the project by helping us think through the framework for the
document; the criteria that should be used in selecting measures; and how the
document might best be organized to be useful to different audiences. A second,
partially overlapping subset of the advisors actually reviewed the draft
document in detail and provided extensive feedback. Of course, none is
responsible for the final product, but their comments were both thoughtful and
helpful and led to significant changes, and we hope, improvements. One
organization, the Center for Health Care Strategies, went above and beyond the
call of duty not only by providing input from their staff, but also by
supporting Ruth Martin, a consultant to the Center. Ruth Martin made written
contributions to the Resource Guide on the important topic of the resources
needed to implement an effective performance measurement system.

We were also supported by many experts who had used, reviewed or
developed measures and measurement systems that could be considered relevant to
the health care needs of people with disabilities. These subject matter experts
provided copies of their own materials, referred us to others working in the
field, shared their own visions and caveats with us, and shaped our thinking
significantly. We are especially indebted to the organizations whose measures,
measurement systems and criteria sets are included in the Resource Guide. Not
only did they provide us with their materials, but they also helped us present
them accurately and completely.

Finally, we would like to acknowledge the continuous encouragement and
assistance of staff of the Office of Disability, Aging and Long-Term Care
Policy (DALTCP) in the Office of the Assistant Secretary for Planning and
Evaluation (ASPE). Mary Harahan, Deputy to the Deputy Assistant
Secretary/DALTCP, had the vision and the courage to pursue this project, the
type of which is a departure from "traditional" ASPE undertakings. Gavin
Kennedy, our Task Order Monitor, was consistently supportive, resourceful and
patient. In the final stages of our work, we were pleased to get additional
input and advice from Lisa Lang, from ASPE's Office of Health Policy; her
expertise in health care quality was especially valuable.

A. The Context: Meeting the Challenge of Serving People
with Disabilities in Managed Care

More and more people covered by Medicaid are enrolling in managed care
organizations (MCOs). Under voluntary Medicaid managed care, beneficiaries can
choose between an MCO and more traditional Medicaid arrangements, such as
fee-for-service (FFS) or the more structured primary care case management
(PCCM) program. Under mandatory Medicaid managed care, beneficiaries must
enroll in an MCO under contract with their State's Medicaid agency. Most State
Medicaid managed care programs began by enrolling people eligible for Aid to
Families with Dependent Children, or as it now called, Temporary Assistance for
Needy Families (TANF). However, several States have now begun to implement, or
seriously consider, the use of managed care for people covered by Medicaid
because they are eligible for Supplemental Security Income (SSI); all eligibles
in this category are by definition persons with a disability.

Several advantages have been attributed to Medicaid managed care.
Proponents believe it will:

enhance access to high quality health care providers who have
previously been unwilling to serve Medicaid eligibles;

do better at integrating and coordinating care over time and across
service delivery settings than the more decentralized FFS system;

avoid unnecessary and costly care;

reduce the inappropriate use of service sites, such as hospital
emergency rooms, that are often used by low-income people with limited access
to primary care physicians; and

focus more on the provision of preventive health services.

However, others have concerns about whether MCOs can and will meet the
health care needs of Medicaid eligibles. They worry, for example, that:

MCO networks will not include traditional providers of care to
Medicaid eligibles (e.g., public hospitals and clinics; community-based health
centers and clinics; and inner-city teaching hospitals) who know the special
health care needs and personal characteristics of Medicaid eligibles, and are
more able and willing to meet those needs; and that

the financial incentives of capitated payments will cause MCOs to
place barriers to access to needed health services.

These concerns become more significant when decisions are being made
about the use of MCOs to care for people with disabilities, many of whom need a
complex (and sometimes uncommon) mix of health and social services to maintain
health and functioning. One advantage of MCOs is that it is easier to measure,
and thus to improve, the performance of a given MCO than it has been to
measure, and especially to improve, the performance of a community's FFS
system. The expansion of managed care has been accompanied by increased
attention to performance measurement for health care delivery systems, and an
increased emphasis on the need to hold systems accountable for their
performance.1

While there is little evidence that Medicaid FFS as a system took very
good care of the population of persons with disabilities, it is clear that
specific health professionals, facilities and programs have provided excellent
and responsive care to particular patients in particular communities. For many,
the ultimate value of managed care will be determined by whether it works well
for people with complex and specialized health care problems and needs, the
kind of problems and need of many persons with disabilities. As more and more
people with disabilities enroll in managed care, it is critical, for us all, to
determine whether MCOs are performing well in meeting their needs.

Comprehensive systems for measuring the performance of health care
systems in caring for persons with disabilities do not yet exist. At the same
time, thousands of individual quality measures exist but it is often difficult
to discern which will be most reliable and relevant to measure MCO performance
in caring for people with disabilities. In these circumstances, it is easy to
respond either by (1) doing little or nothing to measure MCO performance or (2)
mounting costly efforts to measure hundreds of highly specific aspects of
quality that may fail to provide a coherent picture of performance. This
Resource Guide is designed to help those who want to begin to work toward a
comprehensive system, today, by using measures available right now that have a
clear relationship to domains of performance important to the care of persons
with disabilities.

B. Purpose of the Guide

This Resource Guide is designed to support efforts to measure and
improve Medicaid MCO performance for people with disabilities. It presents the
results of an extensive search for existing performance measures that can be
used for quality measurement and improvement by:

State Medicaid agencies;

managed care organizations themselves;

providers within MCOs; and

people who advocate for the health care needs of those with
disabilities.

The purpose of the Resource Guide is threefold:

To provide an overview of the issues to be considered in
measuring the performance of MCO in taking care of people with
disabilities;

To bring together in one volume key information about
measures that now exist, or are about to be made available, that can be
counted on to support efforts to measure and improve the care provided by MCOs
to people with disabilities; and

To make clear the areas where technically strong measures do not
exist, in order to encourage further measurement development in these
areas.

The Resource Guide is the result of a ten-month effort by the Center for
Health Outcomes Improvement Research2 at the George Washington University Medical Center.
This effort was supported by the Office of Disability, Aging and Long-Term Care
Policy of the Office of the Assistant Secretary for Policy and Evaluation
(ASPE) at the U.S. Department of Health and Human Services. ASPE commissioned
the Resource Guide in response to numerous requests from the field to
disseminate, sooner rather than later, the best available measures of
performance in caring for people with disabilities.

Two important features of the project must be noted. First, the project
was not designed to develop new measures. Rather, it identified existing
measures, or measures that are about to be released. The time and resources
available were not sufficient for the development of new measures. It is
relatively easy to identify a dimension of performance for MCOs, and even to
reach consensus across stakeholders that this dimension is significant to
meeting the needs of people with disabilities. It is far more difficult, and
takes considerably longer, to develop a reliable and valid way of
measuring a dimension of performance. The project has helped to identify
the gaps in existing performance measures, i.e. the dimensions of performance
that many people think are critical but for which valid and usable measures are
not available. It can, therefore, serve as a guide to future measurement
development efforts.

A second feature of the project is that it did not focus on measures
that would be used to assess the care provided to people, such as the frail
elderly, who are in nursing homes or other residential long-term care
facilities. Again, given time and resource limitations, the project's goal was
to concentrate on care delivered to other populations of Medicaid eligibles
with disabilities who are being enrolled in MCOs.

An Example: Measuring
Coordination

Virtually everyone agrees that effective
care for people with disabilities requires the coordination of a wide and
complex range of medical and non-medical services. Coordination of care is
clearly a dimension of performance. It is quite difficult to coordinate care,
especially across multiple agencies and providers. It is at least as difficult
to gather pieces of information from multiple agencies and providers that, when
put together, will provide a clear picture of whether or not services are being
coordinated. The project found few reliable measures of coordination of
care.

C. How the Guide is Organized

Following this Introduction, the Resource Guide has four chapters. Chapter Two: Developing Systems to Measure the Performance of
Managed Care in Serving People With Disabilities, discusses issues that
should be considered in the selection and use of performance measures,
including:

the characteristics of people with disabilities;

the health care needs of people with disabilities;

what performance measurement is, who can use performance measurement
and how; and

what it takes to build an effective performance measurement system.

Chapter Three: Domains of Managed Care
Performance in Serving People with Disabilities, presents a framework for
categorizing the aspects of MCO performance considered important in providing
quality care to people with disabilities. This Chapter should be read by
everyone. The framework it presents performs two roles:

It presents the full range of topics that are important to address
with respect to serving people with disabilities.

It serves as an organizing tool for describing and categorizing
particular measures; the framework will be used for that purpose throughout the
Resource Guide.

In Chapter Four: Specific Measures to Assess the
Performance of Managed Care in Serving People with Disabilities, presents
measures and measurement systems that deserve serious consideration for
inclusion in a performance measurement system. We are not recommending that
anyone use ALL these measures. Rather, they are a good range from which to
select a sub-set for implementation. This Chapter includes:

a discussion of how we selected the measures;

a specification of the information that will be provided about each
measure or measurement system;

a matrix that indicates what domains are addressed in each measure or
measurement system in the Resource Guide;

a summary, for each domain, of the specific measures available; and

a detailed description of each measure or measurement system.

Measures vs. Measurement
Systems:

A measure typically addresses a very
specific characteristic. A measurement system is typically a collection of
measures that may be quite comprehensive in scope. For example, one measure of
the performance of MCOs in taking care of children is the rate of complete
immunization of children by the age of two. This specific measure is one of
many that are included in the best known managed care measurement system: the
Health Plan Employer Data and Information Set (HEDIS®) of the National Committee on Quality Assurance (NCQA).
The Resource Guide includes both measures and measurement
systems.

The Resource Guide does not include copies of actual measures, such as
complete surveys, or the full technical specification of measures. The
inclusion of such material would make the document large and cumbersome.
Rather, the Guide provides sufficient information on each measure to help
potential users decide whether further consideration of the measure is
warranted, as well as information about where and how to get actual
instruments, technical specifications, other documentation and in several cases
technical support.

Chapter Five: Criteria Sets for Assessing Managed
Care Organizations, presents material that may also be of use in examining
the performance of MCOs in serving people with disabilities. Several groups
have identified specific characteristics of MCOs that they think have important
consequences for the care of people with disabilities. These are criteria sets,
rather than formal technical measures. That is, no method has been specified
for collecting information to determine whether a characteristic is present or
absent. In some cases, determining whether a characteristic is present may be
relatively easy; in other cases, it may be quite difficult. We have included
three criteria sets in the Resource Guide because:

unlike most technical measures, they have been developed specifically
to address the concerns of people with disabilities;

they can often be used to examine MCOs prior to finalizing contracts;
and

they provide a starting point for the development of more formal
measures and data collection efforts.

Finally, a list of References and
Resources provides information on other resources and documents that
you might find useful in developing a performance measurement system.

A. The Population of People With Disabilities

To determine how best to assess the performance of MCOs in serving
Medicaid eligibles with disabilities, it is important to understand key
characteristics of this population. When the last U.S. Census was conducted in
1991-1992, there were 48.9 million Americans with a disability, or nearly 19.4
percent of the total U.S. population (McNeil, 1993). At that time,
approximately 19 percent of persons with disabilities aged 15 to 64 were
covered by Medicaid.

In 1995, according to the Health Care Financing Administration (HCFA),
approximately six million people with disabilities were covered by Medicaid, of
whom 1.3 million were children. Within Medicaid, those with SSI were the second
fastest growing eligibility group between 1990 and 1994, increasing at an
average annual rate of 10 percent (Davis and O'Brien). These authors also
report that as of 1995, the disabled "make up only 15 percent of all Medicaid
users, while their spending accounts for 39 percent of all program
payments."

The 1994 National Health Interview Survey (NHIS) included a supplement
designed to learn more about people with disabilities. Utilization of medical
care among the Medicaid SSI population was slightly higher than utilization in
the general Medicaid population, but similar to utilization for all people with
disabilities. The survey reveals that many children with disabilities are
eligible for Medicaid not under the SSI, but under TANF. This implies that
measuring the performance of MCOs in serving people with disabilities, and
especially children with disabilities, may be a significant concern even when a
Medicaid program does not enroll SSI-eligible persons in MCOs.

Analysis of the NHIS also reveals that estimates of the size of the
population with disabilities vary widely, depending upon the definition used.
Some respondents who would be classified as having a disability under certain
definitions (e.g., impairments, disease conditions, inability to perform
certain activities of daily living) do not self-identify as having a
disability. This may reflect differences in attitude between people with
disabilities and their health care providers. As Harahan noted in a discussion
of these results, people with disabilities "may demand control over their own
lives and maximum choice."

Key Characteristics of the
Population of People with Disabilities

the number of person with disabilities
who are covered by Medicaid is growing;

Medicaid-eligible persons with
disabilities may be covered under TANF as well as SSI, so that even when a
State enrolls only the TANF and not the SSI population in managed care, they
may still need to measure MCO performance in caring for people (especially
children) with disabilities;

on average, persons with disabilities
need and use more health services than other Medicaid eligibles;

however, the extent and nature of their
need for an use of health services varies widely;

many have a distinctly different
definition of their circumstances and needs than their health care providers;
and

many have strong preferences about their
care that they expect will be honored.

B. What Are the Health Care Needs of People With
Disabilities?

As already noted, the extent and nature of health services needed by
people with disabilities varies widely. This section presents the health care
needs of three subgroups: healthy people with disabilities; people with
disabilities who have ongoing but not particularly complex health conditions;
and people with disabilities who have complex and uncommon health conditions.
Through this discussion, we highlight the issues that must be addressed in
measuring the performance of MCOs in caring for a population of people with
disabilities.

Taking Care of Healthy People with Disabilities

People with disabilities need the same set of preventive and curative
health services as those without disabilities. Indeed, as in the general
population, many people with disabilities need regular preventive care but only
episodic curative care, since they are basically in good health.

In providing health services even to healthy people with disabilities,
however, delivery systems must identify, and take into account, the particular
challenges facing each person. For example, HCFA data cited by Davis and
O'Brien (1996) reveal that "Mental impairments, including mental illness,
mental retardation, and developmental disabilities, predominate among both the
adult and child SSI populations." People with these conditions comprise 67
percent of adult and 57 percent of child SSI recipients. When preventive and
curative medical care is being provided to this group, it must be tailored to
take into account their psychological, social and developmental needs and
problems. This implies two special needs even for the physically healthy person
with a mental or developmental disability:

the need for coordination of care provided by physical and behavioral
health care providers;

the need for health professionals who do not specialize in behavioral
health to understand how mental health and developmental problems can affect
the prevention, diagnosis, treatment and follow-up of other medical
conditions.

A wide variety of other diagnoses and conditions can result in a
disability that makes one eligible for SSI. Medicaid SSI eligibles also include
people who have sensory impairments; neurological problems; mild to very severe
limitations in their motor functioning; and respiratory limitations. As in the
case of those with mental impairments, the physical, psychological and other
challenges presented by their particular condition must be taken into
consideration in ensuring access to medical care and in ensuring care will be
appropriate to the individual. Access and appropriateness are concerns for
everyone. Issues that are often of special concern to persons with disabilities
include:

facilities and equipment that do not present physical and
architectural access barriers;

specialized transportation to care sites;

geographic proximity of health care facilities, with in-home care
needed in some cases;

alternative methods of communicating and interacting with enrollees
that are appropriate to those with sensory or linguistic impairments; and

active participation of the individual people with disabilities and
where appropriate his/her family or other caregivers in decisions about their
care.

Examples:

A system caring for a speech- or
hearing-impaired person must be equipped to communicate effectively with this
person, both on the telephone and in person. A woman of 53 who is quite healthy
but in a wheelchair needs an annual screening mammogram like all women of her
age. However, special equipment and perhaps special outreach efforts may be
needed to ensure that she gets screened.

Ongoing Health Care for Persons with Disabilities

Other persons with disabilities resemble (indeed may be) persons with
chronic medical conditions. That is, in addition to regular preventive services
and episodic curative care, they need specific services on a regular basis for
an ongoing condition. They (and often their family) must often participate
actively to make sure their condition is well managed. Since their condition
may never be "cured," the focus of attention shifts to their ability to live as
full and active a life as possible is maintained, and to avoid preventable
deterioration and complications of their condition. For this group of persons
with disabilities, the following needs become especially important:

access to a regular health care provider (whether a primary care
provider or a specialist) who understands their particular problem and how it
is likely to affect their life;

ability to maintain a relationship with this regular health care
provider over time;

careful tracking by the health care system of the individual's
condition, of their response to treatment, of changes in their life
circumstances and daily functioning; and of the emergence of other medical
problems which can affect, or be affected by, their ongoing condition;

access to patient education, nutrition, exercise and other services
that support effective management of the underlying condition;

access to services needed to prevent complications or deterioration
or to treat them promptly when they do occur;

access to health services that can maintain or enhance the person's
independence and improved functioning in daily life;

links to non-medical services (e.g., housing, education, employment
training and placement, transportation, etc.) that also support independence
and improved functioning in daily life;

active involvement of the person and where appropriate his/her family
in decisions about treatment alternatives and in the implementation of
treatment plans; and

support for family members who care for or are simply significantly
affected by the health and functioning of a person with an ongoing and
disabling condition.

Caring for People with Complex and Uncommon Needs

Compared to the general population, a higher proportion of people with
disabilities face quite serious and often uncommon medical problems, that must
be treated or managed effectively, and that can significantly influence the
treatment of other medical problems that occur over their lifespan. Their
health care needs are complex and specialized. Multiple providers may care for
an individual patient at any given point in time. The specific services
required may change over time. Given these circumstances, the following issues,
in addition to those already mentioned, become particularly important:

access to highly specialized providers and facilities experienced in
diagnosing and treating the individual's condition;

careful tracking of changing circumstances and needs as well as
relevant medical advances.

Most MCOs are not organized, at the outset, to take care of the full
range of complex and uncommon medical problems that may be faced by their
members. Their network may not include the required specialists. Their
formulary may not include the required medications. MCOs that agree to take
responsibility for a population of persons with disabilities need to expect the
unexpected, and be ready, willing and able to respond. The responsiveness
of MCOs to complex and uncommon medical problems is influenced by the following
factors:

the use of a reasonable definition of medical necessity that does not
exclude services essential to diagnosing and treating uncommon conditions;

the willingness and ability to add highly specialized providers and
facilities to their network in order to diagnose and treat uncommon conditions;

the willingness and ability to provide medications not included on
their formulary when essential to treating uncommon conditions or preventing
complications;

the availability of user-friendly, well-understood methods that
members can use to appeal denials of care and get prompt responses; and

the availability of care coordination services.

C. Performance Measurement: What is It? How Can it be
Used?

This section of the Resource Guide addresses how to think about and plan
for a system for measuring MCO performance in serving people with disabilities.
It begins by defining performance measurement, and then describes how
performance measurement can be used by different groups, and what resources are
needed to use it well. The section ends with a discussion of how to enhance,
over time, our efforts to measure and improve performance.

What is Performance Measurement?

In this Guide, performance measurement is defined as:

the process of using formal, scientifically grounded tools and
methods to collect information about a health care delivery system, such as an
MCO, to determine whether its characteristics and actions, and the consequences
of its actions, meet expectations.

Performance measurement is a key element in holding health care delivery
systems accountable for what they do. It is intended to inform decisions and
guide actions. It is not measurement simply for the sake of measurement, or to
conduct research.

As implied in our definition, performance measurement involves comparing
performance against a set of expectations about what an MCO can and should be
doing for its members. These expectations come in part from what research tells
us can and should be done to achieve desired outcomes. However, the performance
measures we choose, and in particular what we choose to measure, invariably
reflect, in addition, values and preferences.

This leads to a question: whose expectations, and whose values and
preferences, should drive the development of performance measures? The
development of effective performance measurement systems will require
participation from many different stakeholders: purchasers (such as State
Medicaid agencies); policy makers; MCOs; health and social service providers;
and consumers, including persons with disabilities and their families and other
informal caregivers. Each of these stakeholders is likely to have distinct
values, preferences and expectations. To develop a performance measurement
program, it is essential to develop sufficient consensus about expectations.
While it is difficult to reach complete agreement on expectations, it is
clearly possible to identify a set of important and widely shared
expectations.

How Can Performance Measurement be Used?

Performance measurement can be undertaken by different organizations,
for different purposes. We will discuss how measuring performance of MCOs in
serving people with disabilities might be used by four different and
significant constituencies: State Medicaid agencies; MCOs; people with
disabilities and those who care or advocate for them; and health care
providers.

State Medicaid agencies can use performance measurement to:

identify areas where improvement is needed in the performance of
particular MCOs, set goals for improvement and in some cases incorporate them
into contract specifications;

identify exemplary behavior in particular MCOs to highlight and
disseminate, or use as a benchmark for the performance of other plans;

provide useful and reliable comparative information to Medicaid
eligibles who are trying to choose among various MCOs they can join;

select an MCO for persons who do not explicitly choose one for
themselves;

decide whether to contract with a given MCO in the future;

determine if there should be a ceiling set for new enrollments in a
given MCO;

in extreme cases, introduce intermediate sanctions or even terminate
a contract with a plan; and

identify patterns of poor performance across multiple plans that may
signal the need for broader changes in policies, procedures or even
legislation.3

Managed care organizations can use performance measurement
to:

identify where they need to focus internal quality improvement
activities;

track their progress over time in making quality improvements;

make decisions about which provider groups or facilities it wants to
include in its network in the future;4 and

market themselves to potential enrollees.

People with disabilities, their caregivers and advocates can use
performance measurement to:

make more informed choices both between managed care and FFS systems
and among various MCOs;

become more capable of making their own choice, instead of being
arbitrarily assigned to an MCO by the State Medicaid agency;

identify and publicize significant problems either in a particular
MCO or across several MCOs, in a way that moves beyond individual
anecdotes;

identify and publicize positive findings to highlight what can be
achieved for people with disabilities, how it can be done, and who is doing
it.

decide which MCO networks they would like to join or remain a part
of.

The Goal of Performance
Measurement

Performance measurement is designed to
achieve a fundamental goal: to improve the overall performance of the health
care delivery system by encouraging and rewarding good performance and by
discouraging or eliminating poor performance. When MCOs know that their
performance is being measured, and that important decisions and actions will be
shaped by the results, they will pay more attention to whether their
performance meets expectations. A set of performance measures is like a test.
In education, people often complain when teachers "teach to the test." In
measuring MCO performance, however, the intention is quite explicit that MCOs
will shape their behavior so they do well on the test. This means that we must
make sure we have the right test. This Resource Guide is intended to help
people put together the right set of performance measures, i.e., the right
test.

Performance Measurement and Comparisons

Performance measurement always involves making comparisons. Here are
some comparisons that can be made using performance measurement:

comparing different MCOs to each other:

who is performing better at ensuring that patients are actively
involved in decisions about their care? who is not doing as well?

comparing each MCO's performance to an explicit standard:

do people get seen for urgent care within 12 hours? do at least 90
percent of children get all required immunizations by the age of two? are sign
language interpreters always available for regularly scheduled appointments
with hearing impaired persons?

comparing how an MCO performs at a certain point in time to how they
performed at another point in time:

is the MCO reducing how long patients wait in the office before they
are seen? or are patients actually waiting longer?

comparing the performance of MCOs in caring for people with
disabilities to their performance in caring for other members:

how do rates of mammography compare between women over 50 with and
without disabilities? how often do people with disabilities disenroll from a
given MCO because of quality concerns, as compared to disenrollment among those
without disabilities?

comparing the performance of MCOs, across the board, with the
performance of other delivery systems such as FFS or PCCM:

how do people with disabilities who join MCOs rate their access to
specialists, as compared to those who remain in FFS or PCCM?

Different kinds of comparisons reflect, sometimes implicitly and
sometimes explicitly, different expectations. For example, when MCOs are being
compared to each other, they are implicitly measuring their performance against
the "average" performance in the group. Saying that a particular MCO is "better
than average" may be misleading if everyone is performing poorly. Saying
performance is "just about average" may similarly be misleading is everyone is
performing very well.

On the other hand, comparing each MCO's performance to a standard makes
expectations very explicit. Standards, like measures, have to be developed.
They can be derived from:

what can actually be achieved because at least some pathbreaking
MCOs have achieved it (this is often called a benchmark):

for example, a small group of MCOs has found methods to ensure that
primary care physicians diagnose depression and make appropriate referrals
where needed for 80 percent of their patients

what research shows is essential, or critical, to achieve desired
outcomes:

for example, when 90 percent of a population has been immunized, the
population as a whole achieves "herd immunity" reducing to virtually nothing
the chance of disease transmission

what stakeholders agree is both achievable and morally right:

for example, a standard can be set that no one should face
physical barriers to access to health care facilities

Finally, when the performance of MCOs in serving people with
disabilities is compared to their performance in serving other members, there
is an implicit expectation that people with disabilities deserve, can, and
should get at least the same quality of care as everyone else.

D. Selecting Performance Measures

There are innumerable attributes of an MCO that can affect the health,
functioning, quality of life and satisfaction of the people it serves. There
are thousands of measures of the quality of medical services, most designed to
determine whether, for a patient with a particular diagnosis or condition, the
correct services were provided at the right time by the appropriate people and
achieved the desired consequences. But very few MCO attributes, and very few
condition-specific quality measures, are likely to work well as part of a
system of performance measurement. Why? For one thing, no one has the resources
to measure everything that could be measured. No one has the time or resources
to use all the information that would be generated by such a massive
measurement effort. Indeed, it is important to avoid spending so many resources
on performance measurement that it detracts from, rather than adds to, our
ability to provide good service to people with disabilities.

To build a performance measurement system, it is critical to select a
limited number of good measures that together provide a coherent picture of a
health care delivery system, rather than measure everything. As noted earlier,
the performance measures selected will get the attention of MCOs, so it is
important to select the right set. Here are some criteria that can be used to
select measures; they were also used to select the measures included in this
Resource Guide:

Criteria for Selecting
Performance Measures

The measure should tell you about
something that is considered of great significance.

The measure should address an aspect of
the performance that the MCO can significantly influence.

The measure should tell you something
that reflects the performance of an MCO as a system of care, rather than a very
narrow aspect of its functioning.

The measure should let you make the
comparisons you want to make.

The measure should address an event that
can be observed often enough to produce reliable results.

The measure should tell you about something that is considered of
great significance.

As noted above, performance measures reflect expectations and
values. Some things that an MCO does (or does not) do are far more significant
that others, depending on your expectations and values.

The measure should address an aspect of the performance that the
MCO can significantly influence.

Performance measurement is a tool to hold MCOs accountable. It is
neither sensible nor fair to hold an organization accountable for what it
cannot influence. Notice this criterion uses the term influence rather than
control. No organization can control anything completely. However, MCOs have
significant influence over many things they do not control.
Another way to put this is this: does the measure address something that is
actionable, something that the MCO can work to improve? does it give the
MCO clear direction about what it needs to work on?

Related to this criterion is the issue of what the MCO is
contractually required to provide to its members, and in particular to its
members with disabilities. Thus, for example, if a contract between a State
Medicaid agency and an MCO does not specify that a given service must be
provided to enrollees, it may not be appropriate to measure whether or not that
service is indeed provided.

The measure should tell you something that reflects the
performance of an MCO as a system of care, rather than a very narrow aspect of
its functioning.

This criterion is somewhat difficult to understand and apply. An
example may help. When health care delivery systems in different countries are
compared, one of the performance measures that is almost always used is the
infant mortality rate. This measure meets our first two criteria well:
increasing the number of newborns who survive to at least the age of one is
clearly important; and while the health care delivery system cannot control all
the factors that influence infant mortality, it has a significant influence on
many others. It also meets this third criteria, because reducing infant
mortality requires that many different parts of the health care system do a
good job. The infant mortality rate reflects whether women get prompt access to
pre-natal care; whether the care is consistent and effective; whether the most
up-to-date tests and treatments are being used; whether potentially high-risk
pregnancies are being identified early; and whether action is being taken to
reduce or address risks swiftly and effectively.

Another way to think about this criterion is in terms of whether a
particular action that an MCO can take would have a big effect on the entire
experience a member or a patient has in the MCO. For example, whether a new
member of an MCO is linked to a primary care provider quickly (say within 30
days) could be critical for their ability to use and benefit from the entire
system. Another example of special importance to people with disabilities is
whether the MCO has an efficient and timely process for bringing new, highly
specialized providers into their network when a new member enrolls who needs
such providers to maintain or improve health and functioning.

The measure should let you make the comparisons you want to
make.

As discussed above, performance measurement always involves
comparisons. Measures should be selected that are relevant to the comparisons
being made. Just as important, measures have to be selected that can be
implemented across all the organizations or systems being compared. In
particular, the data needed for the measure has to be available across all
these organizations and the measures need to be sufficiently specific to ensure
that data are collected exactly the same way.

The measure should address an event that can be observed often
enough to produce reliable results.

Most performance measures are rates; they measure how often, or for
what proportion of people, an event takes place. The denominator of a rate
indicates the maximum number of times, or the maximum number of people, for a
given event. For example, the denominator for a disenrollment rate would be the
number of people enrolled in a plan over a given time period. The numerator of
a rate indicates the actual period of time, or the actual number of people, for
a given event. Thus, the numerator for a disenrollment rate would be the number
of people in a plan who disenrolled (typically for specific reasons related to
their dissatisfaction with care) in that same time period. When the denominator
or the numerator are likely to be quite small, it becomes difficult, for
statistical reasons, to have confidence in the reliability of a rate. It also
becomes very difficult to make comparisons.

This "small number problem" makes it difficult to include as
performance measures events that are specific to a particular health problem or
condition, especially if that condition is not very common in the general
population, in the population of people with disabilities, or in the population
of people enrolled in a particular MCO or group of MCOs. For this reason, we
have emphasized, in the Resource Guide, measures that are applicable to larger
groups of people, rather than condition-specific measures.

E. What it Takes to Build a Performance Measurement
System

Measures are not the only resource needed to build, and to use well, a
system of performance measurement. In this part of the Resource Guide we will
discuss the other resources that need to be in place to build a system.

Planning and Designing a System of Performance
Measurement

As noted above, performance measurement requires that we clarify
expectations. Also as noted, many constituencies have an important stake in
what gets measured, how it is measured, how it gets reported, and to whom. For
this reason, it is highly desirable to design systems of performance
measurement with input from multiple stakeholders. In most but not all cases,
the lead in development of such systems will come from State Medicaid agencies.
The lead organization should convene all stakeholders, including other State
agencies who provide health and related social services to people with
disabilities. Experiences in several States indicates that the initial planning
and design process can take up to a year. The convenor needs to have skills in
planning, in facilitating group processes and in managing conflict, as well as
technical knowledge. Some may find it useful to contract with an external
consultant as a facilitator, since an outsider may appear more neutral than any
of the parties.

Resources Needed to Collect and Analyze Performance
Data

No matter how carefully measures are selected to assess MCO performance
in caring for people with disabilities, the system will fail unless accurate
and comparable data are collected in a systematic manner and unless it is
analyzed and interpreted correctly. Two key players, State Medicaid agencies
and MCOs, are most likely to be directly involved in data collection and
analysis. Two kinds of data are likely to be collected for performance
measures: primary data, collected for example through surveys of members; and
secondary data, collected from existing administrative and clinical records and
information systems.

State Medicaid Agency

Within State Medicaid agencies, a critical resource is people. One or
more dedicated staff members will be needed with knowledge of the
following:

performance and quality measurement;

the strengths and weaknesses of different approaches to data
collection;

basics of survey research and sampling;

contracting and coordinating with third-party vendors to conduct
surveys;

drawing survey samples using State and MCO records;

the structure, potential and limitations of State and MCO management
and clinical information systems;

methods for auditing and validating data provided by MCOs;

data tracking and management;

alternate approaches to the transformation of raw data into rates and
other findings;

methods for risk adjustment of performance and quality data;

comparative analysis of data; and

presentation of data to multiple audiences.

States need to decide how they want to receive performance information.
There are three basic choices:

MCOs submit the calculated performance measure;

MCOs submit the data required to calculate the numerator and
denominator of the performance measure and the State calculates the
measures;

for measures calculated from administrative data only, the MCOs
submit all claims or encounter data and the State calculates the measures.

The approach chosen has implications for the extent and nature of
computer hardware and software resources, and for how many and what types of
staff are needed to receive, clean, and load the data and calculate the
performance measures.

States must also choose how they will audit and validate data. The need
for validation increases as the MCO, rather than the State, takes
responsibility for data collection and the calculation of performance measures.
But much of the data will inevitably be based on MCO records. Unless the data
are audited, the State cannot assess the level of confidence that it should
have in the reported performance measures or correct the problems with the
performance measures. There are several ways in which the State can audit the
data:

using State staff;

using a contractor, such as the External Quality Review Organization
(EQRO); or

requiring that the MCOs hire independent auditors to certify their
performance measures.5

Managed Care Organizations

To participate effectively in a performance measurement system, MCOs
need three broad categories of resources:

staff with appropriate knowledge and analytic skills;

an appropriate management/medical information system; and

staff with appropriate skills to collect data from clinical records.

Specific staff expertise includes:

performance and quality measurement;

how to translate the definition of a performance measure into
detailed operational specifications for data collection;

an understanding of MCO program operations, including related quality
assurance and quality improvement activities;

and understanding of the potential and limitations of the MCOs'
management and clinical information systems (and the quality of information
systems at the provider level);

the history of data collection and information systems to identify
changes that would affect the reliability of trend data;

If MCOs choose to implement performance measures based on surveys, they
will also need either an in-house survey capacity or more likely the capacity
to select and work with an outside vendor.

As performance measurement and reporting have become more critical for
MCOs, the need for management information systems that can support these
efforts has grown. In an ideal world, all medical records would be automated
and MCOs would have direct access to clinical data that could be combined with
administrative data, such as enrollment records, to produce timely, accurate
performance measures. Unfortunately, current reality is far from ideal.
Therefore, clinical information is derived, to the extent possible, from claims
and/or encounter data. Information is further limited by the type of coding
used for office visits and procedures and the coding used for diagnosis. For
Medicaid members, enrollment data come directly from the State and are not
under the control of the MCO; these data are sometimes critical to specifying
the population for which a measure will be relevant. Therefore, collaboration
at the operational level will almost always be needed between the State
Medicaid agency and MCOs in order to generate many performance measures.

An MCO's ability to participate in an effective performance measurement
system depends on its having a management information system (and related
staff) that:

Collects and reconciles enrollment data and maintains an enrollment
history;

Collect and processes claims/encounter data in a timely and accurate
fashion and maintains a claims history;

Can integrate data from other sources, such as pharmacy claims data
from a third-party vendor; and

Has a flexible report-generator or can export data to a report
generation program in order to calculate the performance measures.

Many performance measures will require the use of clinical information
(that may well not be computerized) as well as administrative data (that is
more likely to be computerized).6
This means it is important to have staff with expertise and experience in the
abstraction of medical records. As important is a healthy relationship between
the MCO and its providers. Especially in more loosely organized and
decentralized provider networks, the MCO will need to work carefully with its
providers to ensure that data are reliably and consistently collected. In this
context, auditing and validation are even more important.

Level of Resources

The amount and type of data collection and analysis resources needed by
States and MCOs depend on the number and type of performance measures chosen,
the frequency of reporting, and the level of statistical confidence needed. In
general:

More frequent reporting requires more resources, unless data are
readily available from a computerized management information system.

Measures that require manual data collection (e.g., medical records
review) require more resources than measures that can be calculated directly
from computerized records.

When sampling is used to collect data to calculate measures (for
example with manual medical records review or surveys), higher statistical
confidence levels require more resources than lower confidence levels.

If different samples have to be drawn for multiple measures, more
resources are required. This can be somewhat mitigated when data for several
measures can be collected from a single medical record.

When oversampling or screening is used to ensure that a sufficient
number of people with disabilities are included, higher levels of resources are
needed.

Resources Needed for Disseminating Performance
Information

All too often, based on traditional practice, performance measurement is
viewed as of interest only to technical professionals. In fact, however, a
critical value of performance measurement is that it promotes both general
accountability to the public and the generation of information that can support
decisions and actions made by individual members of the public. Performance
data can only be used by those who have access to it. In planning a performance
measurement system, attention needs to be given to whether, how, when and to
whom performance data will be disseminated.

MCOs and State Medicaid agencies may find themselves uncomfortable at
the thought that they will not be the only people who see, interpret and use
performance information. It does "raise the stakes" both on the relevance and
quality of the data when wider dissemination is planned. In some ways, however,
this a good reason to pursue dissemination: the quality of data may rise when
people know that the public will see it.

The implication is not only that dissemination needs to be considered in
planning, but also that resources will be required to develop and implement the
dissemination strategy. The resources include staff or third-party consultants
and vendors with skills and experience in:

communication, including writing and graphic design;

the use of formal and informal media and channels for reaching target
audiences; and

tailoring the presentation of data to the characteristics and
interests of the audience, including their literacy levels, cognitive and
sensory impairments, language preferences, and trusted information channels.

People with disabilities, their caregivers and advocates are not just
another audience in this context; they understand best how they can be reached
effectively and it is wise to involve them from the outset in planning
dissemination efforts.

Enhancing Efforts to Measure and Improve
Performance

At the outset, State Medicaid agencies and MCOs will not have all the
resources needed to implement an ideal performance measurement system,
including in particular:

skilled and knowledgeable staff;

effective systems for collecting and analyzing data;

methods for disseminating the results of performance measurement; and

an accepted forum where various stakeholders can participate in the
process of planning and trouble-shooting.

Performance measurement systems will be "works in progress" for many
years. They will not be ideal and should not be expected to be ideal at the
outset. This is one area where the adage that "the perfect can be the enemy of
the good" is very applicable.

The development of performance measurement systems has to start
somewhere. We strongly recommend beginning with a limited set of measures that
are scientifically well-grounded, relevant to the concerns of stakeholders, and
feasible to implement given the current state of available data collection and
information systems. The planning process should be designed to help you
identify these measures. It will also serve to identify the more basic
structural barriers to using performance measurement. Often, simultaneous work
may be needed on structural barriers and the first stages of building a
performance measurement system.

Experience can be a harsh but excellent teacher. For this reason, we
also recommend learning about and staying in touch with others who are embarked
on similar efforts, so you can gain from their experience as well as your own.
The reverse is true: share your experiences with others.

The long-range development of performance measurement systems clearly
requires investment of resources. Acquiring and maintaining those resources
also means that the benefits of performance measurement be made clear to those
with influence over resource allocation. Ultimately, however, development of an
effective performance measurement system will require that senior officials in
State Medicaid agencies, MCOs, in the community of people with disabilities,
and among health care providers, make a commitment to using objective
information to assess and improve our systems of care.

This Chapter presents a framework of domains of managed care performance
that the project identified as critical to serving people with disabilities
well. These domains permit measures to be categorized into broad topics that
have been identified as important to serving the needs of people with
disabilities. These domains and the example measures we present in each were
based on:

interviews with numerous experts and stakeholders;

review of research on what people find important about their health
plans; and

The framework represents a synthesis, by project staff, of many sources
of information, including existing approaches to categorizing measures. For
example, quality measures were first categorized by Donabedian into three
types: structure, process and outcome. Many people are familiar with this
framework, but it does not address the actual content and substantive issues
addressed by measures. It is therefore less useful for those trying to think
about performance measurement and to select specific measures.

This framework of domains represents an ideal: it would be highly
desirable to find well-grounded and usable performance measures across all the
domains. However, our search for measures indicates that some domains are
thinly populated compared to others. Nevertheless, it is useful to have a
comprehensive set of issues and concerns to consider, even if measures do not
exist today. By starting with a comprehensive set of domains, it has been
possible to identify clearly the gaps in existing measures, thus providing
direction for future measurement development efforts.

The nine major domains in our framework are presented first. Details and
discussion of each domain follow.

Framework of Major Domains
of MCO Performance

Creating a system with the right
capacities.

Providing access to needed
services.

Supporting member involvement in
decision making and system improvement.

Resolving member problems and
concerns.

High quality interpersonal interactions
between members and providers.

Using preventive services to keep
members healthy and functioning.

Coordinating and integrating medical and
non-medical services.

Using state of the art
treatments.

Improving the outcomes of
care.

Detailed Discussion of Domains

A. Creating a System with the Right Capacities

As noted above, quality measurements have been distinguished in terms of
structure, process and outcome. The earliest quality measures emphasized
structure and were typically used to give a facility a license or to
accredit a program. This first domain of MCO performance also emphasizes
structure, but we use here the broader term capacity. One great
advantage of structural or capacity measures is that they can be used
prospectively, that is before people with disabilities are enrolled in a
particular MCO. They can also be used even after enrollment, to make sure that
capacities are in fact present and perhaps being enhanced. Here are examples of
indicators of the capacity of an MCO to take care of people with disabilities.
As with all the lists in this section of the Resource Guide, these examples are
not considered to be complete; you may well identify additional indicators and
measures, as we have.

The number, mix and qualifications of providers included in the MCO
network;

The stability of the MCO provider network;

The presence of procedures for adding new providers to meet
unexpected needs of members;

The presence of well-designed and active quality assurance and
quality improvement systems;

The presence of a system for assessing and tracking the health and
functional status of members (especially those with disabilities); and

The presence of a method for coordinating the care provided to
members (especially those with disabilities) with complex or exceptional
needs.

Note that in these examples, the emphasis is on whether the capacity is
in place. We view this as a necessary, but not sufficient, condition for high
performance. It is also important to determine whether these capacities are
actually used and whether or not they are effective. These issues are addressed
in other domains.

B. Providing Access to Needed Services

Access to care has long been a critical element of any assessment of a
health care delivery system. When access barriers exist, people have less
chance to get the services and care they need. Here are examples of indicators
of access to needed services that are of particular relevance to people with
disabilities; many are important to anyone.

Facilities, medical equipment and communication equipment accessible
to those with physical challenges, including visual and hearing impairments;

Availability of transportation to care sites;

Availability of convenient times of service;

Timeliness of access to routine and urgent care (e.g., waiting time
for appointments; waiting time after arrival);

Access to needed tests and treatments without extensive delays or
hassles waiting for prior approvals;

Access to needed prescription medications (including medications not
included in the MCOs formulary, although they are on the State's Medicaid
formulary);

Access to identified/selected primary care providers;

Access to specialists;

Access to services of special importance to persons with disabilities
(e.g., durable medical equipment and provision for the prompt repair of such
equipment; rehabilitation services; physical, occupational and speech therapy;
home health care (both skilled and unskilled); respite care); and

Availability of translation and interpretation services for
non-English speakers (including people with speech and hearing disabilities who
use American Sign Language).

C. Supporting Member Involvement in Decision Making and
System Improvement

Increasingly, Americans want to be more active in their interactions
with the health care system. People with disabilities have been pioneers in
pursuing a more active role and greater autonomy. In this domain, there are two
levels at which it is important for MCO members with disabilities to be
involved in decision making and system improvement. The first level is the
individual patient's interaction with the MCO and their health care providers.
The second level is the policy decisions that affect or guide a given MCO, or
the entire program of care for people with disabilities.

Involvement in Decisions and Improvements at the
Individual Level

Provision of understandable information on how to get needed services
and resolve member problems;

Provision of understandable information on methods for filing
complaints and grievances;

Provision of understandable information on the definition of medical
necessity used by the plan, on the criteria applied to determine medical
necessity, and on the process used in making medical necessity decisions,
including the process for appealing such decisions in a timely manner;

Involvement of members in treatment planning for management of
ongoing conditions;

Provision of information on all treatment options; involvement of
members in decisions regarding choice of treatments and specialty providers;
and

Support for the development and implementation of advanced
directives.

Involvement in Decisions and Improvements at the System
Level

Inclusion of members, family and other caregivers and advocates on
MCO governing and advisory boards;

Inclusion of members, family and other caregivers and advocates on
task forces and other groups organized by MCOs, State Medicaid agencies or
others, to plan or track the use of managed care for people with
disabilities;

Participation of members, caregivers and advocates in the planning of
quality improvement activities, including setting priorities on issues and
conditions that should be the focus of such efforts; and

Systematic use of methods to gather qualitative and quantitative
information from members, caregivers and advocates to identify their priorities
and their experiences using the system.

D. Resolving Member Problems and Concerns

Many, though not all, MCOs are relatively inexperienced in serving
people with disabilities. For this reason if no other, it is critical to
determine whether MCOs have put themselves in a position to learn quickly from
their experiences and to correct their mistakes. One place where this would
show is in how well they resolve the problems and concerns of particular
members. In addition, as already noted, some health care needs of people with
disabilities are rare or unexpected, and may only come to the surface in a
crisis situation. Here are examples of indicators of whether an MCO is
resolving member problems and concerns:

Timely, courteous and helpful responses from plan customer service to
member requests for information about the plan and how to use it;

Timely, courteous and helpful responses from plan customer service to
member requests to resolve questions and problems regarding access to
care;

Use of appropriate procedures to hear and address member appeals,
complaints and grievances (including rapid responses); and

Reductions over time in the rate of disenrollment for cause by
members with disabilities.

E. High Quality Interpersonal Interactions Between
Members and Providers

Extensive consumer research indicates that everyone values highly the
opportunity to have high quality interpersonal interactions with people in
health care systems. There is also evidence that good patient-provider
relationships have a positive effect on the management of long-term conditions
and on the outcomes of health care. This domain of performance is an example of
one that is important to everyone and especially important to and for people
with disabilities. Here are examples of indicators in this domain:

Proactive engagement of new members with primary care providers they
have selected from the plan network;

Providers who listen carefully to members and their caregivers;

Providers who explain things clearly to members and their caregivers;

Providers who treat members and their caregivers with courtesy and
respect;

Providers who spend enough time with members and their caregivers;
and

Medical office staff who are courteous, helpful and respectful of
members and their caregivers.

F. Using Preventive Services to Keep Members Healthy and
Functioning

This is another domain which is important to everyone and also very
important for people with disabilities. Equity considerations are of special
relevance here: it is critical that those with disabilities get the services we
know are effective in prevention or early identification of various diseases,
even though it may take special efforts to ensure they get these services at
least as often as people with no disabilities. This remark applies to our first
general indicator:

Delivery of age- and gender-appropriate preventive health
services.

Considerable work has been done to identify, across the age
spectrum, and by gender where relevant, specific primary and secondary
prevention services and interventions of known value. Many measurement systems
have identified sub-sets of these services of particular importance. However,
for people with disabilities, there is another kind of prevention that is
important, what public health people call tertiary prevention, i.e., the
prevention of unnecessary complications and unnecessary deterioration in
functioning for people who already have a defined condition. Our second general
indicator is relevant to this kind of prevention. Unfortunately, less work has
been done to specify and bring together services which are effective in this
kind of prevention, although some have certainly been identified and are
already in use as performance measures.

Delivery of services to maintain and enhance functioning.

G. Coordinating and Integrating Medical and Non-Medical
Services

Coordination of care has been consistently identified as critical for
people with disabilities. One of the greatest hopes many have about managed
care is that it will do a better job of coordinating and integrating care. The
examples below provide different perspectives on coordination that can all be
important to people with disabilities:

Continuity of primary care and specialty providers over time;

Ongoing coordination of medical services provided to a member by
multiple providers;

Coordination of physical and mental health/substance abuse service
providers;

Identification of non-medical services needed to maintain and enhance
member health, functioning and autonomy; referral and follow-up to support
delivery of such services;

Referral to plan- and community-based psycho-social support and
patient education services; and

Ongoing coordination with providers of critical non-medical services
that affect health and functioning.

H. Using State of the Art Treatments

This domain of performance comes closest to traditional quality
assurance and quality improvement efforts. The domain emphasizes process
measures, indicators that health care providers are using the right diagnostic
and treatment procedures, medications, etc., in caring for their patients. To
develop and select performance measures in this domain, it is essential to
focus on a limited number of specific conditions that are sufficiently common
to permit meaningful data collection and that have significant health
consequences. These conditions can be identified in two different manners. The
first is to build on the work already done by multi-stakeholder groups working
with experts (see for example the Sentinel Conditions for Medicaid MCOs
developed collaboratively by plans, State agencies and HCFA). Alternatively,
the set of conditions could also be identified through multi-stakeholder
interactions in a particular State. For these conditions, the strategy would
then be to identify indicators that MCOs:

Use interventions known to be of importance in the management and
resolution of significant medical care problems; and

Use interventions known to be of importance in the management and
resolution of significant behavioral health problems

I. Improving the Outcomes of Care

Many believe that the most significant performance measures are those
that reflect the outcomes of care, that is, whether people get better
and how quickly; whether their conditions are managed effectively even if they
cannot be cured; whether their functioning and quality of life are maintained
to the extent possible; and whether and when they die. In addition to these
traditional definitions of good outcomes, which tend to be driven by the
expectations of professionals, many people believe there is another important
outcome of care: the degree to which patients or plan members are satisfied
with the care they receive.

In spite of the acknowledged importance of medical outcomes, project
staff were consistently advised, even by those who have devoted their lives to
the development of outcome measures and the conduct of outcomes research, that
it would not be sensible to include medical outcome measures in our list of
performance measures. Outcome measures have been used primarily in highly
controlled research studies, in which an identified group of individuals are
carefully tracked over time to see how they respond to a new treatment, as
compared to either no treatment or a more conventional treatment. Care is taken
to ensure that the groups of people who get the different kinds of treatment
are extremely similar to one another. This is all to ensure that any
differences in outcomes can, with confidence, be attributed to differences in
the treatment.

However, these controlled circumstances do not exist when MCO
performance is being measured. First, there is no guarantee that the people
with disabilities who enroll in one MCO are going to be at all similar to the
people with disabilities who enroll in another, or who remain in Medicaid FFS
or PCCM systems. Second, there is no guarantee that these people will stay in
an MCO so their progress (or lack of progress) can be tracked. Finally it is
important to choose performance measures that address things the MCO can
strongly influence. Many factors besides medical care are known to have a
strong influence on outcomes. Research studies are carefully structured to
track these other influences and take them into account, but it is very
difficult and expensive to set up these controls in examining a population of
MCO member.

Given these considerations, this domain should emphasize that MCOs
are:

Demonstrating improvements over time in the overall satisfaction,
among members who have disabilities, with the plan, its providers, and the
quality of care they are receiving.

In addition, MCOs can and should conduct their own outcome studies.
Another indicator in this Domain would therefore be the following:

With respect to conditions of special importance to persons with
disabilities, the MCO conducts studies to:

Support or demonstrate improvements over time in the health
outcomes of care;

Support or demonstrate improvements over time in the functional
status of members; or

Support or demonstrate improvements over time in the quality of
life of members.

This Chapter includes measures and measurement systems that should be
considered for use in assessing the performance of MCOs in serving people with
disabilities. The Chapter begins by describing how measures were selected. The
Chapter then presents a matrix that allows the reader to identify which major
domains of measurement are addressed by each measure or measurement system. For
example, if you are interested in learning about HEDIS® measures, the matrix will show which major domains of
performance are, and are not, covered by these measures. After the matrix, we
present, for each major domain, the specific items in the domain that are
available in different measures and measurement systems. For example, if you
are interested in the domain of Providing Access to Needed Services, you should
consult this section to get a quick overview of the specific access measures
available. Based on these summaries, we then comment briefly on which domains
have extensive measures and which do not, with recommendations for measurement
development.

The Chapter then moves to details about each measure. A standard set of
information is presented for each measure, and these items of information are
described to orient the reader. Finally, each measure or measurement system is
presented.

A. How We Identified and Selected the Measures in the
Guide

The measures in the Resource Guide were identified through an extensive
search process that included the following:

a review of published literature;

a review of unpublished literature, including the proceedings and
hand-out materials for relevant conferences; and

in-depth interviews with experts in performance measurement, and in
the delivery of health care services to people with disabilities.

In selecting the measures to be included in the Resource Guide, we used
the following criteria:

High relevance to the domains of performance included in our
framework (see Chapter Three);

Appropriateness as a measure of the performance of a system of
care rather than for the evaluation of individual health care providers;

Applicability to a population of people with disabilities with a
broad mix of illnesses and impairments;

Sufficient specificity for consistent measurement to occur over
time and across settings;

Evidence of scientific reliability and validity, as demonstrated
either through the procedures used to develop the measure or the results of
psychometric tests conducted after initial use of the measure;

History of development or use of the measure with people with
disabilities;

Ease, cost and burden of data collection;

Availability of the measure in the public domain;

Availability of technical documentation for the measure and
information on how data for the measure should be collected; and

Availability of technical support for the use of the measure,
preferably at low or no cost.

All the measures do not meet all these criteria. Criteria in boldface
are met by all measures in the Guide. With respect to other criteria, some
measures rate more highly than others, the Guide includes information that
permits readers to make their own assessments on these criteria. A few measures
are not available as we go to press, but are expected to be available very
shortly.

B. Summary Matrix of Domains Addressed in Each
Measure

Table 1 below matches each of the five measurement
systems to the nine quality domains we have created. The following is a listing
of the measurement systems and their abbreviations as referenced in the
table:

The Health Plan Employer Data and Information Set (version 3.0) of
the National Committeee on Quality Assurance: HEDIS®

3. Supporting
Member Involvement in Decision Making and System Improvement

-

-

X

-

X

4. Resolving
Problems and Concerns

X

-

X

X

X

5. High
Quality Interpersonal Interactions Between Members and Providers

-

-

X

X

X

6. Using
Preventive Services to Keep Members Healthy and Functioning

X

-

X

X

-

7.
Coordinating and Integrating Medical and Non-Medical Services

X

-

X

X

X

8. State of
the Art Treatments

X

X

-

X

X

9. Improving
the Outcomes of Care

X

X

X

X

X

C. Summary of Specific Items Available in the Measures
for Each Domain

In this section, we list, for each domain in our framework, the specific
items that are measured in each measure or measurement system included in the
Resource Guide. Detailed descriptions of these measures and measurement systems
can be found in Section F below.

Domain 1. Creating a System with the Right
Capacities

HEDIS®

Pediatric mental health network

Chemical dependency services

Quality assessment and improvement systems

Provider turnover

CAHPS

Ability to find a personal doctor or nurse

Oregon Health Plan Survey

Ability to have a choice among different plans

Ease of communication between personal doctor and specialists

Creating a position of Exceptional Needs Care Coordinator (ENCC) in
health plan

D. Areas Where Measures Are and Are Not Available

The preceding summary of the specific measures available in each domain
reveals clearly that in certain domains there is a rich and varied choice of
measures, while in others there are relatively few measures. There is an
especially wide choice of measures in these areas:

Domain 2. Providing Access to Needed Services; and

Domain 5. High Quality Interpersonal Interactions Between Members and
Providers.

Research has indicated that these two areas are of special importance to
all kinds of patients and consumers, in the context of managed care, and they
are certainly of importance to people with disabilities. In addition, it is
easy to gather data on these topics from patients and consumers themselves, who
are often in the best position to provide useful information on these
topics.

There is a reasonable number of measures available in these two
areas:

Domain 6. Using Preventive Services to Keep Members Healthy and
Functioning; and

Domain 8. State of the Art Treatments.

These two areas have been a focus of a good deal of measurement
development efforts that build on the foundations of measurement of clinical
quality of care. However, there is still work needed in these areas to develop
measures that address conditions and concerns specific to people with
disabilities, including more and better measures of mental health and
developmental problems. In contrast, there are relatively few measures
available in Domain 9, Improving the Outcomes of Care. As noted earlier, there
are methodological difficulties in applying traditional clinical outcome
measures in the assessment of the outcomes of care for a population of members
in MCOs. In the early stages of the process of measuring the performance of
MCOs in caring for people with disabilities, other arenas may therefore have
higher priority for new measurement development.

In particular, the scarcity of measures in the following areas is of
concern:

Domain 1. Creating a System with the Right Capacities

Domain 3. Supporting Member Involvement in Decision Making and System
Improvement

Several of these domains can and often are addressed by looking at MCO
structures. Chapter Five includes several criteria sets
that specify structural and procedural characteristics of MCOs that are
believed to be necessary, if not sufficient, for the delivery of high quality
care to people with disabilities. As we note, criteria are not, in and of
themselves, measures, but the judicious use of structural criteria can help to
complement other measures.

However, some aspects of these domains cannot be addressed simply in
structural terms. First, we need to know if structures are not only in place,
but working to meet the needs of people with disabilities. Second, some aspects
of these domains go beyond structural issues. Project staff believe that it is
especially critical that priority be given to measurement development in the
following areas:

the involvement of individual people with disabilities (and where
appropriate their family and other caregivers) in decisions about their health
care;

the involvement of people with disabilities, their families and other
caregivers, and their advocates, in decisions about the design and improvement
of health care delivery systems;

the degree to which member problems, concerns and grievances are
promptly addressed and resolved to their satisfaction;

continuity and coordination of medical care services; and

coordination of medical and related social and ancillary services.

E. Overview of Information to be Provided About Each
Measure or Measurement System

The following information is presented below, wherever it is available,
for each measure and measurement system in the Resource Guide:

Name of Measure: The formal name of the measure or
measurement system

The Author or Organization That Developed the Measure:
This would include the key contact person and location information for where to
get a complete copy of the measure and any documentation that is available for
the measure.

Domains of Performance Addressed by the Measure: Many of
the measures in the Resource Guide address more than one of the domains of
performance we have discussed in Chapter Two, Section E. We
will list each of the domains of performance to which the measure is relevant
and usually give examples of specific items or indicators in each domain that
are addressed in the measure.

Data Collection Strategies Required for Use of the
Measure: This includes the following issues:

What type of data collection is required?

From whom are data collected?

What kind of sampling or oversampling is needed?

What if any methods are available for risk-adjustment of the data
collected?

What Evidence is Available to Support the Validity and Reliability
of Measure: The Guide summarizes evidence drawn from the process of
developing the measure and/or from psychometric testing of the measure that
indicates that it is likely to be valid and reliable.

What Are the Populations and Settings in Which the Measure Has
Been or Could Be Used: Very few performance measures have been
developed specifically to assess care for people with disabilities. This
section provides information on the populations and settings for which the
measure was originally developed, and where it has been used, and gives an
assessment of the additional populations or settings where it could be
used.

What Written Documentation is Available to Support Use of
Measure: This section lets the reader know whether there is written
documentation that can be used in implementing the measure, and the level of
detail of this documentation.

What Consultation is Available to Support Use of Measure:
This section indicates whether technical consultation is available, from the
original developers of the measure or from others, about its use. When
consultation is only available at a charge this is noted.

Limits on Use of the Measure: This final section presents
any limitations on the use of the measure, or on the interpretation of results,
that need to be taken into consideration in a performance measurement
context.

F. Descriptions of Measures and Measuring Systems

1. Selected Measures from the Health Plan Employer Data and Information
Set (HEDIS®), Version 3.0

HEDIS® is one of the most well-known
performance measurement systems for MCOs. It is a measurement system that
includes dozens of specific measures. We have selected, from HEDIS® 3.0, those measures that are most relevant to assessing
the performance of MCOs in caring for persons with disabilities.

NCQA is a non-profit organization formed specifically for the purpose of
assessing and improving the quality and performance of health maintenance
organizations. It is governed by a Board of Directors that includes
representatives from purchasers of health care, from MCOs, and from the public.
NCQA began its work by developing standards for the independent accreditation
of HMOs. It then moved into performance measurement, with the HEDIS® system. HEDIS® 3.0 includes
measures considered relevant for the commercially insured population, for
people on Medicaid, and for people on Medicare. It was developed by the NCQA
Committee on Performance Measurement (CPM), whose membership mirrors that of
the NCQA Board.

Domains of Performance Addressed by the Measure:

HEDIS® 3.0 measures are organized by NCQA
into sub-sets which do not map with our domains. We have chosen specific
measures from several sub-sets, including the following: (1) effectiveness
of care; (2) health plan descriptive information; (3) health plan
stability; and (4) use of services. Using our domains of
performance, we will list the specific HEDIS® 3.0
measures we recommend:7

(Note: Utilization rates can only be interpreted if there is a
well-grounded standard for determining what the "correct" level of utilization
should be for a given population of individuals. For example, if utilization
rates are compared over time or across plans, there needs to be confidence that
the rate at which members in different plans need a specific service is very
similar. Even in that cases, only wide disparities would generate cause for
concern.)

Domain 4. Resolving Problems and Concerns

Disenrollment rates

Domain 6. Using Preventive Services to Keep Members Healthy and
Functioning

Arrangements with public health, educational and social service
organizations

Case management

Domain 8. State of the Art Treatments

Beta blocker treatment after a heart attack

Eye exams for people with diabetes

Treating children's ear infections

Prenatal care in the first trimester

Check-ups after delivery

Follow-up after hospitalization for mental illness

Domain 9. Improving the Outcomes of Care

Readmission for specified mental health disorders

Readmission for chemical dependency

Data Collection Strategies Required for Use of the
Measure:

Type of Data Collection

The measures listed above require a wide variety of data collection
strategies, but are primarily based on administrative and clinical records.

From Whom Are Data Collected

All data are collected by the MCO, using the detailed specification
provided by NCQA. It is essential to validate these data using an independent
auditor. NCQA is currently training and licensing organizations to serve this
role.

Sampling and Oversampling Issues

Hundreds of MCOs collect HEDIS® data, either
on their entire population or on members supported by particular sponsors
(private employers, Medicare, etc.). Especially since NCQA earlier developed a
version of HEDIS® specifically for Medicaid (which
has now been integrated into HEDIS® 3.0), many MCOs
collect HEDIS® data on Medicaid populations.
However, it is very likely that if a routine general population sample is taken
of MCO members for purposes of HEDIS®, the number of
people with disabilities in the sample will not be sufficient to permit
meaningful conclusions. It is therefore essential either to target data
collection at members who are known (for example by their eligibility category)
to have a disability, or to develop methods for "oversampling" the records of
people with disabilities.

Risk Adjustment

The documentation for HEDIS® 3.0 includes
information on whether and how specific measures can be risk adjusted.

Evidence to Support Validity and Reliability of
Measures:

HEDIS® measures are selected using a rigorous
and extended process of examination and testing of candidate items against
criteria of scientific validity and reliability. The documentation for each
measure includes detailed information on the specific evidence of validity and
reliability.

Populations and Settings in Which Measures Have Been/Could Be
Used:

The original HEDIS® measures were designed to
assess care provided to commercially insured populations enrolled in HMOs.
HEDIS® 3.0 incorporates measures targeted to
Medicaid and Medicare populations. As a measurement system, HEDIS® 3.0 is intended to address a cross section of the
American people. To the extent that people with disabilities have been enrolled
in various kinds of MCOs (commercial, Medicaid, Medicare) the measures have
been used with this group. Given appropriate sampling and oversampling,
HEDIS® 3.0 measures can be used specifically to
assess care for people with disabilities.

The extensive use of HEDIS® among populations
without disabilities facilitates comparisons of MCO performance for people with
disabilities to their performance for other populations. For example, HCFA has
mandated the collection of HEDIS® 3.0 data by all
Medicare risk-contract HMOs serving Medicare beneficiaries (including those
eligible because of a disability).

NCQA has developed "Quality Compass," a system through which HMOs
voluntarily provide the results of HEDIS®
measurement to the agency, for incorporation into a data base that provides
information on performance on individual measures for a substantial sample of
HMOs. This can be useful in comparing performance and in developing benchmarks.
However, because HEDIS® measures are specific to
HMOs, they do not permit comparisons with FFS or PCCM.

Written Documentation Available to Support Use of
Measure:

Extensive and detailed documentation is available from NCQA on
individual HEDIS® 3.0 measures and on the entire
HEDIS® data collection process. Since the
documentation is voluminous, there is a charge for it.

Consultation Available to Support Use of Measures:

Some consultation is available from NCQA regarding the use of HEDIS® 3.0. As noted above, NCQA is currently identifying
organizations qualified to validate HEDIS® data
collection efforts. In addition, NCQA has begun to work on the dissemination of
HEDIS® information, and in the design of materials
to report the results of HEDIS® to different
audiences, including consumers. Perhaps more significant, hundreds of HMOs have
used the system and are familiar with it, and many private and public
purchasers, including State Medicaid agencies, have also used the system and
could provide information regarding their experiences on an informal basis.

Limits on Use of the Measures:

Limits on the use of utilization rate measures have already been noted.
It should also be noted that descriptive information provided by HMOs, such as
that included in Domains 1 and 7, is quite difficult to validate. The same is
true for all forms of self-reported structural data.

Founded in 1994, AMBHA is an association of managed behavioral health
organizations (MBHOs) based in Washington, D.C. The member organizations, from
38 States, manage the mental health and substance abuse health plan benefits
for over 98 million people. The association was formed to work on issues of
public accountability, quality, and public policy and communication. The
mission statement of the organization has as one of its objectives: "To promote
health plan, management agent, and provider accountability through performance
measurement."

At the time PERMS 1.0 was first administered, AMBHA included 19 member
organizations, of which 13 responded to parts of PERMS; these firms had covered
lives of approximately 21 million people. As PERMS 2.0 evolved, portions of it
were incorporated within the HEDIS® measures
released by the NCQA in January of 1997. Developmental work on PERMS 2.0 is
still ongoing, and reflects areas "requiring more attention" such as substance
abuse, the severely and persistently mentally ill, and special populations. As
AMBHA develops and refines measures, their goal is to "act as a testing site
and a prod" so that quality measurement organizations that operate on a
national level will incorporate what AMBHA considers to be valid and realistic
behavioral health performance measures. The NCQA has incorporated a number of
these measures into their HEDIS® Measurement system;
other measures for consumer satisfaction may be incorporated into the Consumer
Assessment of Health Plans (CAHPS) surveys, in particular the Behavioral Health
Survey, which are discussed in section 3 below.

Domains of Performance Addressed by the Measure:

PERMS 2.0 categorizes its measures into Access to Care, Quality of Care,
and Consumer Satisfaction under the headings of "Core Measures," and
"Leadership Measures." The Core Measures domain consists of "HEDIS-type"
measures, with the addition of an item on "call abandonment reporting." The
Leadership Measures domain incorporate HEDIS-type measures for diagnoses of
depression, schizophrenia, and alcohol abuse, by age and by gender.

Domain 2. Providing Access to Needed Services

Telecommunications standards--standards such as a call abandonment
rate of less than 5% of calls on-hold for less than 30 seconds, and an average
answer time speed of five rings or fewer

Consumer satisfaction with the time interval to the first appointment

Domain 8. State of the Art Treatments

Encouragement by provider to use self-help or consumer-run programs

Medication management for individuals with schizophrenia

Family visits for children undergoing mental health treatments

Ambulatory follow-up within 7 and 30 days of discharge for mental
health

Ambulatory follow-up within 7 days of discharge for substance abuse

Ambulatory follow-up after hospitalization for major depressive
disorder

Domain 9. Improving the Outcomes of Care

Rates of successful engagement in treatment for substance abuse

Data Collection Strategies Required for Use of the
Measure:

Type of Data Collection

PERMS data are collected through mailed surveys of which are sent to
Harvard Medical School for analysis.

From Whom Are Data Collected

Data are collected from member organizations of AMBHA who respond based
on their administrative and clinical records. These members in turn collect
data from their patients about satisfaction. AMBHA has over 48,000,000
enrollees across 38 States.

Sampling and Oversampling Issues

The AMBHA Managed Behavioral Healthcare Quality and Access Survey was
sent to all members; the patient population of these organizations accesses
behavioral health services through from multiple sources: 26 percent are
covered by employee assistance programs; 27 percent by managed
indemnity/utilization review models; 27 percent by HMOs, and 20 percent by
network-based models. There is no breakdown of Medicaid or commercial
subpopulations. Special arrangements would be needed to identify and target or
oversample people with disabilities in the patient populations of AMBHA members
in order to get a sufficient sample to provide information specifically on this
group.

Evidence to Support Validity and Reliability of
Measures:

The original measures used in PERMS were developed by the AMBHA Policy
Subcommittee on Quality and Access Standards, which has seven working groups.
The groups used research and their own knowledge to develop the indicators to
measure quality and access. A contracted survey firm, Foster Higgins, reviews
the questions for wording and structure "in accordance with accepted research
standards." The measures which are part of other measurement sets (such as
HEDIS®) are tested as part of their protocol. "The
large sample size assures the statistical validity of the information and the
ability to generalize information to the rest of the population," according to
the Foster Higgins firm which conducted the survey.

Populations and Settings in Which Measures Have Been/Could Be
Used:

Measures are considered by AMBHA to be appropriate for all patients of
MBHOs including Medicaid populations.

Written Documentation Available to Support Use of
Measure:

Sections encapsulated by HEDIS® will have the
support of NCQA. A list of AMBHA reports, studies, and publications and
information on how to obtain these documents is available at their web page:
http://www.ambha.org.

Consultation Available to Support Use of Measures:

The availability of consultation for use of PERMS outside of AMBHA's
membership is unclear, but presumably AMBHA staff would be available to their
membership for assistance.

Limits on Use of the Measure:

The results of the "report card" from PERMS 1.0 is available to the
public and to health care purchasers. Results on PERMS 2.0 will also be
available to the public upon publication.

Underlying all measurement of behavioral managed care performance is a
"fundamental lack of consensus about what constitutes quality of care in the
behavioral arena" (Gore, 1998); therefore, the use of the measure may be
limited by differing opinions of the definition of good care in the behavioral
health field. In addition, consultation would be needed with AMBHA and the
experts with whom it works to develop an approach to the use of PERMS measures
by sponsors other than behavioral health care organizations.

3. Consumer Assessment of Health Plans (CAHPS) 2.0
Surveys

Author/Developer: The CAHPS surveys, which are designed to
gather information from people who have been in and used health plans about
their experiences in the plans, and their ratings of the plans, are being
developed by three teams of researchers. Here are the three organizations
leading the teams, with the name of the Principal Investigator(s).

The CAHPS project has been conducted with initial and primary support
from the Agency for Health Care Policy and Research (AHCPR), with supplemental
support from HCFA and private foundations. In addition to their support of the
three CAHPS teams listed above, AHCPR has also created and is supporting a
Survey User's Network (SUN network), whose role is to disseminate the CAHPS
surveys and the reporting templates associated with these surveys, and to
provide hands on support to those interested in sponsoring CAHPS surveying
efforts. The organization operating the SUN network is Westat, Inc., 1650
Research Blvd., Rockville, MD 20850, Primary Contact Person: Vasudha Narayanan.
The CAHPS Hotline is (800) 492-9261

AHCPR's website has information on CAHPS. By accessing
http://www.ahcpr.gov/qual you can get both basic and detailed information on
the project.

Domains of Performance Addressed by the Measure:

CAHPS is a measurement system, rather than a measure. It is a collection
of surveys that focus on different groups of health plan members. Most but not
all CAHPS surveys are designed for use with people enrolled in a wide range of
health insurance plans, including MCOs of various kinds, and FFS plans. CAHPS
surveys have been fully developed that address the following populations:

Adults who are privately insured (this is considered the "core" CAHPS
survey);

Children who are privately insured;

Adults who are eligible for Medicaid (with special emphasis on those
eligible under TANF, i.e., women with children);

Children who are eligible for Medicaid (again with special emphasis
on the TANF population);

Persons with chronic conditions and disabilities; and

Medicare beneficiaries enrolled in HMOs.

Surveys are currently under development that address several other
special populations or issues.

These surveys are not final, and certain items that appear in currently
available drafts may not be present in the final version, or may be
reworded.

CAHPS surveys address a number of domains of performance, highlighting
those for which people in plans can be considered good sources of data. Under
each domain covered by one or more CAHPS surveys, the kinds of questions
relevant to that domain are listed. When a question is found only in one survey
that is indicated.

IMPORTANT
NOTE

It is not appropriate to use individual
CAHPS items. Rather, the entire survey should be used. There are few if any
items in CAHPS surveys that would be irrelevant to the measurement of
performance of MCOs in serving the needs of people with
disabilities.

Domain 1. Creating a System with the Right Capacities

Ability to find a personal doctor or nurse

Domain 2. Providing Access to Needed Services

Problems getting care you and your doctor thought was needed

Problems with delays in treatment while waiting approval from the
health plan

Access to needed help and advice from the doctor's office, by
telephone

Timely access to appointments for regular or routine care

Timely access to a doctor or health care provider for urgently needed
care

Access to specialists

Access to interpreters (Medicaid surveys)

Access to special medical equipment (surveys for Medicare and people
with disabilities)

Access to physical, occupational and speech therapy (Medicare HMO
survey and surveys for people with disabilities)

Access to home health care services (Medicare HMO survey and surveys
for Medicare and people with disabilities)

Access to respite care (Medicare HMO survey and surveys for people
with disabilities)

Access to prescription medications (Medicare surveys)

Domain 3. Supporting Member Involvement in Decision Making and System
Improvement

Involvement in decision making about care (child surveys address
involvement of both child and parent; Behavioral Health Survey may address
involvement of family in decision making)

The CAHPS 2.0 measurement system is a set of surveys. Each can be
administered by either telephone or mail or a mix of the two methods. Testing
has been conducted that indicates no differences in results between phone and
mail administration. In order to administer a CAHPS survey it is essential to
have a sampling frame of members of each health plan whose performance is being
assessed. If a specialized survey is used, the sampling frame must be specific
to the particular group of members for whom that survey is relevant. Note,
however, that initial screening questions confirm the individual's membership
in the plan.

From Whom Are Data Collected

CAHPS requires the collection of survey information from members of
health plans. Typically, several health plans are surveyed because they are
being offered either to a population (the privately insured; Medicaid
eligibles) by a sponsor (a large employer or purchasing coalition; a State
Medicaid agency) that decides to implement CAHPS. Sometimes however, CAHPS
surveys are implemented for all or almost all health plans in a State or
region. The survey(s) used depends on the member populations of interest to the
sponsor. CAHPS surveys are intended to be administered by an independent
organization, not by the health plans themselves. The goal is to generate data
that will be viewed as credible, especially by the public and individuals
making a health plan choice.

Sampling and Oversampling Issues

If a routine general population sample is taken of MCO members for
purposes of a CAHPS survey, it is likely that the number of people with
disabilities in the sample will not be sufficient to permit meaningful
conclusions. It will be essential either to target data collection for members
who are known (for example by their eligibility category) to have a disability,
or to develop methods for "oversampling" the records of people with
disabilities.

Risk Adjustment

Items have been included in all CAHPS surveys that gather information on
the demographics and general health status of respondents. These data permit
basic adjustments for differences between plans in such common risk-adjusters
as age and overall self-reported health. Note that these are intended to be
adjustments that might affect how people report on their experiences with
plans.

Evidence to Support Validity and Reliability of
Measures:

CAHPS measures have been developed by nationally recognized experts on
surveys of consumers' experiences in health plans and their ratings of health
plans. They have been subjected to intensive cognitive testing with the
populations for which they are intended, in order to ensure that all questions
and response options are understood as intended. Surveys have also been
field-tested with relevant populations and psychometric analyses have been
conducted to assess their validity and reliability. In addition, surveys have
been reviewed by external advisers and experts on such issues as the required
educational level for reading surveys (a sixth grade level is the standard for
all surveys), and have been subject to public comment. A number of
demonstration sites have implemented CAHPS surveys in a context where formal
evaluations are being conducted by the CAHPS research teams. Other "early
adopters" have used the survey in the course of their normal performance
measurement activities. State Medicaid agencies have been among both the
demonstration sites and early adopters.

Populations and Settings in Which Measures Have Been/Could Be
Used:

CAHPS surveys, as noted above, have been designed with specific
populations in mind. All surveys include the "core" items; these items make up
the survey of privately insured adults. Other surveys include supplemental item
sets designed specifically to address the concerns and needs of particular
populations. People eligible for Medicaid and people with chronic illnesses and
disabilities were from the outset important survey populations for CAHPS. In
addition, HCFA commissioned the development of a CAHPS Medicare HMO survey,
which includes many items that are similar to those in the survey of people
with chronic illnesses and disabilities. Surveys currently under development,
in particular the survey of Children with Special Needs (being developed by
Harvard) and the Survey of Medicaid SSI (being developed by RAND) are highly
specific to people with disabilities.

One of the major strengths of the CAHPS surveys is that, with the
exception of the Medicare survey, they are designed to permit comparisons of
member experiences across different kinds of delivery systems. Thus, for
example, the survey can be and has been used for people on Medicaid who are in
FFS or PCCM systems. As with HEDIS® measures, HCFA
has mandated the collection of the CAHPS Medicare HMO survey data for all
Medicare risk-contract HMOs serving Medicare beneficiaries (including those
eligible because of a disability). This will provide a useful basis for
comparisons.

Written Documentation Available to Support Use of
Measure:

Extensive and detailed written documentation is available from the
Survey Users Network (SUN Network) on each of the fully developed surveys.
CAHPS surveys are currently undergoing revisions to reflect the integration of
initial CAHPS surveys (Version 1.0) with the NCQA Member Satisfaction Survey.
Version 2.0 of CAHPS surveys and a revised version of the "Survey and Reporting
Kit" will be available in early 1999. In the meantime, however, written
documentation remains available for earlier versions and much of the
information is still valid. As with Version 1.0, demonstration sites and other
early adopters will have access to the most current version of any fully
developed CAHPS survey for implementation, and can get support from SUN.

In addition, CAHPS has developed methods and materials for reporting the
results of CAHPS surveys to consumers, including print materials and an
interactive computer program. Templates of these materials and documentation on
how to adapt them are included in the "Survey and Reporting Kit." These
educational materials have gone through the same process of testing and
scrutiny as the surveys themselves. Certain reporting products are specific to
the Medicaid population.

Consultation Available to Support Use of Measures:

The Survey Users Network contractor can provide consultation on the use
of CAHPS surveys that are fully developed, and information on the status of
surveys under development. CAHPS research teams that are taking the lead in
development of newer surveys would be available for limited consultation on
these surveys. In addition to providing technical assistance, the SUN
contractor periodically brings together users to share what they have learned
about the use of these surveys.

Limits on Use of the Measure:

By definition, surveys of consumers are vehicles for gathering
information that the members of plans and users of health care are in a
particularly good position to assess and rate. Not all aspects of health plan
performance can be assessed and rated by consumers. In addition, there are
limits on the use of surveys by individuals who are cognitively impaired or
mentally dysfunctional. Finally, while to date the Medicaid surveys have been
translated into Spanish, most CAHPS surveys are available only in English.
Resources are currently being sought to permit translation of a wider range of
the surveys into other languages. It is likely that CAHPS research teams would
be willing to work with sites interested in pursuing translation efforts.

4. Oregon Health Plan Adult Satisfaction Survey of
1997

Name of Measurement System: The Oregon Health Plan Adult
Satisfaction Survey of 1997

This is an example of an instrument developed by a State in order to
ascertain both the quality and performance of its managed care and
fee-for-service models, from the perspective of Medicaid clients. The survey
was developed by the Office of Medical Assistance Programs of the Oregon Human
Resources Department in Salem, Oregon.

Domains of Performance Addressed by the Measure:

The Oregon Health Plan Adult Satisfaction Survey questions are directed
to consumers. It includes a series of headings and sub-headings such as "Your
Health Plan," "Thinking About Your Health Plan," and "Thinking About the
Doctor's Office or Clinic." We have chosen to include specific queries from
several headings.

Domain 1. Creating a System with the Right Capacities

Ability to have a choice among different plans

Ease of communication between personal doctor and specialists

Creating a position of Exceptional Needs Care Coordinator (ENCC) in
health plan

Rating of helpfulness of provider in giving information about other
agencies or services

Domain 8. State of the Art Treatments

Rating of home health care

Domain 9. Improving the Outcomes of Care

Patient rating of health status compared to one year previously

Data Collection Strategies Required for Use of the
Measure:

Type of Data Collection

Survey forms are mailed by OMAP to Medicaid beneficiaries. The surveys
are returned to OMAP and then electronically scanned.

From Whom Are Data Collected

Survey recipients are sampled at random from all Oregon Medicaid
beneficiaries who have been enrolled in the Oregon Health Plan for a minimum of
six months. The survey is available in Spanish, Russian, and Vietnamese.
Potential respondents who speak other languages can use an AT&T translation
operator and complete a telephone survey. There is a Client Hot Line to respond
to questions from respondents and a prepared list of questions and answers for
those answering the hot line telephone.

Sampling and Oversampling Issues

In order to generate a sufficiently large sample of people with
disabilities to ensure reliable estimates, this survey, like others presented
in the Guide, would either need to be based on a targeted sampling frame, or
would require the use of oversampling and/or a special set of screening
questions to identify individuals with disabilities.

Risk Adjustment

A limited number of questions in the survey address health status of
respondents; they are probably insufficient to support risk adjustment of
responses. However, depending upon how the sampling frame for the survey was
structured (e.g. by eligibility category) it is possible that responses could
be calculated for different groups.

Evidence to Support Validity and Reliability of
Measures:

Information not available at this time.

Populations and Settings in Which Measures Have Been/Could Be
Used:

The measure has been used with Medicaid and low-income populations who
represent a cross-section of eligibility categories, including people with
disabilities.

Written Documentation Available to Support Use of
Measure:

The survey instrument, as well as sample letters used for mailing the
instrument, and scripts used to address Frequently Asked Questions from survey
recipients (i.e., beneficiaries of Medicaid and other governmentally support
health insurance programs) are available from Judy Mohr-Peterson (see contact
information above).

Consultation Available to Support Use of Measures:

Oregon has provided copies of the measure to several other State
Medicaid agencies and has provided informal consultation on its use.

Limits on Use of the Measure:

The primary limitation on the measure is that some questions, for
example those on Exceptional Needs Care Coordinators, might not be appropriate
for settings in which such resources are not required or available. However, it
may be possible either to eliminate those questions or replace them with items
that are more appropriate to a particular context.

The Picker Institute is a non-profit organization dedicated to
increasing the extent to which health care services are patient-centered. They
have developed a number of patient surveys that focus on specific aspects of
health care, including three that are of particular relevance to the needs of
people with disabilities:

A survey of adult patients who have recently utilized inpatient
hospital services in a medical or surgical ward (including questions that
address admission through an emergency room if that occurred);

A survey of patients who have recently utilized a rehabilitation
program; and

A survey of patients who have recently utilized home care services.

Domains of Performance Addressed by the Measure:

The Picker Institute surveys address several domains of performance. We
indicate each domain, and give examples of specific items covered in one or
more surveys. When an issue is addressed in only one survey, this is so
indicated. Note that in the Picker surveys, questions are often directed to the
patient's experience with a particular kind of provider; thus several issues
are addressed with respect to different providers (e.g., doctors, nurses,
social workers, home health aides etc.)

The Picker Institute instruments are surveys of patients. They are
administered by mail.

From Whom Are Data Collected

Data are collected from individuals who have been identified as having
used a particular kind of facility or service, within a specified time period.
Data are typically collected by The Picker Institute itself, under contract
with health care organizations, MCOs, employers and business or purchaser
coalitions.

Sampling and Oversampling Issues

The key sampling issue for Picker Institute Surveys is the ability to
identify from records individuals who have used particular services and to
contact a random sample from this list. Valid records would need to be provided
either by the MCO or by the health care facilities under contract with the MCO.
Normally, the surveys would not be conducted only for people with disabilities,
although if information were available as to disability status as well as use
of services, this could be done. Sample size will, as in all cases, be an
issue. On the other hand, it is more likely that a sizable portion of people
using rehabilitation and home care services would have some kind of
disability.

Risk Adjustment

Information is collected in the surveys on demographic characteristics
and health status that could be used for risk adjustment purposes.

Evidence to Support Validity and Reliability of
Measures:

Picker Institute measures are developed using a rigorous and extended
process. The process typically begins with focus groups of patients who have
used the service in question, to identify what is of greatest interest and
importance to them. Surveys are usually cognitively tested prior to field
testing. Psychometric analyses are conducted on the reliability, validity and
discriminate ability of the surveys. As data are collected using a particular
survey, these data are analyzed and used to continue psychometric analysis and
survey refinement.

Populations and Settings in Which Measures Have Been/Could Be
Used:

Picker Institute surveys have been used in a wide range of health care
organizations and MCOs. Surveys are not targeted to patients on the basis of
their source of insurance, but rather on their use of the facility in question.
The surveys presented here appear to be relevant to the needs and concerns of
people with disabilities.

Written Documentation Available to Support Use of
Measure:

The Adult Inpatient and Rehabilitation surveys are not in the public
domain; they are copyrighted and can only be used with permission. The Home
Care survey is in the public domain. The Picker Institute has written
documentation of the psychometric analyses conducted on their surveys; some
articles have also appeared in the academic literature.

Consultation Available to Support Use of Measures:

As noted earlier, The Picker Institute typically administers the surveys
themselves for a variety of clients. This would involve a charge to be
negotiated with the client.

Limits on Use of the Measure:

The primary limitation on the use of these surveys measure is that two
of the three are not in the public domain; however, Picker staff are extremely
interested in the widespread use of their surveys and in issues significant to
people with disabilities. Translations of the instruments would be required if
non-English speaking populations were involved.

This Chapter presents work that has been done to identify structural
characteristics and procedures of MCOs that are viewed as highly significant to
their capacity to serve the needs of people with disabilities. By themselves,
structural characteristics can be seen as criteria for assessing MCOs,
rather than as measures. They indicate what should be present, but do
not necessarily provide a way to find out if the characteristic is present. The
great advantage of structural criteria is that they can be used prospectively,
i.e., before an MCO begins to serve people with disabilities. The disadvantage
is that the existence of a structure does not guarantee its appropriate use.
Structures are necessary, but not sufficient, to the delivery of good care.

Another advantage of the criteria sets in this Chapter is that several
were developed with the health care needs and concerns of people with
disabilities specifically in mind. The same kind of information on criteria
sets is offered as that presented in Chapter Four on
measures and measurement systems. For example, criteria are organized using the
framework of domains of MCO performance presented in Chapter
Three. In many cases, the complete criteria set goes beyond the domains of
performance we have laid out in this Resource Guide. Readers are therefore
encouraged to go to the complete sets since there may be additional criteria
they would want to apply.

How can you use structural criteria sets in a performance measurement
system? How can you find out whether a characteristic is in place? There are
several options to consider:

In many cases, structural characteristics and criteria can become
contract specifications. Indeed, one criteria set we present was developed as a
set of specifications for Medicaid MCO contracts.

MCOs can be asked to report whether these characteristics are in fact
present, and to provide documentary evidence of their presence, such as
contracts with providers; provider listings; memoranda of understanding; and
procedure manuals.

Visits can be made and interviews conducted with MCO staff to further
probe these characteristics, to answer questions raised by descriptions or
documentation, and to gather information from multiple individuals who would be
knowledgeable about a given aspect of the MCO's functioning. Such visits and
interviews could also be conducted by MCO staff with provider organizations,
since MCOs want to know the structural characteristics of those in their
network.

Structural criteria often do not specify a clear standard. For example,
while an MCO needs sufficient numbers of the right kind of providers in order
to serve a patient population, there are few if any research-based standards
about how many of what mix of providers is actually sufficient. The Chapter
does not suggest such standards. It may be more appropriate for a group of
people representing the perspectives of multiple constituencies to work to
develop consensus on such standards.

A. Quality Improvement System for Managed Care
(QISMC)

The intent of QISMC is to serve as a system for "assuring that managed
care organizations contracting with Medicare and Medicaid protect and improve
the health and satisfaction of enrolled beneficiaries." HCFA and State Medicaid
agencies are expected to use QISMC as a uniform set of quality standards for
assessing Medicaid and Medicare contractors. Given the health plan performance
and quality requirements embodied in the 1997 Balanced Budget Act (BBA), it is
likely that the final version of QISMC will become mandatory for Medicare
contracts and optional for state Medicaid contracts. States opting to use QISMC
in their Medicaid contracting will likely meet the BBA requirements.

QISMC is described in a 132-page document divided into four parts:

A description of a framework for understanding QISMC requirements,
including a background on the origins and goals of the project;

A list of standards that each organization must meet under QISMC,
grouped under four domains:

quality assessment and performance improvement;

enrollee rights;

health services management; and

delegation (i.e., scope and applicability of standards as they
apply to subcontractors of the contracting health organization);

Detailed guidelines for interpreting the standards.

The process for implementing QISMIC

Name of Criteria Set: Quality Improvement System For
Managed Care (QISMC)

Author/Developer: The National Academy for State Health
Policy (NASHP) for the Health Care Financing Administration (HCFA) of the U.S.
Department of Health and Human Services

NASHP is a non-profit multidisciplinary forum of and for State health
policy leaders from the executive and legislative branches. Under contract from
HCFA, NASHP began development of QISMC in 1996 and released a draft version for
public comment in January 1997. Numerous comments were received, and panels of
representatives of plans, providers, regulators and advocates were convened. A
second round of public comments on the January 1997 draft was initiated by HCFA
in May 1998. Based on the comments received on this draft in both rounds, and
on its own internal review, HCFA will revise the system and issue it in final
form. Readers are advised to contact HCFA at the above address for information
regarding the final content of the system.

Domains of Performance Addressed by the Criteria Set:

We have selected the criteria that are most representative of our
domains. QISMC contains many more provisions relating to a broad range of
health care and delivery issues.

Domain 1. Creating a System with the Right Capacities

Projects conducted under the organization's QAPI [Quality
Assurance/Performance Improvement] program address and achieve improvement in
major focus areas of clinical and non-clinical services. Clinical focus areas
applicable to Medicaid enrollees include the following:

chronic diseases affecting elderly or disabled Medicaid
enrollees;

care of Medicaid enrollees residing in long-term care facilities;

care of Medicaid enrollees who are unusually dependent on others
or on devices but who do not reside in long-term care facilities;

mental health treatment for Medicaid enrollees;

substance abuse treatment for Medicaid enrollees; and

care of Medicaid enrollees who are developmentally disabled.

The organization maintains and monitors a network of providers,
supported by written arrangements, that is sufficient to provide timely access
to covered services.

A new contractor, or an established contractor seeking an
expansion of its service area, demonstrates that the numbers and types of
providers available to enrollees are sufficient to meet anticipated needs of
the population and area to be served.

For each physician or other individual practitioner, including each
practitioner within a contracting group who provides services to the
organization's enrollees, the process includes:

Formal selection and retention criteria that do not discriminate
against practitioners who serve high-risk populations or who specialize in the
treatment of costly conditions.

Domain 2. Providing Access to Needed Services

The organization assures that services are accessible to all
enrollees, including those with limited English proficiency or reading skills,
with diverse cultural and ethnic backgrounds, and with physical and mental
disabilities.

Enrollee information is

readable and easily understood;

available in the language(s) of the major population groups
served and, as needed, in alternative formats for the visually impaired.

An established contractor establishes standards for timeliness of
access to care and member services that meet or exceed such standards as may be
established by HCFA or the State Medicaid agency, continuously monitors the
extent to which it meets these standards, and takes corrective action as
necessary.

Domain 3. Supporting Member Involvement in Decision Making and System
Improvement

The QAPI program provides opportunities for enrollees to participate
in the selection ofproject topics and the formulation of project goals.

Each enrollee has a right

to choose providers from among those affiliated with the
organization

Each enrollee may select his or her primary care provider
from among those accepting new Medicare or Medicaid enrollees;

Each enrollee may refuse care from specific practitioners or
providers.

to participate in decision-making regarding his or her health
care

The organization provides for the enrollee's representative
to facilitate care or treatment decisions when the enrollee is unable to do so.

The organization provides for enrollee or representative
involvement in decisions to withhold resuscitative service, or to forgo or
withdraw life-sustaining treatment, and complies with requirements of Federal
and State law with respect to advance directives.

to receive information on available treatment options or
alternative courses of care;

to have access to his or her medical records in accordance with
applicable Federal and State laws

Domain 4. Resolving Problems and Concerns

Each enrollee has a right

to obtain a prompt resolution, through established procedures, of
issues raised by the enrollee, including complaints or grievances and issues
relating to authorization, coverage, or payment of services.

The organization has a system for resolving issues raised by
enrollees, including complaints or grievances; issues relating to authorization
of, coverage of, or payment for services; and, in the case of Medicare
enrollees, issues relating to a discontinuation of a service. [NOTE: references
to an enrollee in these standards include reference to an enrollee's
representative.]

Domain 6. Using Preventive Services to Keep Members Healthy and
Functioning

The organization conducts a quality assessment and performance
improvement (QAPI) that achieves, through ongoing measurement and intervention,
demonstrable and sustained improvement in significant aspects of clinical care
and non-clinical services that can be expected to affect enrollee health
status, functional status, and satisfaction.

[QAPI includes] measures to assure that enrollees are informed of
specific health care needs that require follow-up and receive, as appropriate,
training in self-care and other measures enrollees may take to promote their
own health.

The organization promotes continuity of care and integration of
services through:

Programs for coordination of care, including:

Identification of enrollees with complex needs and
development of services and programs to assist them in meeting those
needs;

Coordination of medical care, mental health services and
substance abuse services, and social services and community resources.

Domain 8. State of the Art Treatments

The organization adopts and disseminates practice guidelines or
criteria for the provision of specific services.

Guidelines are based on reasonable medical evidence or a
consensus of relevant practitioners, are developed in consultation with
affiliated providers, and are reviewed and updated periodically.

The organization implements written policies and procedures for
evaluating new medical technologies and new uses of existing technologies.

Evidence to Support Validity and Reliability of the Criteria
Set:

These standards were developed by NASHP utilizing its recognized
expertise in analyzing the myriad roles and responsibilities of States as
purchasers, regulators, and partners of health plans, particularly as these
roles have become increasingly complex with the growth of managed care. A
broad-based consensus development process that incorporated the perspectives of
diverse stakeholders was employed to bring both depth and quality to QISMC.
HCFA is continuing this approach as each of the standards are "vetted" during
its own public comment and review period. The fact that HCFA intends for QISMC
to become a required feature of Medicare contracting and that the agency highly
recommends it as an option for State Medicaid contracting is evidence of its
confidence in the utility and reliability of the system.

Populations and Settings in Which Criteria Set Has Been/Could Be
Used:

QISMC was developed to apply across the needs of all Medicare and
Medicaid managed care enrollees.

NASHP staff are available to provide consultation and technical
assistance to State-level government officials regarding the history and
development of QISMC free of charge. Other users are advised to contact NASHP
regarding their policies on rates for consulting. Inquiries to HCFA should be
directed to Brian Agnew of that office for appropriate guidance on the scope
and cost of consultation and technical assistance.

Limits on Use of the Criteria Set:

Among the comments received during the April 1998 Quality of Care
meeting were: (1) "Plans and some other commenters noted that plans can't
identify some of the populations referred to, such as the developmentally
disabled or persons dependent on assistance or devices; even encounter data
might not identify these enrollees;" and (2) "Advocates wanted stronger
requirements in [the area of coordination]. There was agreement to add
'identification and assessment' to the standard [3.2.2]. There was a
wider consensus that guidelines should reflect the need for screening enrollees
to determine who has complex needs and may require an assessment, and to better
define the term 'complex needs.'"8
HCFA has taken these, as well as all other comments, under advisement as it
completes the final version of QISMC. Readers interested in how it may
ultimately reflect the needs of people with disabilities should contact HCFA
for the most up-to-date information.

B. Key Indicators from the 1997 Santa Fe Summit on
Behavioral Health

In 1997 the American College of Mental Health Administration (ACMHA)
held a "Summit" in Santa Fe, New Mexico as the final process of consensus
development on core values and indicators that should guide the assessment of
behavioral health programs. Their final report presents these core values and
indicators, and also, unlike other criteria sets in this Chapter, presents
information on potential approaches to measurement for these indicators. Since
measurement strategies have not been fully developed for all indicators,
however, it is more appropriate at this time to present them as a criteria set.
Work is ongoing, however, to move these indicators closer to the measurement
process.

Name of Criteria Set: Key Indicators from the 1997 Santa
Fe Summit on Behavioral Health

Author/Developer: The indicators are clearly the result of
the work of multiple individuals and their organizations. However, the final
report was produced by The American College of Mental Health Administrators,
7625 West Hutchinson Avenue, Pittsburgh, PA 15218-1248, Phone (412) 244-0670.
Contact: Lawrence Heller, Ph.D., Executive Director

Domains of Performance Addressed by the Criteria Set:

The Summit Final Report categorizes its indicators into four domains:
Outcomes, Process, Access and Structure. We have reorganized these into the
domains used in the framework presented in this Guide.9

Domain 1. Creating a System with the Right Capacities

The organization's structure is consistent with the delivery of
mental and addictive disorder treatment, with effective consumer and
professional representation in policy making.

Staffing levels are appropriate for delivery of the array of services
and provide for meeting the diverse needs of the individuals served, including
linguistic and cultural competence.

Data on clients are secure, available only to those who need to know.

There is a single, fixed point of responsibility for each client.

There is a quality assurance system in place to examine adverse
clinical events.

Domain 2. Providing Access to Needed Services

Service denials, terminations or refusals are assessed.

Penetration (i.e. utilization) rates demonstrate benchmarked levels
of service delivery to like populations.

Access to a full range of services is demonstrable.

Children and their families receive the appropriate services that
they need, when they need them.

Consumer experiences of treatment (both positive and negative) are
assessed on dimensions of appropriateness, timeliness and sensitivity of
services delivered.

Domain 3. Supporting Member Involvement in Decision Making and System
Improvement

The system of care assumes responsibility for continuous and
integrated care appropriate to the needs of families and children.

There are appropriate linkages to other service systems with which
consumers need to interact.

There is continuity of care within the organization and effective
integration with external caregiving systems.

Domain 8. State of the Art Treatments

Consumers who use inpatient services (i.e., 24-hour,
medically-supervised services for a primary mental or substance abuse
diagnosis) receive face-to-face follow-up care within seven days of
discharge.

Consumers with mental health and addictive disorders are engaged in
treatment (i.e., continue in treatment).

Children and their families are being assessed for and offered
services at appropriate levels (i.e., with respect to use of out-of-home or
restrictive environments).

Domain 9. Improving the Outcomes of Care

The Summit indicators are especially useful with respect to their
specification of outcome criteria. They have developed three for all adults,
including those with serious and persistent disorders, and five relevant only
for adults with serious and persistent disorders:

Indicators for All Adults

Adults reside in their own homes or living arrangements of their own
choosing.

Adults are working (defined to include education, job training,
volunteer work).

Adults have good physical health and report good mental health
(psychological well-being).

Adults with serious and persistent mental or chemical dependency
disorders avoid trouble with the law.

Adults with serious and persistent mental or chemical dependency
disorders maintain a social support network.

Adults with serious and persistent mental or chemical dependency
disorders are able to manage their daily lives.

Adults with serious and persistent mental or chemical dependency
disorders report a positive quality of life.

Evidence to Support Validity and Reliability of the Criteria
Set:

The Summit Indicators were developed through extended discussions of
multiple working groups. Participants in these groups included mental health
and substance abuse professionals working in both private and public sector
organizations; academic experts in these field; and some representatives of
people with mental health and substance abuse diagnoses. The process used is
similar to the consensus development process often used within the National
Institutes of Health with respect to the appropriateness of a particular
approach to a medical intervention, with this difference, that the values and
expectations of participants in the process, as well as their expertise and
experience, are brought to bear in the discussions.

Populations and Settings in Which Criteria Set Has Been/Could Be
Used:

The criteria were designed for use in assessing the performance of
behavioral health care organizations, and could be applied to the performance
of MCOs providing behavioral health services to their members.

Written Documentation Available to Support Use of Criteria
Set:

The complete Final Report on the Santa Fe Summit, entitled Preserving
Quality and Value in the Managed Care Equation is available from ACMHA at
the address noted above. Additional materials which go into greater detail on
indicators and on measures related to these indicators may also be available
from the organization and its collaborators.

Consultation Available to Support Use of Criteria Set:

ACMHA indicates that it would be available to discuss the report with
interested parties.

Limits on Use of the Criteria Set:

Some of the indicators are at a very high level of generality; others
are more specific and would be easier to use as a basis of assessments. It is
important to note, however, that this effort is ongoing, in particular with
respect to the dimension of identifying and coming to consensus on appropriate
measures for the indicators.

C. National Health Law Program (NHeLP) Advocacy
Checklist for People With Disabilities

The NheLP Advocacy Checklist was originally developed as a guide for
advocates to use in reviewing the specifications of contracts between State
Medicaid agencies and MCOs. While contract specifications are one approach to
determine whether or not structural characteristics are in place, the NHeLP
Advocacy Checklist can be viewed as a set of criteria that can be assessed
using other methods besides inclusion in contracts.

Name of Criteria Set: National Health Law Program (NHeLP)
Advocacy Checklist for People with Disabilities

The National Health Law Program, with offices in Los Angeles and
Washington, D.C., is a non-profit organization that has served as a resource
center for legal services programs and other advocates for the health care
needs of low-income people. The Advocacy Checklist was originally developed, as
its title suggests, for use by advocates for people with disabilities. However,
the criteria can be used not only by such individuals and organizations, but by
State Medicaid agencies, MCOs and health care organizations and
professionals.

Domains of Performance Addressed by the Criteria Set:

The NHeLP Advocacy Checklist includes criteria that go well beyond the
domains of performance included in this Resource Guide. For example, they
address marketing issues, the use of third-party organizations to enroll
Medicaid eligibles in MCOs, public disclosure issues, etc. We present only
those criteria and standards that are relevant to the domains of performance we
have articulated. Readers are advised to review the entire checklist.

Note that in some cases a criterion incorporates a specific standard,
e.g., that face-to-face health assessments be provided with 60 days of
enrollment.

Domain 1. Creating a System with the Right Capacities

MCO is accredited by NCQA

MCO adheres to reporting requirements of HEDIS® 3.0

MCO implements a quality assurance and improvement plan

MCO uses HCFA QARI quality assurance measures

In building its network, MCO utilizes maximum patient to FTE primary
care ratios that take into account physician participation in multiple networks
and with commercial patients

MCO has providers available, including pediatric specialists, who are
no more than 20 minutes away from members in urban areas and 30 minutes away
from members in rural areas

MCO ensures that members whose primary language is not English and
members with special medical needs have access to primary care providers and
specialists qualified to meet their needs

Definition of medical necessity for behavioral health services is
consistent with Federal and State law and recognizes the role of the
member/family, least restrictive treatment setting and wraparound services

Domain 2. Providing Access to Needed Services

For persons with disabilities, chronic and/or complex conditions,
ability to choose a specialist as a primary care provider

For children with disabilities, chronic and/or complex conditions,
ability to choose a pediatric subspecialist as a primary care provider

Where the MCO network does not include an appropriate specialist, or
if there are fewer than two specialists or sub-specialists of a certain type,
access to out-of-network specialists

Availability of "standing referrals" to specialists for individuals
with ongoing treatment needs

Appointments for routine care available within ten days; appointments
for specialty care available within three weeks; appointments for urgent care
available within 24 hours

Emergency care available at nearest facility, even if not in network

"Prudent lay person" definition of appropriate use of emergency room
services is used

MCO specifies maximum in-office waiting times

MCO provides access to clinical studies

A simple process is in place to obtain prescription medications not
included in the formulary

Domain 3. Supporting Member Involvement in Decision Making and System
Improvement

MCO provides information on available providers and member rights and
responsibilities, including information on selecting a primary care
provider

MCO provides member information (verbally and in writing) at a
reading level set by the State, and in the recipient's primary language, and in
alternative formats including TTY and telecommunications devices, Braille,
large print and cassette

Each eligible person can choose his or her own primary care provider
from among the MCO's participating providers

Members can change primary care providers with cause at any time

Definition of medical necessity is clearly spelled out in contracts
and sub-contracts

MCO provides case management to facilitate necessary medical,
educational, social and other services

Domain 8. State of the Art Treatments

MCOs honor ongoing plans of care initiated prior to enrollment until
enrollee is evaluated by new primary care provider and a new plan of care is
established

MCO provides face-to-face initial health assessment for all new
members within 60 days of enrollment

Evidence to Support Validity and Reliability of the Criteria
Set:

These criteria are based on NHeLP's experience in providing advocacy to
people with disabilities, as well as other people on Medicaid. In some cases,
the criteria and standards are based on research evidence, but in most cases
the criteria have what could be called "face validity," i.e. those with
concerns about how health services are provided to persons with disabilities
believe that the conditions are necessary to the delivery of good care. It is
important to note that people concerned about people with disabilities may
disagree about the appropriateness of certain criteria. These criteria reflect,
as do all performance measures, values and expectations, which may differ from
person to person.

Populations and Settings in Which Criteria Set Has Been/Could Be
Used:

This criteria set was developed specifically to address the needs of
Medicaid eligible people with disabilities, including both adults and
children.

Written Documentation Available to Support Use of Criteria
Set:

The complete Advocacy Checklist is available from NHeLP's Website, the
address of which is http://www.healthlaw.org/checklist-disabilities.

Consultation Available to Support Use of Criteria Set:

NheLP staff would be available to discuss the checklist and its
development, but are not experts in the translation of criteria into formal
measures.

Limits on Use of the Criteria Set:

As noted above, these criteria do not represent fully developed
measures. Additional work is needed to develop methods for collecting data to
determine if the criteria are being met. In addition, many criteria represent
necessary but not sufficient conditions for high performance, and must be
supplemented by the use of other performance measures. Finally, work will be
needed to develop consensus on which criteria are appropriate in a given
context, and where necessary to specify standards as well as criteria.

Indeed, this accelerated attention has begun to
come full circle: many are now demanding approaches to accountability that
level the playing field, by measuring the performance not only of MCOs but of
FFS and PCCM systems as well.

Shoshanna Sofaer, Dr.P.H., served as the
Principal Investigator for this project. She was formerly the Director of The
Center for Health Outcomes Improvement Research, which closed on July 1, 1998.
Dr. Sofaer coordinated final production of the Resource Guide under the
auspices of the George Washington University's Center for Health Policy
Research. Inquiries to the George Washington University should be directed to
D. Richard Mauery, MPH at (202)530-2376. Dr. Sofaer is currently the
Schering-Plough Professor of Health Policy at the School of Public Affairs at
Baruch College, New York, where she can be contacted at (212)802-5980.

At this point, policymakers at higher levels
than the State agency, as well as legislators, may become involved.

This requires that the number of persons served
by provider groups is large enough to permit such comparisons.

Some of the measures included in the Resource
Guide come from the HEDIS® measurement system of the
National Committee on Quality Assurance (NCQA). NCQA has begun to identify and
provide standardized training to organizations it believes are qualified to
conduct HEDIS® measures.

HEDIS® 3.0 included a
Member Satisfaction Survey. We have not included measures from that Survey,
since in a recent decision, NCQA decided to collaborate with the Consumer
Assessment of Health Plans (CAHPS) project to develop a single survey of health
plan members. The version of the CAHPS survey which has grown out of this
collaboration is separately presented in the part 3 of this section.

"Quality Improvement in Managed Care: Summary of
Key Public Comments and Quality of Care Group Recommendations". NASHP, April 7,
1998. Available at http://www.nashp.org/pubs/qlty0001.htm. Accessed July 1,
1998.

The report also includes a discussion of
outcomes relevant to children and adolescents with serious emotional disorders.
These outcomes were developed after the Santa Fe Summit but mirror its
structure and underlying values. They are included in the Final Report cited
here.

American College of Mental Health Administration. 1997. The Santa Fe
Summit on Behavioral Health: Preserving Quality and Value in the Managed Care
Evaluation, Final Report. Pittsburgh, Pennsylvania: American College of
Mental Health Administration.

Booth, M. Look Before You Leap: Assuring the Quality of Care of
Managed Care Programs Serving Older Persons and Persons with Disabilities.
1996. Portland, Maine: National Academy for State Health Policy.

Goldman, S.K. and De La Cruz Irvine, M. 1997. Report of the Child and
Adolescent Panel for the Mental Health Managed Care and Workforce Training
Project. Center for Mental Health Services, Substance Abuse and Mental
Health Services Administration, and The Center for Mental Health Health Policy
and Services Research at the University of Pennsylvania Medical Center.

Harahan, M. 1997. Defining the Populations. Slide presentation
based on data from the 1994 National Health Interview Survey to the
AHCPR User Liaison Program Meeting on Managed Care and Persons with
Disabilities and Chronic Ilness, in Phoenix, Arizona, November 5, 1997.

Health Care Financing Administration. 1998. April 1998 Draft: Key
Approaches to the Use of Managed Care Systems for Persons with Special Health
Care Needs. Website: http:://www.hcfa.gov/medicaid/smd-snp.htm. Accessed
May 26, 1998.

Health Care Financing Administration Medicaid Managed Care Technical
Advisory Group and The Center for Vulnerable Populations. 1996. A Framework
for the Development of Managed Care Contracting Specifications for Dually
Eligible Adults. Portland, Maine: The National Academy for State Health
Policy.

Institute on Disability at the University of New Hampshire. 1998.
MCARE: Charting the Changing Tides of Managed Care and Long-Term
Services. Website: http://www.mcare.net/brochure.html. Accessed March 19,
1998.

National Academy for State Health Policy as part of the Quality
Improvement System for Managed Care (QISMC) Initiative. 1997. Public Review
Draft: Standards and Guidelines for Review of Medicare and Medicaid Managed
Care Organizations. Washington, D.C.: Health Care Financing
Administration.

National Health Law Program (NHeLP), Inc. 1998. Medicaid Managed Care
Contracts: An Advocacy Checklist for People with Disabilities. Website:
http://www.healthlaw.org/checklist-disabilites.html. Accessed March 19,
1998.

Newman, S.J., and Envall, K. 1995. The Effects of Supports on
Sustaining Older Disabled Persons in the Community. Washington, D.C.:
American Association of Retired Persons.

Office of the Assistant Secretary for Planning and Evaluation, Health
Care Financing Administration, and the American Association of Retired Persons.
1996. Beyond the Water's Edge: Charting the Course of Managed Care for
People with Disabilities. Conference materials from the meeting in St.
Michaels, Maryland, November 20-22, 1996.

Sofaer, S. 1995. Performance Indicators: A Commentary From the
Perspective of an Expanded View of Health. The Center for the Advancement
of Health and The Western Consortium for Public Health.

United Cerebral Palsy Associations, Inc. 1997. Report on the
Development of Indicators to Monitor the Performance of Systems of Acute Health
Care for Children with Disabilities. Washington, D.C.: U.S. Departments of
Education and Health and Human Services.

APPENDIX: COLLECTING DATA FOR A PARTICULAR
PERFORMANCE MEASURE: AN EXAMPLE

This example illustrates in some detail the process, resources and
issues in collecting data for a particular performance measure, the HEDIS® 3.0 childhood immunization measure. These data could
only be produced from claims/encounter data if:

All children who turn 24-months-of-age during the reporting period
were continuously enrolled from date of birth through their second birthday;
AND

All children received their immunizations from their MCO primary care
physician or another MCO practitioner; AND

The MCO practitioner correctly coded the specific immunization given
on the claim/encounter form, regardless of whether the practitioner was
directly compensated for the immunization and the serum.

However, recognizing that Medicaid children may have short periods of
enrollment, the HEDIS® 3.0 specifications require
that for a Medicaid child to be included in the at-risk population, the child
need only be enrolled for 11 out of 12 months in the second year of life. Thus,
the child may not have received any immunizations through the MCO in the first
year of life and the MCO may not have claims/encounter data for these services.
Additionally, even while enrolled in the MCO, the child may have received
immunizations through other providers, such as the health department, at
immunization health fairs, in a Head Start program, etc. Again, the MCO would
not have claims/encounter data. Lastly, the MCO may have difficulty getting
accurate, detailed claims/encounter information from its practitioners if it
does not reimburse its practitioners on a fee-for-service basis for these
services. For example, if practitioners receive free vaccine through the health
department, they only bill the MCO for administration of the vaccine and may
not be concerned about using the correct vaccine code when they are receiving a
standard fee for administration of the vaccine. Similarly, primary care
physicians who are paid on a capitation basis may not be highly motivated to
provide timely and accurate immunization encounter data since the data are not
directly related to reimbursement.

Consequently, almost all MCOs that report the HEDIS® 3.0 childhood immunization measures must use a
combination of claims/encounter data and review of medical records to collect
the data. Medical record review is costly because it is labor-intensive,
requires knowledgeable data abstractors, and involves data entry. In sum,
collecting data on performance measures is dependent on how MCOs are organized,
and on the sophistication and reach of their management and clinical
information systems. States may require, through contract specifications for
example, that specific data be provided by MCOs in a certain form. However, it
is important that all participants in performance measurement systems be
realistic about whether resources are available to collect data that inspire
confidence.