Frequently Asked Questions About CAHPS

Search or browse for answers to questions about the CAHPS program, patient experience surveys, and the CAHPS Database. Please send additional questions to cahps1@westat.com.

Search

Why are the target response rates different for Medicaid and commercial plans?

The Consortium sets a lower response rate target for the Medicaid population (40 percent) than for the commercial population (50 percent) because of the greater challenges associated with collecting data from Medicaid recipients (e.g., worse contact information, lower education level).

However, it is important to note that these rates are targets, not requirements. Many sponsors are able to produce usable data with lower response rates.

How many completed surveys are needed for analysis of the CAHPS Health Plan Survey?

The recommended number of completed questionnaires for the CAHPS Health Plan Survey is 300; that number is consistent with prior versions of the survey. The sample size needed to gather that many completed questionnaires depends on the anticipated response rate.

Why does CAHPS repeat the reference period in every question?

The reference period is the time period the respondent is being asked to consider when answering the question. Questions that fail to make reference periods explicit leave room for different individuals to interpret the items differently. In particular, without a specified reference period, respondents with varying enrollment histories might base their answers on different periods of time. A person who has been enrolled in a plan for 5 years could answer about the previous 5 years, while a person enrolled for 8 months could respond about those 8 months. To standardize the assessment of quality for different respondents, the CAHPS instruments use an explicit reference period.

Why does CAHPS order response options or questions in such a way that the negative wording comes first?

CAHPS surveys present the never-to-always response options in the order from "never" to "always." Studies have shown that respondents tend to be reluctant to use negative response options. Putting the negative responses first yields a better distribution of responses.

Why does CAHPS use a 0 to 10 rating scale and not an adjectival rating scale?

During the cognitive testing phases for the CAHPS Health Plan Survey, the development team examined four alternative rating scales:

Excellent, very good, good, fair, poor.

Very good, good, OK, not very good, not good at all.

Excellent, very good, good, OK, not very good, not good at all.

A scale from 0 to 10, where 0 is as bad as something can be, 10 is as good as something can be, and 5 is okay or average.

The team arrived at the following conclusions about four key issues related to rating scales:

1. Psychometrics. In general, having more response alternatives is better in terms of reliability and discrimination.

2. Appropriateness for self-administration and telephone use. An advantage of the 0 to 10 response task is that respondents have no difficulty in retaining awareness of all the response alternatives on the telephone. Respondents have more trouble with the task of recalling five or six adjectives.

3. Appropriateness for use in other languages. An advantage of the 0 to 10 task is the ease of translation of numbers from one language to another. The words "fair" and "poor" in particular have been found to be difficult to translate into Spanish in a way that retains equivalent distance between categories.

4. Respondent acceptance. When respondents were asked about their preferences among the various rating scales outlined above, the numerical rating was as acceptable to respondents as the adjectival ratings.

Thus, according to each of the criteria considered, the 0 to 10 rating scale compares favorably with any of the adjectival alternatives.

Why are the CAHPS materials available in only one foreign language (Spanish)?

The CAHPS program developed and tested the questionnaires in English and Spanish. The number of Spanish speakers in the United States was a factor in the decision to produce a Spanish-language translation of the surveys. According to U.S. census data, Spanish is the most common non-English language spoken in homes in the United States. A second factor was that the questionnaires and other materials could be translated into Spanish using terms that are understood by almost all Spanish speakers, including those who speak different dialects. This is not the case with other languages.

What is the rationale behind the groupings in the composite measures?

The composite measures for CAHPS surveys are designed to summarize specific categories of enrollees' or patients' experiences with health care providers and plans. Composite measures assist consumers by grouping together the results for two or more related questions, thereby reducing the amount of information contained in the report.

The CAHPS Consortium used focus groups to evaluate possible conceptual groupings and conducted cognitive interviews with consumers to evaluate interpretations of the resulting composite measures and their labels. Finally, we conducted psychometric analyses of field test data to assess the reliability and validity (correlations with global ratings) of these measures.

The CAHPS surveys use a two-column format because it reduces the number of pages in the surveys and is user-friendly. The two-column format has been tested with a variety of consumers including Medicare beneficiaries. Results show that respondents are comfortable using the two-column format to complete the CAHPS surveys.

Why don't CAHPS surveys collect information on technical quality?

Technical quality of care is usually assessed by measuring outcomes such as morbidity, mortality, and/or health-related quality of life, or by assessing process measures that have been shown in studies to be related to outcomes. In the absence of data about the relationship between processes and outcomes, care quality can be defined using expert consensus. Assessing technical quality accurately often requires detailed information about a patient's medical condition, as well as care processes that are not directly observed by patients and/or outcomes that are best measured a substantial period after the care is provided. Although patients can report accurately about certain processes (e.g., whether they received a flu shot), CAHPS surveys are usually not the best way to assess technical quality of care because most patients do not have all the necessary information to make such assessments. Such measures often require the use of information from administrative records and medical charts.

The CAHPS race and ethnicity items were developed by the Federal Office of Management and Budget in order to create standard measures for use by Federal agencies and others to collect uniform data on race and ethnicity. Race and ethnicity are often used for descriptive purposes in analyses and presentations of CAHPS data.

How do I know that the results of CAHPS surveys will be reliable and accurate?

Two different and complementary approaches are used during the development of a CAHPS survey to assess its reliability and validity:

1. Cognitive testing, which bases its assessments on feedback from interviews with people who are asked to react to the survey questions.
2. Psychometric testing, which consists of analyses of data collected using the questionnaire.

Cognitive testing provides useful information about respondents' comprehension of the questions, their ability to answer the questions, and the adequacy of the response choices. It also helps identify words that can be used to describe health care providers accurately and consistently across a range of consumers (e.g., commercially insured, Medicaid, fee-for-service, managed care, lower socioeconomic status (SES), middle SES, low literacy, higher literacy) and explores whether key words and concepts work equally well in both English and Spanish.

Field tests and psychometric analyses provide information about the items' reliability and validity. Many existing questionnaires about health care have been tested primarily or exclusively using a psychometric approach, but the CAHPS team views the combination of cognitive and psychometric approaches as essential to producing the best possible survey instrument.

The CAHPS surveys and support materials are designed to meet standards of informed consent and patient privacy consistent with institutional review board policies. Recommendations for recruitment, survey cover letters, and surveys provide the necessary and required information for participants to give informed consent. This information includes:

The sponsorship and goals of the research;

The benefits and risks of participation;

An estimate of the time it will take to complete the survey;

The names of all organizations who will have access to data or information that participants provide;

An assurance of confidentiality;

The name of a contact person; and

Information on how long direct identifiers will be retained.

These requirements apply specifically to the use of CAHPS surveys for research purposes. However, AHRQ's CAHPS Consortium expects all survey sponsors and vendors to adhere to high standards of privacy and confidentiality protection when conducting CAHPS surveys and handling data. CAHPS sponsors and surveyors are expected to

Inform respondents that participation is voluntary;

Take steps to prevent the unauthorized release of CAHPS sampling data and survey responses to third parties; and

Use CAHPS survey responses only for the purposes for which they were collected.

All research activities conducted by the Consortium to develop CAHPS instruments and reports meet the requirements of 42 CFR 46 (the Common Rule, which applies to research) and 42 CFR 64 (the Health Insurance Portability and Accountability Act [HIPAA], which governs the use of protected health information in operations and research). The CAHPS Database, which acquires non-identifiable CAHPS data sets for use by researchers and others seeking benchmarks for their own CAHPS data, reviews and approves the confidentiality and privacy provisions of all applications for CAHPS data before data will be released to applicants. Organizations requesting the dataset must submit Institutional Review Board (IRB) approval of the proposed research project, or a justification of why IRB approval is not needed for the proposed research.

Can we assist patients in completing their survey?

To ensure that the responses to a CAHPS survey reflect the respondent's own experiences with care, consumers should complete their own surveys. It is especially important that providers not assist their patients because of the possibility of bias.

Since some people cannot respond to a mail questionnaire, we encourage administration of the survey by telephone using a third-party vendor. We also provide Spanish versions of most questionnaires in order to overcome the most common language barrier.

The one exception to this guideline is the Medicare version of the Health Plan Survey, for which proxy respondents are allowed.

Why does CAHPS emphasize standardization?

Standardization is critical to supporting valid comparisons and benchmarking across health care settings and sponsors. This comparability is what makes the information from CAHPS surveys useful for quality improvement as well as public reporting. It also assures users of the results that the validity and reliability built into the instrument by the developers is maintained by the survey's sponsors. While we recognize and support the need for survey sponsors to customize the questionnaire to meet the needs of their organizations and markets, certain aspects of CAHPS surveys are standardized:

The instrument. The contents and format of CAHPS questionnaires are standardized so that everyone administering the survey is asking the same questions in the same way. However, sponsors are free to add additional questions following the CAHPS items to meet their own needs.

Data collection protocol. The protocol for fielding the survey is standardized so that everyone adopts the same approach to drawing the sample, communicating with potential respondents, and collecting the data. This is important because the method of survey administration can affect the results.

Analyses. CAHPS surveys include a set of analysis programs and instructions to minimize variations in how sponsors and vendors score and interpret the results of the survey.

Where can I find information about the CAHPS surveys administered by the Centers for Medicare & Medicaid Services?

The Centers for Medicare & Medicaid Services (CMS) implements several CAHPS surveys nationwide. The AHRQ CAHPS site provides only basic information about these surveys. All information, including survey instruments, administration specifications, and other instructions, is available from CMS.

From the beginning of the CAHPS program, the reading level of the CAHPS surveys has been a subject of attention because the surveys are intended for use across a wide range of literacy levels. As part of the development process, independent consultants have evaluated the questionnaires and recommended ways to lower the reading level. The questionnaires have also undergone extensive cognitive testing across all population groups to reach the desired reading level.

The Health Plan Survey is at a sixth grade reading level. While many Medicaid programs are mandated to provide written materials at the fourth grade reading level, the questionnaires necessarily include topics and word choices that could only be reduced to the sixth grade level. The reading level of the Clinician & Group Survey is seventh grade. Some of the supplemental items available for these CAHPS surveys are even more complex, and as a result have a higher reading level.

The CAHPS team encourages use of layout, spacing, and type styles that optimize the readability of the survey questions. However, if you anticipate that the reading level of the survey will be a problem for your sampled population, you may want to conduct the survey over the telephone.