Program evaluations can play an important role in formulating goals, objectives, and implementation strategies for a variety of planning activities throughout the Department of Health and Human Services (HHS). Program evaluations also tell us whether our efforts are successful. While there are still gaps in what we know, we now are beginning to assemble a body of evaluative information that supports the way we craft our various goals and objectives and substantiates the effectiveness of strategies to achieve those goals and objectives. To illustrate this, we provide a discussion of the evaluative information that contributed to setting our goals and objectives. We also discuss program evaluations that demonstrate the effectiveness of implementation strategies that we will use. Finally, we provide a list of future evaluations that will provide additional insight into the effectiveness of our strategies and cumulative impact of our efforts (1). A goal by goal discussion follows.

GOAL 1: REDUCE THE MAJOR THREATS TO THE HEALTH AND PRODUCTIVITY OF ALL AMERICANS

SETTING THE GOAL/OBJECTIVES

A variety of statistical data on health trends in the United States contributed to the creation of Goal 1. For example, information from the National Vital Statistics Report provided the basis for establishing strategic objectives that address major causes of premature mortality and morbidity in the United States. Also useful was a wide variety of information on specific behavioral trends and incidences of disease available from national surveys and public health surveillance systems, such as the Behavioral Risk Factor Surveillance System, the Total Diet Survey (Food and Drug Administration) and the National Household Survey on Drug Abuse (2).

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

Available evaluation studies underline the effectiveness of a number of the strategies that the Department will use to achieve its objectives. For example, a key element in our strategy to reduce tobacco use among youth is the support of tobacco education programs. A recent evaluation of a major anti-tobacco media campaign in Florida demonstrated the effectiveness of education programs in preventing tobacco use, especially when targeted to younger persons. As a deterrent to tobacco sales to minors, the effectiveness of strategies to enforce the prohibition on sales to minors (Synar) is supported by recent evaluations; e.g., a 1998 Battelle study.

Similarly, evaluations, such as the review of the Child and Adolescent Trial for Cardiovascular Health (CATCH) program, point to the effectiveness of education programs in changing behaviors and attitudes toward diet and physical activity (Objective 1.3). Also, findings from Food and Drug Administration's (FDA) Food Label and Nutrition Tracking System indicate that consumers are reading, understanding, and changing their minds about food products as a result of FDA food labeling activities. The positive impact of consumer education on diets is also supported by the study The Effects of Education and Information Source on Consumer Awareness of Diet-Disease Relationships.

Examples of other evaluations that underline the effectiveness of our strategies in Goal 1 include: preliminary results from the National Cross-site Evaluation of High Risk Youth substance abuse prevention programs (Objectives 1.4 and 1.5); results from the National Treatment Improvement Evaluation Studies (NTIES) evaluation showing that treatment works (Objective 1.5); and evaluations of behavior counseling programs such as Project RESPECT (Objective 1.6).

FUTURE EVALUATIONS

Objective

Subject

Methodology

End Date

Agency

Objective 1.1

Reduce tobacco use, especially among youth

Evaluation of the effectiveness of population-based tobacco prevention and control programs

National/state prevalence surveys and demand models based on tobacco pricing and state policies

Ongoing with annual updates

Centers for Disease Control and Prevention (CDC)

Objective 1.2

Reduce the incidence and impact of injuries and violence in American society

Evaluation of state and local school-based programs designed to prevent chronic disease, including dietary patterns, physical activity, and overweight/obesity

Evaluation of interventions to increase proper restraint use for children

To be developed

Being developed

CDC

Evaluation of the effect of state and local residential smoke alarm legislation on smoke alarm use and reductions in injuries and fire related deaths

To be developed

Being developed

CDC

Objective 1.3

Improve the diet and level of physical activity of Americans

Evaluation of consumer knowledge of relationship between diet and health and dietary supplement labels

National Sample (telephone survey)

2002

FDA

Objective 1.4

Reduce alcohol abuse and prevent underage drinking

Evaluation of the effectiveness of the Hispanic/Latino media campaign to encourage dialogue between parents and children about substance abuse

Survey

2001

Substance Abuse and Mental Health Services Administration (SAMHSA)

Objective 1.6

Reduce unsafe sexual behaviors

Evaluation of state and local school-based programs designed to improve adolescent reproductive health, including prevention of HIV, other sexually transmitted diseases (STDs), and teenage pregnancy

Quasi- experimental school-based studies

Ongoing

CDC

Objective 1.7

Reduce the incidence and impact of infectious diseases

Analysis of influenza and pneumoccocal reports/data

Claims Database analysis

2002

Health Care Financing Administration (HCFA)

GOAL 2: IMPROVE THE ECONOMIC AND SOCIAL WELL-BEING OF INDIVIDUALS, FAMILIES, AND COMMUNITIES IN THE UNITED STATES

SETTING THE GOAL/OBJECTIVES

Data from a variety of national, state and program-specific sources provided valuable insights and information useful for the development of Goal 2, including development of the objectives and implementation strategies. For example, the national evaluation of welfare-to-work activities provided information on the effectiveness of the JOBS program in seven sites and was the basis for many of the reforms in the Personal Responsibility and Work Opportunity Reconciliation Act of 1996. Additionally, state and program administrative data were particularly useful in assessing trends and establishing the objectives for child welfare, abuse and neglect, early learning (Head Start) and child care. The Department's third annual report, America's Children: Key National Indicators of Well-Being, provided a secondary source of trend data for these objectives. Projections by the Department's micro simulation model, the Transfer Income Model (TRIM), were useful in testing alternative approaches and strategies for human services programs. Census data and data from surveys of the National Center for Health Statistics (CDC) contributed to the development of objectives that address trends in aging and long term care.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

In Goal 2, there are a number of implementation strategies that focus on identifying effective program practices and disseminating these to states and other service providers through Federal technical assistance and capacity development activities. Evaluative assessments of these efforts point to their value and argue for continuing to help identify and disseminate best practices as a key strategy to achieve our objectives in Goal 2. For example, there is substantial evidence that many welfare-to-work experiments supported by the Department have been adopted by states. Many elements of successful demonstrations were adopted in the welfare reform provisions of Temporary Assistance to Needy Families (TANF).

In other areas, we point to the success of bonus payments and technical assistance to help states reduce barriers to adoption (Objective 2.4). Recent evaluative data indicate substantial increases in adoptions in FY 1997 and FY 1998, indicating the success of these strategies. Also, the Government Accounting Office (GAO) recently found that the use of adoption incentives increases the likelihood of adoption of older and minority children and is cost effective (GAO/HEHS-97-73).

There is evaluative evidence going back a number of years that demonstrates the success of working through programs such as Head Start to link children to health care. These include A Descriptive Study of Head Start Health Services (a 1996 study of a representative sample of 1,200 children in 40 Head Start programs), current Head Start Program Information Report data, and Child Care State plans. Similarly, evaluations are beginning to show that providing access to quality childcare is effective in promoting healthy childhood development. (See the 1999 study, Access to Child Care for Low-Income working Families, and a National Institute of Child Health and Human Development study examining outcomes for children attending centers that meet professional standards.) Finally, results from the Family and Child Experiences Survey are beginning to show the positive impact of Head Start on child performance in cognitive and social skills, indicating learning readiness for kindergarten (Objective 2.3).

FUTURE EVALUATIONS

Objective

Subject

Methodology

End Date

Agency

Objective 2.1

Improve the economic independence of low income families including those receiving welfare

Evaluation of employment retention and advancement strategies; impact of welfare reform on child outcome measures; rural welfare to work strategies; and the effectiveness of serving special populations

Experimental and non-experimental

Variable 2001-2005

Administration on Children and Families (ACF)

Objective 2.3

Improve the healthy development and learning readiness of preschool children

Continuing evaluation of the impact of Head Start (FACES) on the social and cognitive progress of children

Surveys, observations, childhood assessments

Ongoing

ACF

Continuation ofA National Study of Child Care for Low-Income Families

Analysis of administrative data, surveys, interviews

2002

ACF

Objective 2.4

Improve the safety and security of children and youth

Continuation of a national longitudinal study of child welfare that looks at the effectiveness of services provided for families and children

Surveys, interviews

Ongoing in three to five year cycles

ACF

Assessment of child welfare outcomes in areas of safety, permanency, and child and family well-being

Surveys, interviews

Ongoing

ACF

Evaluation of the impact of family preservation and support services

Surveys, interviews

Date-phased

ACF

Objective 2.5

Increase the proportion of older Americans who stay active and healthy

Increase independence and quality of life of persons with long-term-care needs

Evaluation of the home and community-based services waiver program

Evaluation of multi-state demonstrations for integrating acute and long-term-care services

Descriptive analysis and consumer survey

Quasi-experimental using surveys, case studies and database analysis

2002

2002

Health Care Financing Administration (HCFA)

HCFA

Multi-state evaluation of dual eligibles demonstrations (cash and counseling demonstrations for making individuals more involved in planning and directing their community-based long-term-care services)

Control group

2003

HCFA

Objective 2.7

Improve the economic and social development of distressed communities

Evaluation of impact of Individual Development Accounts

To be determined

2005

ACF

GOAL 3: IMPROVE ACCESS TO HEALTH SERVICES AND ENSURE THE INTEGRITY OF THE NATION'S HEALTH ENTITLEMENT AND SAFETY NET PROGRAMS

SETTING THE GOAL/OBJECTIVES

Data from a number of health related surveys, such as the U.S. Census and Current Population Survey, were instrumental in helping set Goal 3 objectives that address problems such as the lack of access to health care insurance and services and health disparities. Examples include the Medical Expenditure Panel Survey (Agency for Healthcare Research and Quality), the National Survey of Health Insurance (Kaiser/Commonwealth), the National Vital Statistics System (CDC), policy briefs of the National Center for Cultural Competence, and the report of the Surgeon General on Mental Health. Also useful were program data from the Health Resources and Services Administration on medical shortage areas and cost data from the Agency for Healthcare Research and Quality (AHRQ) on the cost of services to persons with HIV/AIDS, (see AHRQ Publication No. 99-RO28). Information from the Department's Office of the Inspector General (OIG) contributed to the development of our objective on the integrity of the Medicare and Medicaid programs. Data from the Medicare Current Beneficiary Survey were useful for assessing issues related to the effectiveness of and access to Medicare services.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

A number of evaluative studies and other evidence illustrate the effectiveness of Goal 3 implementation strategies in increasing access to and effectiveness of health care services. For example, increasing the supply of physicians in under-served areas is a successful strategy for improving access to health care services. In addition, there is a similar impact where community health centers are located, and considerable evidence supports the success of Ryan White programs in increasing access to health care services for persons with HIV/AIDS. Finally, a continuing national evaluation of the strategy to support comprehensive community mental health services for children and their families shows improvements in a range of child outcome indicators (e.g., school attendance and behavior).

FUTURE EVALUATIONS

Objective

Subject

Methodology

End Date

Agency

Objective 3.1

Increase the percentage of the nation's children and adults who have health insurance coverage

Assessment of the impact of welfare reform on Medicaid

Evaluation of the State Children s Health Insurance Program

Analysis of claim and eligibility files
National analysis of enrollment and service use files and meta-analysis of state evaluations

Increase the availability and effectiveness of health services for children with special health care needs

Study on Children with Special Health Care Needs

State and Local Area Integrated Telephone Survey (SLAITS) interview mechanism

2002

HRSA, CDC

GOAL 4: IMPROVE THE QUALITY OF HEALTH CARE AND HUMAN SERVICES

SETTING THE GOAL/OBJECTIVES

Goal 4 development was substantially influenced by recent findings of the Advisory Commission on Consumer Protection and Quality in the Health Care Industry and the Institute of Medicine's report on medical errors. The Commission found that medical errors occur in hospitals, nursing homes, pharmacies, urgent care centers, and home care, and that all medical errors cost the nation approximately $37.6 billion annually. Also, the challenge to improve health care quality in the United States is well outlined in the Department's report, The Challenge and Potential for Assuring Quality Health Care for the 21st Century. In developing Goal 4, we also considered the continuing need to improve the quality of human services based on widely available trend data on the well-being of children and families in the United States.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

Although much of the initiative to improve care quality is new and evaluations of programs and activities are just beginning, some evidence of effectiveness has emerged. This is linked to the design of our strategies. For example, a key component of our quality improvement strategies is to develop evidence-based findings on effective health services and promote use of the findings. Evaluation findings of the Agency for Healthcare Research and Quality in 1999 (Publication No. 99-R043) and evidence from other studies (Publication No. 95-N012) support the conclusion that evidence-based research is making its way into practice and, in turn, is contributing to improvements in patient outcomes. Similarly, GAO testified in 1995 (GAO/T-HEHS-95-221) that AHRQ practice guidelines seemed to have a positive impact on patient outcomes.

Efforts to increase consumer and patient use of health care information are the focus of an ongoing evaluation of the Health Care Financing Administration education program-Medicare & You-which is designed to help beneficiaries make the best use of new benefits and program flexibility. This evaluation will continue to provide feedback on the program and guide future directions. Finally, ongoing assessments of the impact and effectiveness of the Mammography Quality Standards Act has shown the value of certification and inspection strategies as an effective means of addressing patient protections.

A key implementation strategy for improving the quality of human services programs is the development of a broad framework that includes quality data, performance measurement systems, and program evaluations. As policy and program design devolve to state and local levels, it is vital that these levels of government have reliable information on which to base their decisions and that the effects of different policy and program choices on quality and accessibility is understood. Documenting, understanding, interpreting, and facilitating the exchange of information and experiences among states is essential to providing high quality services that promote the well-being of families and children.

FUTURE EVALUATIONS

Objective

Subject

Methodology

End Date

Agency

Objective 4.1

Enhance the appropriate use of effective health services

Evaluation of Centers for Education and Research on Therapeutics to assess their effectiveness in translating and disseminating objective information on the appropriate and safe use of therapeutics

Citation analysis; other methodologies under development evaluation to commence in FY 2001

FY 2002 (first quarter)

AHRQ

Assessment of State Rules and Practice Regarding Collection and Reporting of Racial and Ethnic Data by Health Insurers and Managed Care Plans

Database analyses and selected site visits

9/2001

Office of Public Health Service (OPHS)

Objective 4.2

Increase consumer and patient use of health care quality information

Expanded evaluation of Medicare & You Handbook: 2000 (the national education campaign to help Medicare beneficiaries make choices among health benefits and plans)

Survey

Ongoing

HCFA

Evaluation of the use of web sites as an effective means of disseminating health care quality information

Analysis of consumer feedback data

2001

HCFA

Objective 4.3

Improve consumer and patient protection

Evaluation of the use of a quality indicator format in the end stage renal disease survey process
National Assessment of Culturally and Linguistically Appropriate Services in Managed Care Organizations (MCOs) Serving Racially and Ethnically Diverse Communities

Expert assessment

Survey of random sample of MCOs

2003

9/2001

HCFA

OPHS

Objective 4.4

Develop knowledge that improves the quality and effectiveness of human services practice

See Goal 2 evaluations

See Goal 2

See Goal 2

See Goal 2

GOAL 5: IMPROVE THE NATION'S PUBLIC HEALTH SYSTEMS

SETTING THE GOAL/OBJECTIVES

A variety of assessments of the capacity of the Public Health Service to identify and respond to health problems in the United States are available and support the need to strengthen the public health infrastructure. For example, a 1999 GAO study (GAO/HEHS-99-26) documented problems with laboratory capacity. This is supported by a Department assessment (1997), Public Health Workforce: An Agenda for the 21st Century, which singles out laboratory capacity as a pressing problem. Healthy People 2010 documents the need for better information technology. The Institute of Medicine (1988) published perhaps the most comprehensive view of the challenge, The Future of Public Health.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

Achievement of Goal 5 rests largely on the dual strategies of improving the surveillance and response capacity of federal, state, and local health agencies and improving the effectiveness and timeliness of communications throughout the public health system. Our adoption of these strategies is supported by assessments that are beginning to show successes in several areas. For example, assessments of efforts in the National Center for Health Statistics (NCHS) to improve the timely release of surveillance and survey data have been positive. Also, an assessment of the CDC Assessment Initiative to enhance the ability of state and local health departments to use data for policy making has been positive.

In the area of medical device safety, evidence suggests that FDA information dissemination about faulty medical products, transmitted through advisories, has a positive impact on product safety, although further review is indicated. There is evidence that the implementation of FDA Adverse Event Reporting System for Biologics has resulted in improved products through changes in product labeling. Betaseron is one example. The success of food safety consumer education strategies is supported by analytic findings in Background Research and Recommendations for the Food Safety Campaign. Finally, strategies to make drug prescription information more easily available and understandable seem to be successful, according to information obtained through our biennial National Survey of Prescription Medicine Information Received by Consumers.

FUTURE EVALUATIONS

Objective

Subject

Methodology

End Date

Agency

Objective 5.1

Improve the capacity of the public health system to identify and respond to threats to the health of the nation's population

Evaluation of how the public health infrastructure affects the performance of public health programs and interventions
Evaluation of the impact of new information systems, technology, and informatics training on public health practice

Evaluation of the impact of public health workforce development (training) on public health organization performance and the public health

Evaluation of the effectiveness of safety and public health advisory issuances

National Survey of Prescription Medicine Information Received by Consumers

Evaluation of consumer knowledge of food safety issues

Survey

Survey

Ongoing

Biennial
2002

FDA

FDA

GOAL 6: STRENGTHEN THE NATION'S HEALTH SCIENCES RESEARCH ENTERPRISE AND ENHANCE ITS PRODUCTIVITY

SETTING THE GOALS/OBJECTIVES

Almost every day, the American health science research community announces new discoveries that hold tremendous potential for the prevention and treatment of disease and injury. The promise of these discoveries argues for the nurture of the research infrastructure that produces the discoveries. As a result of this productivity, strengthening this country's health sciences enterprise has become and remains one of the strategic goals of the Department.

EFFECTIVENESS OF OUR IMPLEMENTATION STRATEGIES

Success in achieving Goal 6 will rely on how effectively our strategies nourish health research. One element is to facilitate the conduct of research and to move successful research into practice and products. Evaluative information supporting our direction is continuing to emerge. For example, attempts to accelerate the development of new medical products through streamlining the product application and review process has led to shorter review times and we are seeing new products approved under Fast Track processes. (Two such products for the treatment of HIV were approved in 1999.) Overall, streamlining efforts in response to the Prescription Drug User Fee Act (PDUFA) and the Food and Drug Administration Modernization Act (FDAMA) efforts are working to decrease product approval times, as reported in the FY 1999 Performance Report to Congress.

FUTURE EVALUATIONS

Objective

Subject

Methodology

End Date

Agency

Objective 6.1

Advance the scientific understanding of normal and abnormal biological functions and behaviors

Review and assessment of results achieved from funded research, conducted as a normal part of scientific planning and priority setting

Various mechanisms, involving numerous internal and external groups

Continuous

NIH

Objective 6.2

Improve our understanding of how to prevent, diagnose, and treat disease and disability

Review and assessment of results achieved from funded research, conducted as a normal part of scientific planning and priority setting

Various mechanisms, involving numerous internal and external groups

Continuous

NIH

Objective 6.4

Accelerate private sector development of new drugs, biologic therapies, and medical technology

Evaluation of statutory performance under PDUFA and FDAMA

Administrative Data Analysis

Ongoing

FDA

Objective 6.5

Strengthen and diversify the base of well-qualified health researchers

Survey Disclaimer

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0379. The time required to complete this information collection is estimated to average 5 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer.