Performance Plans for FY 2000 and 2001 and Performance Report for FY 1999

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

GPRA Goal 3: Foster translation and dissemination of new knowledge into practice by developing
and providing information, products, and tools on outcomes; quality; and access, cost, and use of
care. (HCQO)

Strategy

This phase of the cycle of research bridges the gap between the development of new knowledge and its
implementation in the health care system. AHRQ has taken its commitment to "ensure that the knowledge gained through
health care research is translated into measurable improvements in the American health system" and integrated it in its
approach to promoting the adoption and use of research findings.

Through an investment in demonstration projects,
public (Federal, State, and local government) and private-sector partnerships, and targeted dissemination activities,
AHRQ is focusing on closing the gap between what we know and what we do. We have named this focus "Translating
Research Into Practice."

Building on the previous 10 years of research findings, AHRQ will identify ongoing gaps between what we know now and
what we do in health care and will begin to close those gaps through research and demonstrations that develop and test
implementation strategies in different settings in the health care system. A major focus within this goal is identify ing
existing implementation strategies in use in health care settings and demonstrating their applicability to widespread
dissemination in other areas of the system.

Types of Indicators

AHRQ uses output indicators, with some process indicators, to assess its progress in the
translation and dissemination of research. The indicators regarding number of partnerships, attendees at User Liaison
Program meetings, or hits on the AHRQ Web site helps the Agency determine that what it produces is of use to major
audience segments. The Agency will evaluate the results of the GPRA plan indicators in combination with other
information such as details about what products were released, feedback from attendees at programs, where the hits are
on the Web site, and feedback from customers to manage and improve its dissemination efforts.

Frequently, the results of research are not readily implemented in the health care system without an interim step such as
the creation of a tool that facilitates use. A major focus for the Goal 3 indicators, therefore, is to look at the creation and
use of tools. The indicators for the Agency's investment in training helps the Agency track its success in furthering the
field of health services research by fostering new talent. The number of trainees funded are a reflection of Agency
commitment and the success of the training programs in attracting successful candidates. This data can be used in
combination with other information about individual trainees, their research projects, professional credentials of
professors and mentors, etc. to assess the overall success of the program.

Use of Results by AHRQ

The Goal 3 indicators are used to assess AHRQ's ongoing efforts to Translate Research Into Practice. The statistics on such things as usage of the Web sites, number of and
attendance at User Liaison Programs, and/or the production of evidence reports by the Evidence-based Practice Centers
are used to measure output—Is the Agency developing and disseminating the products needed by users?

AHRQ
combines these statistics with evaluations of customer satisfaction and the use and usefulness of the products in order to
assure quality as well as quantity. The Agency also uses the information to allocate resources, for example, providing the
staff needed to maintain and update the Web site, expand its capacity, and respond to user suggestions for improvements.
As previously stated in this report, the indicators of actual use of the research and/or products provides AHRQ with data
on its ultimate goal of getting research into use in the health care system. The volume of the use that we document is an
indicator of the total use and helps the Agency identify implementation issues.

Data Issues

Data collection for Goal 3 falls into two basic categories: collection through Agency data systems and
collection through routine program management.

AHRQ has implemented several computer-based reporting tools to
monitor usage of Agency information systems and websites. Accurate statistics are recorded periodically on the usage of
the National Guideline Clearinghouse™, Publications Clearinghouse, and various other Agency websites and systems using
commercially available reliable and accurate tools, e.g., WebTrends. These tools are used by many corporations and
government agencies nationwide to monitor usage and have been certified by various information technology testing and
review groups. This category includes information on categories of grants. Information on all grants, which can be word
searched, is included in the Agency Management Information System.

AHRQ tracks print media (newspapers, health care-related trade journals and newsletters, and consumer magazines)
news about or involving the Agency through a contractor—Burrelle's—one of the Nation's largest and oldest news clipping
services. Beginning in calendar year 2000, Burrelle's will begin monitoring online news services. AHRQ staff is currently doing this
task. AHRQ also monitors TV and radio news reports on selected studies in major markets around the United States
through another contractor, Video Monitoring Service.

The other statistics are maintained by Agency program staff during the normal monitoring of contracts and grants. Certain items, such as the release of a CONQUEST product, are
documented on the AHRQ Web site when ready so that consumers are aware of the availability. Other items such as the
statistics on the ULP program are monitored through the management of the support contracts, travel arrangements, and
other records kept in administering the program. Anecdotal information is verified with the primary source before being
used by the Agency in this report or for any other uses.

Previous Successes in Implementing Research

AHRQ research has identified numerous opportunities for improvement in the quality of care. Some recent findings from
AHRQ research include the following:

Schizophrenia PORT

The Schizophrenia PORT produced the evidence needed for the development of treatment recommendations which have
been adopted by Massachusetts in a statewide quality improvement program. Initiated by the State Mental Health
Director, all treatment facilities will use the recommendations to guide their management of schizophrenic patients. The
National Alliance for the Mentally Ill (NAMI) has also used the recommendations to develop a consumer booklet offering
advice to patients and family members, which was disseminated to members nationwide.

One section of the treatment recommendations developed through the work of the Schizophrenia PORT is focused on
Assertive Community Treatment (ACT) programs, a multi-disciplinary team approach that shares caseloads and offers
24-hour mobile crisis teams, assertive outreach for treatment in the community, individualized treatment, medication,
rehabilitation and support services. HCFA has issued a letter to all State Medicaid Directors endorsing the use of ACT
programs and confirming Medicaid coverage of additional costs if this treatment model is implemented. Again, NAMI has
launched an initiative to promote these programs in the remaining 25 states that have no similar program.

Medical Errors

AHRQ's medical errors study by Leape helped influence three major facilities in the Boston-based Partners HealthCare
System as they focused on reducing medication errors:

Massachusetts General Hospital.

Brigham and Women's Hospital (where the AHRQ study was conducted).

Dana Farber Cancer Institute.

These facilities have added
special software to their computerized information systems to reduce medication errors. Called the "Physician Computer
Order Entry," the program allows doctors to enter their medication orders, including dosage, route and frequency, directly
on computer terminals, thereby eliminating handwritten orders. The system also alerts doctors when an order contains a
possible error, such as a potential drug interaction or allergic reaction by the patient.

A test of the software at the 714-bed Brigham and Women's Hospital found that it decreased the rate of serious
"nonintercepted" errors—mistakes that could have or did cause an adverse drug event and which were not caught before
reaching the patient—by more than half. In addition to protecting patients, the new software is estimated to save the
hospital between $5 and $10 million annually, even after accounting for development, start-up and maintenance costs.

GPRA Goal 3 Fiscal Year 1999 Results

Formation of a minimum of five partnerships to support dissemination of AHRQ products through intermediary
organizations, such as health plans and professional organizations.

Results

Thirty public-private and public-public partnerships were formed in Fiscal Year 1999. See details below.

Public-Private Partnerships

Put Prevention Into Practice (PPIP)—Materials to support a national campaign to improve the delivery of clinical
preventive services such as screening tests, immunizations, and counseling for behavior change. PPIP materials include
a clinicians handbook and health guides for adults and children. Select for PPIP Materials.

Staying Healthy at 50+—HRSA and AARP are partnering with AHRQ and have developed a PPIP Personal Health Guide for Adults Over 50,
which was launched at the Healthy People 2010 conference on January 25, 2000.

Quality Navigational Tool (QNT)—an interactive tool designed to help people use evidence-based information on quality
and to take a more active role in their health care.

Newsletters and Web sites—Fourteen companies/organizations disseminating the information to their employees, including it in their newsletters or on Web
sites. Examples:

Midwest Business Group in Health.

United Parcel Service (UPS).

Safeway.

Henry Ford Health Plan.

Blue Cross/Blue Shield of Michigan.

IBM.

Erie Insurance.

National Consumers League.

Smoking Cessation—clinical practice guidelines for physicians issued in 1996 that provide evidence-based information on
how to help patients stop smoking and patient brochures containing recommendations on how to stop smoking.

Fourteen companies/organizations reprinted and disseminated smoking cessation materials based on the guideline. Examples:

The National Medical Association.

Pharmacy Council on Tobacco Dependence (PCTD).

The Utah Tobacco Prevention and Control Program.

American Academy of Pediatrics.

Michigan Department of Community Health.

American Cancer Society.

Additionally, all evidence reports and technology assessments are undertaken only when partners have been identified to
take the findings and use them in developing a practice guideline or some other tool that will facilitate their use in the
health care system.

Objective 3.2: Maximize dissemination of information, tools, and products developed from research
results for use in practice settings.

Indicator

Number of hits on the Web site (Baseline: 2.9 million per year in 1997, nearly triple the hits in 1996.)

Results

15.5 million hits.

Indicator

Number of inquiries handled on Web site. (Baseline in Fiscal Year 1997—1300; in Fiscal Year 1998—2500.

Results

2,950 inquiries.

Indicator

Number of uploaded documents. Baseline in Fiscal Year 1997—950; in Fiscal Year 1998—1450).

Results

4,000 files/docs uploaded.

Indicator

Number of State and local governments trained in the understanding and use of health services research
findings through User Liaison Program (ULP) Workshops.

Number of ULP meetings held. Baseline—Meetings held. 10 held in Fiscal Year 1997; 9 held in Fiscal Year 1998; 12 scheduled in Fiscal Year 1999.

Results

Thirteen 2 1/2-day national workshops were held: State and local health policymakers from all States were invited to attend 12 of these and only State and territorial legislators were invited to
attend 1.

Two 1 1/2-day national seminars were held: one included State and local health policymakers from all States and the other included only State rural health directors, directors of Aging, and one
other official with a rural or aging focus from 10 States in HHS regions V and VII.

One 1-day "Masters" seminar was held that included only very senior State health officials entitled, "What Can States Do to Foster Reengineering of the Health Care Delivery System."

Two 1-day "State-specific" workshops were held to which only State representatives from the organizing State were invited to attend:

"Vermont: Uniting for Health Care" cosponsored by the Vermont Division of Health Care Administration, the Vermont
Employers Health Alliance, the Vermont Program for Quality in Health Care, and AHRQ.

States represented. Baseline—Fiscal Year 1997 and 1998—all 50 states and Puerto Rico.

Results

48 States plus D.C., Puerto Rico, Virgin Islands, Micro Polynesian Islands, and Guam. (Hawaii and North Dakota are
the only two States not represented.)

Indicator

Number of hits on National Guideline Clearinghouse™ (NGC) with analysis of use by page, type of guidelines
accessed, whether the guideline was downloaded, and linkages to other sites.

Results

The measures listed in the original performance plan were developed prior to the development of the NGC contract. These were changed to indicators that provide the
Agency with better indicators of actual usage.

Definitions:

Hit: Any connection to an Internet site, including online images and errors.

Request: Any hit that successfully retrieves content.

Visit: A series of consecutive requests from a user
to an Internet site.

User: Anyone who visits the site at least once.

Total users: 329,715.

Average visit/user: 2.24.

Average users/organization: 9.68.

Average number of requests/user: 23.57.

Number of hits: 13,590,013.

Number of requests: 7,771,095.

Average number of requests/visit: 10.53.

Number of organizations: 34,064.

Number of U.S. organizations: 10,045.

Number of Canadian organizations: 158.

Number of International organizations: 3, 627.

Unknown: 20,227.

Indicator

At least five purchasers/businesses use AHRQ findings to make decisions.

Results

Evidence Reports:

Use of Erythropoietin in Hematology/Oncology: The Health Care Financing Administration will revise its coverage
decision guidance on the topic based on the evidence report.

Criteria To Determine Disability In Patients with end-stage renal disease (ESRD) (ECRI EPC): The Social Security Administration will use to
determine if more research is needed and if its coverage decision guidance on this topic requires revision.

Criteria for Referral of Patients with Epilepsy: Will be used by the Centers for Disease Control and Prevention for
inclusion in a clinical practice guideline.

Diagnosis and Treatment of Dysphagia: Department of Veterans Affairs are considering initiating a study to fill some
of the gaps in the research identified in this report. The Health Care Financing Administration will revise its coverage
decision guidance on the topic based on the evidence report.

Testosterone Suppression Treatment for Prostatic Cancer: the Health Care Financing Administration will use to update
its coverage decision guidance on this topic.

Consumer Assessment of Health Plans Surveys (CAHPS®):

The following are examples of organizations using CAHPS® to inform consumers' choices of health plans:

In all, there are a total of 25 States functioning as a purchasing agency for its employees or for Medicaid beneficiaries
have used CAHPS® Those not listed above include: Alaska, Arkansas, California, Georgia, Lousiana, Massachusetts,
Michigan, North Carolina, Oregon, Pennsylvania, Utah. CAHPS® has been used in a total of 41 States when those for which it has been used by the U.S. Office of Personnel
Management are included.

Objective 3.3: Develop and facilitate the use of new tools, talent, products, and implementation methodologies
stemming from research portfolio.

Indicator

Evidence-based Practice Centers (EPCs) produce a minimum of 12 evidence reports and technology
assessments that can serve as the basis for interventions to enhance health outcomes and quality by improving practice
(i.e., practice guidelines, quality measures, and other quality improvement tools). At least four reports are being used by
customers to develop practice guidelines or other interventions. Baseline in Fiscal Year 1998—12 reports produced. Fiscal Year 1999 will
be the first year any interventions will be in development based on the reports.

Results

In Fiscal Year 1999, 10 evidence reports were published and three more were "in press" at the end of the fiscal year.
Thirty additional reports are currently under development. Nineteen evidence reports are being used to develop clinical
practice guidelines by organizations such as the American Psychiatric Association, American Academy of Pediatrics,
American College of Obstetrics and Gynecology, American Academy of Family Physicians, the Consortium for Spinal
Cord Medicine, American Academy of Cardiology, and American Heart Association.

Indicator

The AHRQ software product, CONQUEST 2.0, was released in Fiscal Year 1999 and contained new measures, including
measures for new conditions, and updated measures. A contract was awarded to create Web-based product for more timely
updating of information contained within the product.

Results

CONQUEST 2.0 was released in March 1999. Over 3000 hard copies of the product have been distributed
and also additional copies have been downloaded from the Web. The contract due to be awarded in Fiscal Year 1999 (the
National Measures Clearinghouse) was canceled in order for the Agency to better assess the future integration of the
Web-based CONQUEST product with the AHRQ National Guideline Clearinghouse™ effort. The contract to create a Web-based product is expected to be awarded by September 30, 2000.

Indicator

Funding of a minimum of five major projects that will develop products, tools, or methodologies for
implementing research findings into practice in significant segments of the health care system (i.e., potential to be
generalizable across health care systems, provider-types, or clinical areas.)

Results

Three enhancements to the CAHPS® instrument are underway: Group Practice Level CAHPS®, Medicare Disenrollment CAHPS®, Nursing Home CAHPS®. Three small business innovative
research grants were funded. They will develop:

A system to automate the management and delivery of clinical
preventive services using an integrated approach.

A home-based cardiac rehabilitation program utilizing the Internet as the primary link between case
managers, patients, and family members.

Also, a number of grants funded under the Translating Research Into Practice
initiative contribute to this indicator. Examples include grants that will:

Validate a clinical guideline for community-acquired pneumonia one.

Implement a computer-based health support systems.

Explore methods for translating research on pain management into clinical practice with a specific focus on elderly
hospitalized patients.

Study methods to improve and increase screening for Chlamydia.

Develop and test methods to evaluate the efficacy of acupuncture treatment for major depression during pregnancy.

Develop analytical tools and methods for performing meta-analysis of findings from clinical studies that exhibit
substantial heterogeneity to estimate treatment effects. (The findings will be useful to the Evidence-base Practice Centers
and other groups responsible for analyzing data and providing evidence reports.)

Develop patient-centered methods to assess the effectiveness of treatments for chronic neurologic diseases.

Indicator

At least two new tools, products, or methodologies become available from projects funded between Fiscal Year 1993
and Fiscal Year 1996. Baseline: Fiscal Year 1999 results.

Results

HCUPnet (Health Care Costs and Utilization Project) is now available for public access on the Agency's
Web site. HCUPnet allows users to tailor an online query of HCUP's National Inpatient Sample (NIS), the largest all-payer
inpatient database in the U.S. (Select for more detail.)

Eleven of the 22 Statewide Inpatient Databases (SID) from HCUP are now available from a single point of access,
under the auspices of AHRQ. Prior to September l999, the only means to access SID data was to approach each
HCUP partner state on an individual basis, determine if the data organizations released their SID, obtain information
about state-specific application processes, and successfully complete the application processes. (Select for more detail.)

Three products developed through the Small Business Innovative Research contracts:

Johnston Zabor and
Associates developed SmartChoice to help employees choose health insurance plans. It has been purchased by
several large employers. OPM and NIH also purchased this product and developed a demonstration Web site that
helped Federal employees in the Washington-Baltimore area choose health insurance plans during the Fiscal Year 1998 open
season.

Abacus developed a workbook and video in English and Spanish to assist low income workers choose
health plans. The materials are available commercially and have been integrated into Abacus' benefits management
services.

Benova developed a computerized decision tool to help Medicaid beneficiaries choose health plans. The
tool is available commercially as a stand alone product or can be included in Benova's Medicaid enrollment programs.

Indicator

Support a minimum of 150 pre- and post-doctoral trainees. Baseline: 150 trainees funded per year.
Commitment is to maintain the current level of support in Fiscal Year 1999.

Formation of a minimum of 5
partnerships to support
dissemination of AHRQ products
through intermediary organizations,
such as health plans and
professional organizations.

Baseline in Fiscal Year 1999: 30
partnerships used to disseminate
materials.

See Objective 3.2

Fiscal Year 2000 Objective 3.2: Maximize dissemination of
information, tools, and products
developed from research results
for use in practice settings.

Changed To:

Fiscal Year 2001 Objective 3.1: Maximize dissemination of
information, tools, and products
developed from research results
for use in practice settings.

Web site:

Number of hits on the Web site.
(Baseline: Fiscal Year 1999 - 15.5 million hits.)

Number of inquiries handled on
Web site. (Baseline in Fiscal Year 1999 -
2950.)

Number of Uploaded documents.
Baseline in Fiscal Year 1999 - 4000.)

Reports from user surveys on
how the information requested was
used.

User Liaison Program:

Number of meetings held.
(Baseline - 13+ meetings held in
Fiscal Year 1999. See details of 1999 results.)

Number of State and local
governments trained in the
understanding and use of health
services research findings through
User Liaison Program (ULP)
Workshops. (Baseline - 834
attendees in calendar year 1999).

Reports from annual participants
on how the information was used in
decisionmaking.

National Guideline
Clearinghouse™ (NGC):

Statistics on usage of National
Guideline Clearinghouse including
number of hits, requests,
organizations, and total users.
(Baseline: See Fiscal Year 1999 results for
details.)

Use of research findings:

At least 10 purchasers/businesses
use AHRQ findings to make
decisions.

Web site:

Same indicators used.

User Liaison Program:

Same indicators used.

Partnerships:

At least five public-private
partnerships are formed to
implement research findings for
decisionmakers. Budget:
Commitment Base.

Formation of a minimum of 10
partnerships to support
dissemination of AHRQ products
through intermediary organizations,
such as health plans and
professional organizations.
Budget: Commitment Base.

Objective 3.3: Develop and
facilitate the use of new tools,
talent, products, and
implementation methodologies
stemming from research
portfolio. Fiscal Year 2000 Priority (3),
"Translating Research into
Practice," focuses on the
translation and dissemination of
research findings, products, and
tools to foster adoption and use
in health care settings.

Objective 3.2 in Fiscal Year 2001: Develop
and facilitate the use of new
tools, talent, products, and
implementation methodologies
stemming from research
portfolio.

Demonstration of use of at least three
AHRQ research findings in
systematic efforts to Translate
Research Into Practice. Baseline:
Under development.

Funding of a minimum of five major
projects that will develop products,
tools, or methodologies for
implementing research findings into
practice in significant segments of
the health care system (i.e.,
potential to be generalizable across
health care systems, provider-types, or clinical areas.) (Baseline:
Under development.)

At least two new tools, products, or
methodologies become available
from projects funded between Fiscal Year
1993 and Fiscal Year 1996. (Baseline: 16
projects identified in Fiscal Year 1999.)

Support a 5-percent increase,
at a minimum, in number of pre-
and post-doctoral trainees.
(Baseline: 167 trainees funded in
Fiscal Year 1999.)

Evidence-based practice centers
(EPCs) will produce a minimum of
12 evidence reports and
technology assessments that can
serve as the basis for interventions
to enhance health outcomes and
quality by improving practice.
Budget: Commitment Base.

Support a minimum of 165 pre- and
post-doctoral trainees. Budget:
Commitment Base.