4

Evaluating HRPPP Pilot Accreditation Programs

Launching the human research participant protection program (HRPPP) accreditation programs already in motion will take at least a year or two, and it will require at least another year or two of experience before a judgment about the costs and benefits of an accreditation strategy will be possible. Even as the pilot projects are being planned and implemented, however, forethought about how to evaluate them is in order.

Given the nature of the accreditation process, only limited quantitative data are likely to be available at the end of a 3- to 5-year pilot period. Some HRPPPs may have gone out of business. They may choose to contract with a fee-based independent institutional review board (IRB), to affiliate with larger institutions that have operating HRPPPs, or to stop conducting research altogether. The cessation of research because of an inability to demonstrate that research practices respect the rights and interests of research participants is not necessarily an undesirable effect.

The accreditation process will show how many organizations apply for accreditation and what fraction succeed. It is unlikely, however, that the accreditation process itself can produce data that would enable policy makers 5 years from now to make an informed decision about whether accreditation has, on balance, improved the HRPPP system. It is also unlikely that it can produce data on the cost of any enhancement compared with the achievements that could be made by alternative uses of the same resources. The answers to these questions are unlikely to be decided by quantitative data alone, so another evaluation strategy is needed.

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 89
Preserving Public Trust: Accreditation and Human Research Participant Protection Programs
4
Evaluating HRPPP Pilot Accreditation Programs
Launching the human research participant protection program (HRPPP) accreditation programs already in motion will take at least a year or two, and it will require at least another year or two of experience before a judgment about the costs and benefits of an accreditation strategy will be possible. Even as the pilot projects are being planned and implemented, however, forethought about how to evaluate them is in order.
Given the nature of the accreditation process, only limited quantitative data are likely to be available at the end of a 3- to 5-year pilot period. Some HRPPPs may have gone out of business. They may choose to contract with a fee-based independent institutional review board (IRB), to affiliate with larger institutions that have operating HRPPPs, or to stop conducting research altogether. The cessation of research because of an inability to demonstrate that research practices respect the rights and interests of research participants is not necessarily an undesirable effect.
The accreditation process will show how many organizations apply for accreditation and what fraction succeed. It is unlikely, however, that the accreditation process itself can produce data that would enable policy makers 5 years from now to make an informed decision about whether accreditation has, on balance, improved the HRPPP system. It is also unlikely that it can produce data on the cost of any enhancement compared with the achievements that could be made by alternative uses of the same resources. The answers to these questions are unlikely to be decided by quantitative data alone, so another evaluation strategy is needed.

OCR for page 89
Preserving Public Trust: Accreditation and Human Research Participant Protection Programs
Information of two kinds can better guide decisions about improving HRPPPs in general and the role of accreditation in particular. First, a research program is needed both to establish the current baseline (current practices in human research) and to study ways in which that baseline might be improved. Second, the committee believes that an evaluation process that is independent of the Association for the Accreditation of Human Research Protection Programs (AAHRPP), the National Committee for Quality Assurance (NCQA), and any other accreditation bodies that may emerge will be necessary. The committee recommends that federal agencies with a track record of evaluating HRPPPs, such as the U.S. Department of Health and Human Services (DHHS), monitor the accreditation pilot programs.
Recommendation 10: Begin Collecting Data and Assessing Impacts of Accreditation Now
DHHS should commission studies to gather baseline data on the current system of protections for human participants in the research that it oversees and to assess whether the system is improving over time.
Baseline data are needed on the following:
a taxonomy of research institutions: the number of institutions conducting research with human participants and the number of studies of different types (e.g., clinical trials, surveys, student projects, and behavioral studies) approved by their HRPPPs;
a taxonomy of IRBs: the number of IRBs and what fraction of them are primarily devoted to studies of particular types;
a taxonomy of studies with humans: the number and distribution of investigations with humans under way by type of study, for example, clinical trials of various stages, observational studies, cross-sectional and longitudinal surveys, and social science experiments;
the number of people involved in research and, among them, how many are involved in research with more than minimal risk;
the fraction of studies with more than minimal risk that have formal safety monitoring boards and how (and how well) those boards operate;
the type and number of inquiries, investigations, and sanctions by the Food and Drug Administration and the Office for Human Research Protections; and
the type and number of serious or unanticipated adverse events attributable to research.
DHHS should also commission studies of how the databases for existing clinical trials and other research resources could be used to assess how well the system of research protections is operating and, specifically, whether accredita-

OCR for page 89
Preserving Public Trust: Accreditation and Human Research Participant Protection Programs
tion is having measurable impacts (e.g., by comparing accredited and nonaccredited institutions or by comparing institutions before and after accreditation).
Other studies are also needed to bolster the nascent literature on how well research participants understand the studies that they join, which risks matter most to them, and what forms of informed consent are most effective. Several new initiatives to enhance clinical research in particular are under way, and the National Institutes of Health has initiated new programs to improve research monitoring. DHHS should evaluate these efforts not only for their primary purpose of improving clinical research but also for how they can improve HRPPPs.
The research pursued under this recommendation should have several uses. It will provide essential data on which to base policy decisions in the future. It will also point to ways in which the system can be improved. It may help assign priorities among strategies to improve the HRPPP system by pointing to strengths and weaknesses in the current system. It is likely to uncover and document problems in the current system, some of which are already known and perhaps others of which are not fully appreciated. Finally, it could reassure the public and policy makers about those aspects of the current system that are functioning well.
Recommendation 11: Initiate Federal Studies Evaluating Accreditation
The U.S. Congress should request an evaluation of accreditation pilot programs from the General Accounting Office. The Secretary of Health and Human Services should consider requesting a parallel evaluation from the Office of the Inspector General of DHHS.
An evaluation process that is independent of AAHRPP, NCQA, and other accreditation bodies can help policy makers decide on the value of accreditation as an improvement strategy several years hence. Without such an evaluation, Congress and the executive branch will be positioned little better than they are today to make prudent choices about how to improve HRPPPs in 5 years. Research pursued under Recommendation 10 can provide some baseline information, but it cannot substitute for a thorough evaluation of the accreditation pilot projects themselves. Furthermore, the evaluation efforts would benefit in several respects if they were initiated soon, while the pilot projects are getting under way. Evaluators could observe which organizations seek accreditation and which ones do not. They could also conduct interviews with organization officials who are making choices to find out why a particular choice was made and what they perceive the benefits or problems of HRPPP accreditation programs to be. If multiple accreditation bodies emerge, the evaluation should compare their effectiveness.
The evaluation methods are likely to be primarily qualitative, supplemented where possible by quantitative data. Interviews, surveys, “shadowing ” of IRB staff and accreditation site visit teams, and other methods used while the pilot

OCR for page 89
Preserving Public Trust: Accreditation and Human Research Participant Protection Programs
project is being launched would capture information that is valuable for judging its success or failure and that will otherwise be lost.
The evaluations should take costs of accreditation into account. Accreditation costs in comparable contexts vary over a wide range. The Lewin Group (1998) noted that accreditation for a mammography facility was $900 (plus $1,178 if a site visit was used) whereas accreditation of a hospital by the Joint Commission on Accreditation of Healthcare Organizations was $50,000 (plus $1,500 per inspector per inspection, usually involving three to four inspectors). The 5-year contract between the U.S. Department of Veterans Affairs (VA) and NCQA is $5.8 million (for 141 facilities, or just over $40,000 per facility, 40 of which are affiliated with major research centers), and very preliminary estimates by the nascent AAHRPP anticipated a cost of $15,000 to $20,000 per accreditation cycle (including a site visit to each facility) (David Korn, Association of American Medical Colleges, personal communication, February 2001). These costs are borne by the accredited body. Additional external costs borne by the institutions in preparing for and following up on accreditation are not covered in these estimates, but they may be even higher than direct costs.1
The HRPPP accreditation process should be evaluated not only according to whether it has improved protections for human research participants but also according to whether resources devoted to accreditation could be spent to equal or better effect on other ways to improve HRPPP oversight such as education, research monitoring, and improved feedback mechanisms. Evaluation should take into account both the costs of establishing a national accreditation system and the costs to applicant organizations (i.e., both direct and preparatory).
Once complete, evaluations from the General Accounting Office (GAO) or the Office of the Inspector General (OIG) of DHHS, or both, will have to be translated into recommendations for action by the federal government, accreditation bodies, federal and nongovernmental research sponsors, and organizations seeking accreditation. A comparison and synthesis of the findings would be especially important if different evaluations reach slightly different conclusions or make recommendations that differ in detail, which they are likely to do. The
1
The committee sought information about current costs of IRB operations and also about projected costs of accreditation. It judged the best data, such as those in a 1998 report commissioned by NIH (Bell et al., 1998), are too incomplete to form the basis for cost estimates. The AAALAC accreditation program, for example, has eight categories of fees for accreditation and annual maintenance, and a similar fee schedule will likely develop for HRPPPs. In light of the variety of organizations, the incompleteness of cost data, and the fact that the accreditation process outside the VA system has not been specified in any detail, the committee believed it would be premature to specify cost benchmarks now. Such benchmarks should emerge from pilot testing. Estimating overall costs is even more difficult, but will nonetheless have to be part of the evaluation. The value of the accreditation program as a whole will turn on its added value compared to its marginal costs.

OCR for page 89
Preserving Public Trust: Accreditation and Human Research Participant Protection Programs
National Human Research Protections Advisory Committee (NHRPAC), a new advisory body created with the mission of improving the HRPPP system, has the expertise to perform this task.
Final evaluation reports on the accreditation pilot projects are unlikely before 2005 or 2006, although interim reports may be useful in 2002 or 2003, on the basis of initial experience with the launch phase of the NCQA and AAHRPP accreditation pilot projects (after initial site visits, for example). NHRPAC's charter will expire in June 2002. For NHRPAC to receive and respond to GAO or OIG evaluations with a set of recommendations, its charter would have to be extended. Its authorized staff and funding of one and a half staff members would have to be augmented, at least transiently for 1 year, to perform this function.
Another logical receptor for the OIG and GAO evaluations would be an independent agency to oversee the protection of human participants in research in both the public and the private sectors, if the recommendation of the National Bioethics Advisory Commission to create such an agency is carried out by Congress. In the event that NHRPAC's charter has expired and no independent oversight agency has been formed, then the synthesis of evaluations would have to be carried out by an independent advisory committee created for that purpose or delegated to an existing nongovernmental organization.

OCR for page 89
Preserving Public Trust: Accreditation and Human Research Participant Protection Programs
This page in the original is blank.

Bookmark this page

Important Notice

As of 2013, the National Science Education Standards have been replaced by the Next Generation Science Standards (NGSS), available as a print book, free PDF download, and online with our OpenBook platform.