Structured Abstract

Objectives:

The Global Health Evidence Evaluation Framework project is aimed at developing an evidence framework for the purposes of informing efficacious, effective, sustainable global health programs at the community and scale level.

Data sources:

Literature search of published and gray literature, input from a multidisciplinary Technical Expert Panel (TEP).

Methods:

With input from the TEP we identified six existing evidence frameworks for public health/global health interventions or programs and applied these frameworks to the evidence bases for three exemplar interventions chosen to represent a diverse set of global health programs or interventions: household water chlorination, prevention of mother-to-child transmission of HIV, and lay community health workers to reduce child mortality. Review of the findings with the TEP identified an important gap as the reporting of information about the implementation of the intervention or program. We identified three existing criteria sets for implementation reporting, and selected from them 10 criteria that covered areas identified by the TEP. We pilot tested these 10 criteria on three published articles of effectiveness for each of the three exemplar interventions.

Results:

Assessing the same body of evidence yields different conclusions regarding strength of evidence depending on which framework is used, some of which were extreme (with the same evidence base assessed as “strong” in one framework and “low quality” in another framework). All frameworks focus on efficacy and/or effectiveness with most attention going to the allocation method of study participants to the intervention (randomization versus other method) in assessing study quality. Many fail to consider implementation issues, and none explicitly assessed costs or sustainability at the community or scale level. Incorporating insights from the frameworks for implementation criteria helped to address some of these gaps, yet our pilot test of 10 implementation reporting criteria on nine published studies showed great variability both within and across studies. Some criteria were classified as having “good” reporting in almost all studies (criteria about the setting of the study, the characteristics of the recipients and the mode of delivery of the intervention or program) while others were reported on rarely or never (the rationale for the intervention, the costs of the intervention, an assessment of the population needs, and two criteria about barriers and facilitators of the implementation). Similarly, two articles had “good” or “fair” documentation for greater than 75 percent of criteria, while three articles had “poor or none” documentation for more than 50 percent of criteria.

Conclusions:

Existing frameworks for the assessment of public health evidence do not deliver key pieces of information to inform best practices for community and large-scale global health programs, with the lack of information about implementation and sustainability being an important identified gap. In a pilot study, our application of existing criteria for the reporting of implementation information shows great variability within and across published global health intervention studies. The reporting of implementation information using criteria that have broad support across diverse stakeholders and that can be reliably assessed would provide policymakers a stronger evidence base upon which to make decisions.

This report is based on research conducted by the Southern California Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10062-I). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.

None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.