Wolters Kluwer Health
may email you for journal alerts and information, but is committed
to maintaining your privacy and will not share your personal information without
your express consent. For more information, please refer to our Privacy Policy.

Background: President's Emergency Plan for AIDS Relief (PEPFAR) has been criticized for its vertical or “stove-piping” structure, with resources targeting a specific disease rather than working to strengthen the underlying health system. This study aimed to evaluate whether PEPFAR activities were associated with system-wide improvements in both proximal and distal indicators of health systems strengthening.

Methods: The World Bank database provided 12 indicators of health systems strengthening that were analyzed for their relationship to PEPFAR. Poisson and linear regression models were used to estimate the time trend. We evaluated the PEPFAR impact on health outcomes by comparing the time trend in each of the above indicators between 2 time periods: from 1995 to 2002 (pre-PEPFAR) and from 2004 to 2010 (during PEPFAR).

Conclusions: The progressive scale-up of PEPFAR-supported activities was associated with consistent improvements in proximal indicators of health systems strengthening. It was also associated with improvements in broader measures of health system strength, most clearly life expectancy. Given the limited number of health measures available for this type of analysis, more data must be collected for other indicators to evaluate the effectiveness of the many multibillion dollar global health initiatives.

Funding was provided by the US Department of Defense via the Walter Reed Army Institute of Research.

The authors have no conflicts of interest to disclose.

The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Department of Defense, the Military HIV Research Program, PEPFAR, or the US Government.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's Web site (www.jaids.com).

INTRODUCTION

The co-epidemics of HIV and tuberculosis (TB) continue to present major public health challenges in Africa. Millennium Development Goal (MDG) 6 focuses on reversing the spread of HIV, malaria, and other diseases and achieving universal access to treatment for all who need it.1 Large global health initiatives (GHIs) have been founded to fight these global plagues,2 of which the US President's Emergency Plan for AIDS Relief (PEPFAR) is the best funded.3 From 2004 to 2010, PEPFAR disbursed $32.4 billion to implementing agencies concentrated in 15 focus countries, 12 of them in Africa.4,5 These efforts have come amidst partial success to stanch the HIV epidemic. Although more than 25 million people were newly infected over the past decade, the global incidence rate for HIV peaked in 1999 and is now clearly decreasing6. Similarly, global and African TB incidence peaked in 2004.7

Since its inception, PEPFAR has experienced several criticisms.9–11 Perhaps most importantly, PEPFAR has been criticized for failing to focus attention during its early years on strengthening the health systems of partner countries.12 Many in the global health community objected to PEPFAR's vertical or “stove-piping” structure, with resources targeting a specific disease rather than working to strengthen the underlying health system and so address the many different health issues that affect the same population.12,13 In time, PEPFAR has adapted to many of these criticisms.14 In 2008, Congress reauthorized PEPFAR with a new strategy focused on the following 5 goals: (1) improving sustainability, (2) expanding coverage, (3) bolstering partner government capacity to lead in-country efforts, (4) achieving and measuring a clear impact on health outcomes, and (5) integrating PEPFAR efforts with existing health infrastructure to maximally strengthen the health system.15

Systematic evaluations of the PEPFAR impact are ongoing, but published data quantifying its effect are sparse. A 2009 study examining the PEPFAR effect on health outcomes over its first 4 years suggested improvements in indicators proximally related to the program such as HIV mortality.11 Further analysis found that PEPFAR was associated with a greater reduction in all-cause mortality in focus countries compared with nonfocus countries and differentially within focus countries nearer those areas where PEPFAR operated intensively.16 However, it was not possible to discern whether these reductions in all-cause mortality were only because of lower HIV mortality or to broader health system effects. If PEPFAR had effects on health systems strengthening as well, other health measures more distally related to HIV treatment programs may also have been impacted, including those relating to child health (MDG 4) and maternal health (MDG 5) and to other prominent diseases (MDG 6). This has not been quantitatively evaluated, but qualitative data suggest that PEPFAR may be having some beneficial effects.17 The objective of this study was to quantitatively evaluate whether PEPFAR activities were associated with system-wide improvements in both proximal and distal indicators of health systems strengthening.

METHODS

Country Selection

All countries of sub-Saharan Africa were considered eligible for the analysis. We excluded nations for which the HIV epidemic was not generalized. “Generalized epidemic” was defined as HIV prevalence of more than 1% in antenatal clinics and a predominantly heterosexual mode of transmission. Thus, our study tested the same focus and control groups as a previous study.11 The PEPFAR focus countries and control countries are listed in Table 1.

Indicator Selection and Data Sources

We searched several global health databases for indicators related to health systems strengthening. These included the World Bank Databank,18 the World Health Organization (WHO) Global Health Observatory,19 and the United Nations UNdata site.20 We considered a health indicator to be eligible for our analysis if data were recorded for it for all or nearly all (defined as at least 83.3% of country-years) of the 41 countries in Table 1 for every year from 1995 to 2010. The World Bank database provided 12 non-HIV eligible indicators: 5 related to TB, 4 related to childhood vaccinations, and 3 broader measurements of national health—infant mortality rate (IMR), under-5 mortality rate (U5MR), and overall life expectancy at birth. The World Bank and WHO published identical data; we used the World Bank databank because of a simpler user interface. The indicators were grouped into proximal and distal indicators of health systems strengthening. The proximal indicators were defined as those with predominantly direct effects attributable to HIV treatment, which included TB and life expectancy. Distal indicators were defined as those with more predominantly indirect effects from increased healthcare funding, which included IMR, U5MR, and vaccination prevalence.

None of the databases contained complete data sets for indicators relating to maternal health, malaria, HIV-related behaviors, or health workforce size. Different countries reported values for various years, but none of these indicators were reported consistently enough across all 41 of our study countries to be used in our analysis.

Study Periods

We evaluated the PEPFAR impact on health outcomes by comparing the trend over time in each of the above indicators between 2 time periods: from 1995 to 2002 (pre-PEPFAR) and from 2004 to 2010 (during PEPFAR).

Statistical Analysis

SAS 9.3 (SAS Institute Inc., Cary, NC) was used for all analyses. We calculated the difference in the time trend for each outcome by PEPFAR country status and time period. Using the individual countries as the unit, we performed Poisson regression, estimating the intercept of the starting year and the slope over time for each outcome for both the focus and the control groups before and during PEPFAR. For all variables except life expectancy, we used 1 Poisson regression model for each outcome variable:

where yi was the expected count for country i and ni was the population of country i. The vector Xi represents [time (year) as continuous variable, time period as a binary variable (before 2003 or after 2003), and PEPFAR country status (also as a binary variable)]. The β represents a vector of coefficients and εi represents the error. The 3-way interaction term present in each model examines the time trends' difference by country status and time period. The SAS/GENMOD procedure was used to analyze the data.

For life expectancy, we assume the human being life time to be normally distributed. We therefore used the general linear model:

where yi was the expected mean lifetime for country i and Xi, β, and εi were defined as in the Poisson modeling. The SAS GLM procedure was used, controlled for year, time period, and PEPFAR status.

RESULTS

Demographic Comparison of Focus and Control Groups

The demographic characteristics for the focus and control groups are shown in the Supplemental Digital Content (see Table S1,http://links.lww.com/QAI/A383). The focus countries had a significantly higher average population and higher government effectiveness scores than the control countries. gross domestic product per capita in the focus countries was nearly double that of the control countries. There were no significant differences in control of corruption, political instability, or per capita disbursements from the Global Fund.

TB Indicators

The time trend patterns for TB incidence, prevalence, and mortality after the introduction of PEPFAR were similar (Fig. 1A–C). For TB prevalence, the peak in both the focus and the control countries occurred in 2003. These date coincide with the African and global TB incidence rate peak and are a few years after the African HIV prevalence peak. From 1995 to 2002, TB prevalence increased significantly faster in the focus countries than in the control countries. After 2003, prevalence stopped increasing in the control countries, remaining nearly constant through 2010, and this change in slope was significant (P < 0.0001). However, the improvement in the focus countries after the introduction of PEPFAR was even more pronounced. After 2003, the time trend for TB prevalence in the focus countries reversed from increasing to decreasing. The 3-way interaction was significant (P < 0.0001), implying a significantly greater improvement in the focus countries compared with the control countries (Table 2). The pattern for TB incidence rate and mortality rate followed a similar course. The 3-way interaction was significant for both of these indicators as well (P < 0.0001).

From 1995 to 2002, case detection of TB was comparable in the focus countries and control countries (Fig. 1D). During this time, the focus countries experienced a slow but steady increase in case detection, where the detection rate in the control countries stayed almost constant. After 2003, these trends continued, and from 2004 to 2010, case detection was higher in the focus countries than in the control countries (P < 0.03), although its rate of improvement had slowed. This slowing in the focus countries yielded a nearly significant 3-way interaction (P = 0.052). However, this finding was skewed by a very low rate of case detection in the focus countries in 1995. Repeating the analysis excluding 1995 showed that the PEPFAR effect on the slope was not significant (P = 0.46, data not shown).

The percentage of TB patients who successfully completed therapy showed a similar trend to that of TB case detection, with lower completion in focus countries before 2003 and higher completion after 2003 (Fig. 1E). The 3-way interaction was not significant.

Vaccinations (% of Children Aged 12–23 Months Who Received At Least 1 Dose)

The trends for all 4 vaccinations studied were similar (Fig. 2). From 1995 to 1999, control countries immunized significantly fewer children than focus countries. From 2000 to 2002, control countries improved their vaccination rates, where focus countries stayed roughly constant. After 2003, focus countries and control countries showed a similar rate of increase to each other, although progress was slower in the focus countries from 2004 to 2007 and then accelerated from 2008 to 2010. The focus countries therefore had a significantly larger increase in vaccination rate after PEPFAR implementation for 3 of the 4 vaccines (P < 0.05). However, excluding the years 1995–1999 eliminated this significance.

General Health Indicators

IMR and U5MR steadily decreased for both the focus and the control groups before and after the PEPFAR introduction (Fig. 3A,B). The rates of decrease were greater in the focus countries than in the control countries before 2003, and this gap widened further after 2003. Nevertheless, this acceleration after 2003 was not significant for either IMR or U5MR (P = 0.15 and 0.14, respectively).

Overall life expectancy in sub-Saharan Africa had been decreasing since the onset of the HIV epidemic in the 1980s and reached its nadir in 1996. In the control countries, life expectancy increased slowly from 1995 to 2002, and then the rate of increase doubled after 2003. In the focus countries, life expectancy did not start rising until 1998, showed a slow rate of increase until 2003, and then rose more sharply. The rate of increase in life expectancy accelerated significantly faster in the focus countries compared with the control countries after the PEPFAR introduction (P = 0.003) (Fig. 3C).

DISCUSSION

PEPFAR activities showed strong associations with improvements in proximal indicators of health systems strengthening, specifically TB indicators, and the distal indicator life expectancy. There were also nonsignificant associations with other distal indicators of health systems strengthening, namely IMR and U5MR. We also found surprisingly few quantitative indicators of health systems strengthening available to evaluate the effectiveness of PEPFAR.

Because of the strong protective effect of ART on TB incidence,21 the association between PEPFAR and improvement in HIV indicators was expected. Few studies exist that evaluate the effectiveness of PEPFAR in improving proximal indicators of health systems. PEPFAR has previously been found to be associated with a decline in HIV and all-cause adult mortality,11,16 but there was no analysis of morbidity and mortality related to TB or other proximal indicators. One study comparing the fractional change in TB indicators from 2000 to 2006 found little to no effect on these health outcomes attributable to PEPFAR,22 but that analysis did not account for the differing time trends of these 2 periods nor did it include data after 2006. Accounting for pre- and post-PEPFAR time trends and the additional implementation and expansion of PEPFAR programs after 2006 may account for the differing results of the 2 analyses. We chose to compare time trends because these take into account pre-PEPFAR health trajectories in both sets of countries. Our study found strong and consistent associations between PEPFAR and improvements in TB indicators. TB prevalence, incidence, and mortality all declined sharply after the PEPFAR introduction in the focus countries while declining much more slowly or not at all in the control countries.

This study shows that PEPFAR activities have been associated with a strong impact on TB control. In 2004, the interim WHO policy on collaborative TB/HIV activities was published23; it was updated in 201224; PEPFAR and other GHIs have funded the activities and interventions specified in these guidelines. Since the PEPFAR introduction in 2004 approximately 4% of spending has been allocated to TB control activities,5 and this spending seems to be yielding clear results. However, given the interrelatedness of the TB and HIV co-epidemics, TB burden can improve for several other reasons: improved HIV control, increased case detection, increased therapy completion, a stronger health system, and others. Case detection and treatment success rate did improve in focus countries during PEPFAR activities, suggesting a contribution from the increased resources directed by PEPFAR at TB, but the magnitude of these improvements was not significant in our analysis. Additionally, the observed decline of TB in focus countries can be partially explained by the PEPFAR HIV control activities. Antiretroviral therapy improves the immune status of people infected with HIV, lowering their susceptibility to TB,25 thereby lowering TB incidence and mortality even without additional TB control measures. Furthermore, in addition to funding HIV care, PEPFAR has also directed resources to coordination with national TB programs, nongovernmental organizations, financial partners, and WHO,26 Coordination efforts include expanding coverage, monitoring of health facilities to ensure adherence to standards, and providing technical assistance for clinics in their efforts against both TB and HIV.26 Such activities contribute to a strengthened health system, and it seems likely that all played a role in the improvements in TB indicators seen since the initiation of PEPFAR.

Even fewer studies exist that evaluate the effectiveness of PEPFAR in improving distal indicators of health systems strengthening. The same study cited earlier, which compared the fractional change in health indicators from 2000 to 2006 attributable to PEPFAR, found little to no effect on the health outcomes of IMR, U5MR, childhood vaccinations, or life expectancy.22 Our analyses found some evidence that PEPFAR contributed to the improvement of these indicators, but the association was only significant for life expectancy.

Perhaps the best measures of health systems strength are infant and child mortality because these reflect overall success in caring for these most vulnerable members of a population. Before PEPFAR, infant and under-5 death rates in focus countries were already decreasing more quickly than in the control countries, likely because of greater wealth and better governance, and possibly other unmeasured factors. There was a greater acceleration in the number of infant and child deaths averted after the PEPFAR introduction in the focus countries compared with the control countries, suggesting that PEPFAR may indeed be strengthening the health system enough to produce desirable results, but this acceleration was not significant. Whether PEPFAR countries continue to improve child health faster in the coming years or whether the control countries begin to catch up remains an interesting open question. It is clear that at a minimum PEPFAR is not harming broader health system activities, which is both an important an encouraging finding. However, criticisms that donor organizations coordinate poorly with each other13 suggest that streamlining of the existing aid structure could lend further benefit to the IMR and U5MR trends observed here.

Although our analysis detects a possible contribution of PEPFAR to improved vaccination rates, this may be an artifact because of the low rates of vaccination in control countries from 1995 to 1999, causing the pre-PEPFAR slope in the control countries to be positive, where the focus countries was flat. From 2000 to 2010, vaccination rates were comparable for all 4 vaccines. There is thus no compelling evidence that PEPFAR has had an impact on vaccination rates but also no evidence that PEPFAR activities have diverted resources or attention away from other vital public health programs.

Finally, life expectancy presents another important measure of a country's overall health system. In sub-Saharan Africa, the health systems were disproportionately overwhelmed by the HIV epidemic in the 1980s and 1990s and life expectancy decreased accordingly. Now that HIV incidence and death rates are declining, life expectancy is increasing across the continent. In PEPFAR countries, where the gains have been greater against HIV, TB, and child mortality, life expectancy is increasing faster than in the control countries. We detected a sharp and significant increase in focus countries after the PEPFAR introduction in 2003, which is consistent with other recently published data16 and may imply positive effects on overall health systems strength. However, it is not possible to conclude this with certainty because causes of death cannot be differentiated.

Given all the above data, there is no evidence that PEPFAR is diverting vital resources from other public health activities and may indeed be having a beneficial effect on overall health systems strength.

Limitations

This report has several limitations. First, collecting reliable TB data in Africa have been challenging, although the most recent revisions to the data sets in 2010 included an expert review to maximize their accuracy.7,28,29 Second, during the period analyzed, multiple other public health interventions were occurring contemporaneously. The most relevant of these potential confounders on HIV and TB indicators are the Global Fund2 and the Stop TB partnership.30 The Stop TB Partnership and the Global Fund began supporting TB treatment across Africa in 2001 and 2003, respectively, within 2 years of PEPFAR, and alongside these efforts many countries revamped their domestic TB control programs. Global Fund activities are not likely to have significantly skewed this study because the per capita disbursements by the Global Fund were similar between the focus and the control countries (see Table S1, Supplemental Digital Content, http://links.lww.com/QAI/A383). Other domestic (largely governmental public health activities) and international donor programs likely also played critical roles in the improvement of TB indicators; however, their funding was inconsistently reported to the WHO and therefore could not be accounted for in this analysis.7 Furthermore, even where such programs were not operating directly, there may be indirect carryover effects from surrounding programs in other countries and programs that impacted both PEPFAR and non-PEPFAR countries. Other important confounders include the effects of governance,31 preexisting health and health care status, and poverty. The small number of countries studied may have led to insufficient power to detect a difference when one exists. The short time interval since the beginning of PEPFAR implementation also may lead to a misclassification bias because of insufficient time for the program to have been fully implemented, and even if fully implemented, the time may have been too short for the program to have an effect. Furthermore, the data sources used are highly aggregated and undergo smoothing and reconciliation before publication. All these factors would tend to underestimate the impact of PEPFAR and thus support the findings in this study.

Perhaps the most important limitation of the study is the surprising lack of data on indicators of health systems strengthening currently available. We were only able to find 12 indicators reliably reported over the time period studied. Other resources may be available through the PEPFAR and GHI organizations (32–33), and additional sources may also exist that were missed in this analysis. Because of the importance of program evaluation in guiding future PEPFAR and other GHI interventions and expenditures, this is an extremely important finding of this study because programs cannot be evaluated if their performance cannot be measured. There is thus an urgent need for the collection and dissemination of more and better quality health systems data in Africa both in PEPFAR and in non-PEPFAR countries.

CONCLUSIONS

The progressive scale-up of PEPFAR-supported activities was associated with consistent improvements in proximal indicators of health systems strengthening such as TB morbidity and mortality. It may also have contributed to the improvement of distal indicators of health systems strengthening, such as IMR, U5MR, and life expectancy, although the associations were only significant for life expectancy. We found no evidence that the resources dedicated to HIV via PEPFAR have diverted resources from other vital public health programs. Improvements in governance, a remitting HIV epidemic, and increased international and country-specific health programs also contributed to the improvements in both proximal and distal indicators of health systems strengthening. Other health indicators that might have been analyzed for an impact by PEPFAR—from maternal health to malaria control to health workforce strength—could not be assessed because data were lacking for most African countries in most years. Fortunately, some of these data are beginning to be collected, and the push for an evidence basis for global health practice continues to increase. Still, global health efforts must further prioritize the collection of more comprehensive and reliable data sets on all the major health challenges of our time. Reliable sources of accurate data are essential to enabling evidence-based evaluations about the effectiveness of multi-billion dollar global health initiatives.