Non-Randomised and Quasi-Experimental Designs

Often, random allocation of the intervention under study is not possible and in such cases the primary challenge for investigators is to control confounding.

Members of the Centre for Evaluation organised a multi-disciplinary symposium in London in 2006 to discuss barriers to randomisation, review the issues, and identify practical solutions. The following two papers summarise the arguments presented, drawing on examples from high- and low-income countries:

Aurélia Lépine has used difference-in-differences, synthetic control, and propensity score matching in her research and her expertise also includes regression discontinuity, and instrumental variables. Examples of her work includes an impact evaluation of the free care policy in Zambia using synthetic control, propensity score matching combined with difference-in-differences. Free primary care in Zambia: an impact evaluation using a pooled synthetic control method. Currently, she is working on evaluating the registration programme for sex workers with HIV/AIDS in Senegal using propensity score matching.

James is currently undertaking a PhD at LSHTM as an MRC Population Health Scientist looking into the application of interrupted time series designs for the evaluation of public health interventions.

_

Difference in Differences

This method is used to evaluate the impact of interventions that are non-randomly allocated to a sub-set of potentially eligible places. The change in the outcomes in places that got the intervention (the ‘difference’) is compared to the change in the outcomes in the places that did not get the intervention: hence the difference-in-the-differences.

This approach requires data from before and after the intervention is delivered, in places the do and do not get the intervention, and is often estimated as the interaction between the change over time and the allocation group (i.e. whether or not a place got the intervention) in a regression model.

It is possible that the places that receive the intervention are different at baseline from the places that do not receive the intervention in terms of the outcome of interest, and this method accounts for this possibility. However, the method assumes that in the absence of the intervention the change over time in the outcome of interest occurs at the same rate in the intervention and comparison places – this is often referred to as the ‘parallel lines assumption’. Therefore, while the method can account for differences at baseline, it cannot account for a varying rate of change over time that is not due to the intervention. This assumption cannot be directly tested since it is an assumption about the counterfactual state: i.e. what would have happened without the intervention, which was not observed. Researchers can look at trends in other related outcomes, or trends in the outcome of interest before the intervention started, to try to find evidence that supports the assumption about the trends that they cannot actually see.

In the following paper, Tim Powel-Jackson and colleagues investigated the effect of a demand-side financial incentives intervention to increase the uptake of maternity services in India using difference-in-differences, and a number of diagnostics to assess the assumptions underlying the method:

_

Regression Discontinuity

Regression discontinuity is used to evaluate the impact of interventions when allocation is determined by a cut-off value on a numerical scale. For example, if counties with a population of over one million are allocated to receive an intervention, while those with a lower population are not, then regression discontinuity could be used.

Regression discontinuity compares outcomes in places that fall within a narrow range on either side of the cut-off value. For example, any place with a population short of or over one million by, say, 50,000 people could be included in the comparison. This method assumes that places on either side of the cut-off value are very similar, and therefore, the allocation of an intervention based solely on an arbitrary cut-off value may be as good as a random allocation. The method requires few additional assumptions and has been shown to be valid.

It is important to bear in mind that the effect is estimated only for places that fall within a range around the cut-off value, and therefore cannot be generalised to places that are markedly different, such as those with much smaller or much larger populations.

In the paper below, Arcand and colleagues investigated the effect of an HIV education intervention in Cameroon that was allocated according to the number of schools in the town.

_

Interrupted Time Series

The interrupted time series method is used to estimate the effect of interventions by examining the change in the trend of an outcome after an intervention is introduced. It can be used in a situation when comparison places are not available as all eligible places receive the intervention.

This method requires a large amount of data to be collected before and after the intervention is introduced, and from a number of time points, to allow modelling of what the trend in the outcome would have been if the intervention was not introduced. The model is compared to what actually occurs. Any change in the level of the outcome or in the rate of change over time, compared to the model, can be interpreted as the effect of the intervention.

It is possible that changes in the trend in the outcome may be due to factors other than the intervention. This can be accounted for quantitatively: by investigating events or policy changes that took place at the same time. Alternatively, like the approach used in the difference-in-differences method to assess the counterfactual rate of change over time, researchers may investigate ‘control trends’ in outcomes. This is done by investigating other related outcomes that might be affected by most of the possible alternative explanations for the change in the trend observed, but not affected by the actual intervention.

In the paper below, the authors investigate the effect of a pneumonia vaccine on pneumonia admissions. They considered that changes in the wider healthcare system might have also affected pneumonia admissions, so they investigated the trends in another related outcome: admissions for dehydration. The assumptions made were that the majority of the possible alternative explanations, such as policy changes or changes to delivery of healthcare, would have affected dehydration admissions to the same extent as pneumonia admissions; that dehydration admissions would not be affected by the vaccine; and that pneumonia did not cause dehydration in large amounts. Using this approach, they were able to show more convincingly that the vaccine brought about the change in the trend.

_

Synthetic Controls

Synthetic controls is a relatively new method for evaluating the impact of interventions using data from places that did not get the intervention, collected over time. The method works by first looking at the trends in the outcome of interest before the intervention was introduced. The data from various places that do not ultimately get the intervention are each given a weight so that the weighted-average of their data look as much as possible like the trend in the places that will get the intervention. This weighted-average is the ‘synthetic control’. The weights, unchanged, are then applied to the places without the intervention after the intervention has been introduced, and this weighted average is compared to the actual trend in the place with the intervention. This comparison can be used to estimate the impact. Similar to the other methods discussed earlier, researchers must assume that there is not another intervention or policy change that is happening in the places getting the intervention at the same time. The method requires a lot of data, both from many places and over time. It does not use or require parameterised models, so inferential statistics are calculated using permutations rather than more traditional methods.

In the paper below, Abadie and colleagues introduced the method and applied it to investigate the impact of a tobacco control policy change on cigarette consumption in California, by comparing the trend in California with a weighted-average of the trends in the other states in the USA.