Inside the Black Box: Why do things work?

There are now a variety of well-known experimental and non-experimental methods that economists use to learn whether a given program works or not. However, our tools for learning why or why not something works are much more limited.

In a useful new paper, Kosuke Imai and co-authors carefully set out the additional assumptions needed for researchers to move from estimating causal effects to also identifying causal mechanisms. They give a couple of examples from the political science literature to illustrate their points. I’ll try here to summarize some key ideas through applying them to an economics example.

Consider Business Training, a popular policy aimed at improving the productivity and increasing the incomes of small business owners. I’ll blog about new results from several randomized experiments on business training in future posts. But for now, suppose we find that business training is actually increasing business profits. Then we would like to know why?

Even if we have a randomized experiment which randomizes which owners get trained and which don’t, further assumptions are needed to learn which mechanism is operating. Suppose we want to understand how much of the improvement in profits comes from better business practices (appropriately measured) rather than through other mechanisms.

Imai and co-authors call the key assumption sequential ignorability. In our context, this amounts to assuming that conditional on whether or not someone has been trained, and on other observable controls, their level of business practices is independent of profits. This is quite a strong assumption – which would be violated for example if those with better business practices were also more educated, and if education had affects business profits through channels other than business practices.

Is this too strong an assumption? Remember it is an assumption conditional on observables, so puts us squarely back in the world of observational studies – if we can control for enough stuff, perhaps it is not so bad.

The money quote from Imai et al (p. 12):

“It is worth recalling that, in general, research with observational data is built upon a strong assumption that conditional on covariates the treatment variable is ignorable. Despite this, much can be learned from observational data. In fact, many social science theories result from simple observations and many experimental studies confirm the results of observational studies”.

So under this sequential ignorability assumption, we can estimate:

Profits = a1+ b1*Business Training + c1’X + e1, and

Profits = a2+b2*Business Training + c2’X + d2*Business Practices + e2

And then get b1, the total average treatment effect of business training; b1-b2 as the estimate of the average causal mediation effect (ACME) of how business training affects profits through the channel of business practices, and b2 as the average direct effect, measuring how business training affects business profits through other channels. Imai et al also provide non-parametric ways to estimate this in cases where linearity is not desirable.

Fine you say, but I don’t believe this assumption, can’t I do better?

One intuitively appealing approach might be to try and randomize more steps along the causal chain. For example, if we could use our first experiment to estimate the causal effect of business training on business practices, and then do a second experiment where we somehow persuaded businesses to randomly alter their business practices (by sending in useless consultants for example) to identify the causal effect of business practices on profits, couldn’t we just multiply the two effects together?

The answer is that:

If the treatment effects are constant, then this works

if the causal effects vary across individuals, this only works if the sequential ignorability assumption holds (see section 7.1 of the paper). Intuitively, the concern is that it could be that the type of people for whom business training leads to big changes in business practices may not be the type of people for whom changes in business practices make that much difference for profits.

So what else can we do? Imai and co-authors discuss several other approaches:

Use sensitivity analysis to see how much of a violation of the assumption there would need to be to rule out your causal mechanism

Randomized experiments where the mediating variable (aka our channel of interest) is randomly varied within each treatment status

Examining treatment effect heterogeneity according to pre-treatment covariates – they discuss some pros and cons of this approach, and note it also requires additional assumptions.

There are a number of alternative approaches to exploring causal mechanisms, including qualitative approaches, but to date most papers in economics typically report average treatment effects, and then sometimes use theory to suggest where we might see heterogeneity in these effects. I therefore believe learning what other approaches people are trying, especially when they clearly set out the assumptions needed for them to work, offers some encouraging avenues to take economic impact evaluations going forwards.

Comments

I must admit I find myself closer to the authors of "Enough Already about 'Black Box' Experiments" (http://ann.sagepub.com/content/628/1/200.abstract). Just demonstrating an effect in an experiment is no easy task. Even when effects are manifest, experiments suffer from a lack of generalizability to other contexts and experimental units. To me, these seem like the bigger needs to address.
Perhaps the best thing I can say about causal mediation analysis is that is provides a principled method for *theory building*. In the example paths from business training to profits, two of the channels suggest clear manipulations for a follow up study (providing training on accounting only and providing training on credit only). If funding existed for only one follow up experiment, why not select the one indicated by mediation analysis? While I would be skeptical that the original business training experiment revealed the causal pathway, there is still useful data to generate new hypotheses for testing.

Economists and political scientists don't read enough of each others' work, so it is great to get these pointers to articles that I wouldn't otherwise come across. The Enough already paper makes some of the same points at the Imai paper, although less technically.
Both sets of authors agree it is hard to get at the channels through which an effect operates, but seem to differ in their assessment of the worth of approaches to explore these channels - or perhaps in their frustration as to how these attempts are sold in papers. The basic point (which comes up in Mark's comment too) is that we need more assumptions to get more interpretation - whether these assumptions should be statistical ones or theoretical ones is then the challenge facing researchers.
We hope to have more posts on this topic in the future, so will get some different takes on this issue.

Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed.zx I’m sure you had fun writing this article.