Fooled by randomisation

Publication by Market Systems Development for Decent Work (‘the Lab’) of the International Labour Organisation (ILO)

You’re probably one of the many who’ve been told that randomised control trials (RCTs) are the ‘gold standard’ methodology for assessing impact. Sadly, the people who say this to you do so without any sense of irony, or indeed history.

The original gold standard was a monetary system where currency was based on a fixed quantity of gold. For a long time considered imperfect and subject to constant debates about its effectiveness, the gold standard was eventually abandoned for a better system – the fiat system of banknotes in your wallet right now – in the early 20th century. Not a single country in the world uses it anymore.

RCTs may be much more like this gold standard than their champions would like to believe, and end up being consigned to the history bin in a similar manner. Critiques of randomisation are not new, and have been well-documented . They are expensive, deliver big data dumps rather than more rapid feedback, require a consistent and homogenous treatment, don’t say much about why change happens, and lack context. Let’s even leave aside the ethical issues for now.

This, however, has not stopped a huge rise in the number of experimental studies being commissioned. A ‘randomista’ movement, which came out of medicine and into social science, has largely focused on the development impact of health and microfinance interventions but is now creeping into the field of enterprise development. But before we get preoccupied with whether or not RCTs could fit into this field, it’s worthwhile stopping to think whether or not they should.

These scientific experiments have the potential to do as much harm as good, especially if they’re cookie-cuttered into a private sector development (PSD) context. Here’s why.