Reimagining Reversals

Beyond the extraordinary “Why RCTs?” examples

If you’ve been following our “Why RCTs?” series, you’ve read about some drastic examples across several fields of study- many in the medical field- where RCTs have turned common knowledge and practices upside down. These examples are meant to be shocking; they show us the very real and serious implications that can follow from not testing interventions at all or with non-randomized interventions. The accounts I’ve highlighted in the series illustrate several things:

First, just because one has a gut feeling of what works, even when one is well-versed in the field—does not mean one is even close to knowing the existence and direction of causal effects;

Second, randomized testing may at first seem frightening in high-risk situations; however such evaluation may be most needed and appropriate exactly for these types of situations; and

Finally, it is important to note that, while the shocking examples we have highlighted may appear anomalous, reversals of medical practices due to rigorous evaluation are not that few and far between.

Throughout the “Why RCTs?” series, as well as in other blog posts, the A2J Lab has drawn many parallels between the medical profession and the legal profession. We’ve been drawn to the experiences of the medical profession since the field has been transformed, from essentially the dark ages of the 1920s to a flourishing science that embraces rigorous evaluation and growth.

The meta-analysis points to numerous examples, including hormone replacement therapy, that highlight how new interventions (e.g., technological or medicinal improvements) actually proved to be not better, or worse, than previous trends or no study at all. Articles were classified in one of four ways:

Think about this. Research physicians are constantly upending their conventional wisdom thanks to the power of RCTs. Lawyers have the same tools at their disposal, and the A2J Lab’s mission is geared toward implementing them in the service of narrowing the justice gap.

Continuous testing in a field is necessary lest it become calcified and detached from emerging evidence. We can learn a lot from the results of any RCT, not just the “shocking” ones. Such rigorous evaluation develops a deeper understanding of how to allocated resources more efficiently and how to reach people who need scarce services the most. Testing in the legal profession doesn’t just mean that some services will be found to be more effective or less effective than others. Rather rigorous evaluation can start to create an evidence-based culture of practice. It will enable legal services providers to continue doing their best work and helping the clients for whom they are champions.