#2015APPAM Student Summary: Using Experiments for Evidence-Based Policy: Lessons from the Private Sector

By Abraham Song, PhD Student
George Mason University

The era of evidence-based research is upon us. Though the general public may be unaware, the private sector -- profit and non-profit -- has for many years been taking advantage of large data and cutting edge research to evaluate policies and practices. Today, some 150 corporations run over 100 tests annually to evaluate their business practices (15,000!). The use of randomized control trials has enabled firms to evaluate the efficacy of programs with less risk-taking or capital commitment. The late-comer to the scene is the public sector. The Obama administration has significantly propelled evidence-based policymaking, and this new emphasis has quickly redefined perspectives and approaches.

Jim Manzi of the Applied Predictive Technologies highlighted the opportunities and limitations of evidence-based research particularly that of the randomized control trials. He highlighted three important observations: 1) coming up with an innovative idea remains difficult 2) innovative ideas that work tend to have a small effect 3) the innovative ideas of small effect tend to be almost always conditional. In short, successful policies or business practices are not “moonshots” but the result of a “mountain of pebbles.”

John Baron from the Laura and John Arnold Foundation took a somewhat different angle and suggested expanding specific policies with a proven track record (e.g. welfare policies of the late 1980s in California).

Rebecca Maynard underscored the significance of protecting privacy and remembering that behind numbers are real lives that matter. More observations should not eliminate the rigor of analytics. Also, cheaper pilot studies of easily-accessed data are a good prerequisite for researchers before they jumping into larger, more expensive studies.