Abstract: Over the last two decades, there has been a surge of opioid-related overdose
deaths resulting in a myriad of state policy responses. Researchers have
evaluated the effectiveness of such policies using a wide-range of statistical
models, each of which requires multiple design choices that can influence the
accuracy and precision of the estimated policy effects. This simulation study
used real-world data to compare model performance across a range of important
statistical constructs to better understand which methods are appropriate for
measuring the impacts of state-level opioid policies on opioid-related
mortality. Our findings show that many commonly-used methods have very low
statistical power to detect a significant policy effect (< 10%) when the policy
effect size is small yet impactful (e.g., 5% reduction in opioid mortality).
Many methods yielded high rates of Type I error, raising concerns of spurious
conclusions about policy effectiveness. Finally, model performance was reduced
when policy effectiveness had incremental, rather than instantaneous, onset.
These findings highlight the limitations of existing statistical methods under
scenarios that are likely to affect real-world policy studies. Given the
necessity of identifying and implementing effective opioid-related policies,
researchers and policymakers should be mindful of evaluation study statistical
design.