Data Science/Health IT Talk, Ziad Obermeyer, MD, Brigham and Womens Hospital: Are we over-testing? Using machine learning to understand physician decision making

(Joint work with Sendhil Mullainathan)
Low-value health carecare that provides little health benefit relative to its costis a central concern for policy makers. Identifying exactly which care is likely to be of low-value ex ante, however, has proven challenging. We use machine learning to gauge the extent of low-value care, focusing on testing decisions for heart attack in emergency departments (EDs). This approach finds that a large set of patients tested by doctors have extremely low ex ante predicted risk of having a heart attack. Indeed, focusing on testing decisions on the margin reveals that the rate of over-testing is substantially higher than we would think if we simply measured overall effectiveness of the test: the marginal test has far lower value than the average test, and our approach can quantify this difference. We also find that many patients who go untested in fact appear high risk to the algorithm. Doctors decisions not to test these patients does not appear to reflect private information: these patients develop serious complications (or death) at remarkably high rates in the weeks after emergency visits. By isolating specific conditions under which patients in emergency departments are quasi-randomly assigned to doctors, we are able to minimize the influence of unobservables. These results suggest that both under-testing and over-testing are prevalent. We conclude with exploratory analysis of the behavioral mechanisms underlying under-testing, using traditional methods as well as a deep learning approach to electrocardiographic waveform data.