Search Site

Thursday, June 11, 2009

Evidence Based Policy, Law, and Uncertainty

In some cases, incorporating systematic reviews into legal proceedings will be relatively easy. When the review conclusively (according to some decision rule) establishes that causation either exists or does not exist (or, more precisely, conclusively establishes such a finding for now), then the judge's next step is straightforward. The judge can include one side's testimony and exclude the other's, and the risk of jury confusion is low. But what happens when the systematic review returns an uncertain result--causation may exist, but the research base is currently too weak to say much either way?

This is no small problem. Mark Petticrew suggests that this may happen more often than not in many circumstances. There is a lot of bad empirical work out there, so the more demanding the standards the fewer original studies survive quality evaluation; too few studies and it becomes hard to draw causal conclusions either way. What should a judge do in this case?

Under our current rules, it seems that both sides should be allowed to make their causal case. Daubert, for example, is not intended to allow the judge to (wholly) invade the jury's realm. This looks like the classic contestable fact that juries were established to resolve. But even putting aside the epistemic problems I have been talking about for a while now, viewing this uncertainty as a factual debate for the jury is not the right way to think about things.

Why? Because there is nothing to debate. A systematic review returning a result of uncertainty has provided us with the answer: we know we do not know. What we have, in other words, is a "known unknown." It is unfortunate that Donald Rumsfeld was oftenmocked for his known-unknownsspeech, since the point is epistemically spot on and far too often underappreciated (that you can hear journalists laughing in the background is more a comment on the journalists than Rumsfeld). The law in particular is uncomfortable with the idea.

In a land of dueling authoritative pronouncements, perhaps it can be hard to separate the known unknowns from cases of false uncertainty. Perhaps one expert really is telling the better story or relying on the better studies. But the comprehensiveness, transparency, and greater objectivity of systematic reviews--especially as quality criteria are themselves subjected to empirical validation--helps make it all the clearer where true known unknowns exist.

As an aside, note that the need to acknowledge known unknowns may point to an upside of the problem of dueling guidelines. If both sets of guidelines rely on empirically validated standards, and if there is no other clear objective reason for favoring one set of guidelines over another, then disagreement between the two sets of guidelines is informative. It provides (meta-?) evidence of uncertainty.

But back to the main point. How should the law handle the known unknowns? There is an easy--but problematic--answer: when a review returns a known unknown, the party bearing the burden of proof loses on that point (and the case, if that point is dispositive). Christopher Guzelian makes just this point in the context of torts: torts requires proof of causation, uncertain results do not equal proof, so the plaintiff loses when the results are uncertain.

This is clearly a strongly pro-defendant (or perhaps, more generally, pro-status quo) position, and it is not one we should necessarily comfortable with. Though individual cases deal with individual plaintiffs and defendants, our tort system in practice acts as an important source of broader, social regulation. And it is not just torts: consider forensics. Unless the Innocence Project's efforts to create a federal agency to regulate forensic science succeed, it will fall to the courts to respond to the inevitable challenges that will follow the National Academy of Science's blistering critique of the discipline. Again, it will be an individual defendant challenging, say, the reliability of eye-witness testimony, but the court's opinion will have wider regulatory impact.

In other words, there is a strong overlap between courts and agencies. It isn't surprising that regulatory agencies are allowed to regulate--to effectively "rule for the plaintiff," to extend the analogy--in the presence of uncertainty. As Wendy Wagner and others have shown, the permanent presence of "transscience" alone would make it impossible for them to act otherwise. It may be scientifically established that chemical x causes cancer in rats or in doses a thousand times the normal level of exposure, for example, but making the jump to the effect of regular exposure to humans may require untestable assumptions about interspecies physiology or dose-response curves.

Such assumptions are ultimately policy decisions. Whether an agency adopts an aggressive dose-response curve assumption or not is a political choice about whether it wants to regulative aggressively. We are comfortable about agencies making such political choices, since they are located in a political branch of government.

But we are fooling ourselves if we think the judicial branch can avoid this issue. By implicitly demanding that our courts engage in regulation, we have forced the choice on them. But our legal doctrines and decision rules do not always reflect this. Debates about dose-response curves are not issues of fact for a jury to decide, because they are not issues of fact. Dose-response curves, at least in some cases, are known unknowns. We either have to embrace Guzelian's point that tort law is not intended to handle these issues and toss all cases that rely on such extrapolation, or we have to rethink how policy decisions in the presence of known uncertainty are handled by courts.

In other words, we need to think more carefully about our decision rules. There have been some efforts along these lines. Margaret Berger, for example, suggests that when scientific evidence is uncertain in toxic tort cases, the focus should shift to emotional harm. And Wagner confronts Petticrew's concern about a too-small evidence base more directly. She argues that when the defendant industry--which is in by far the best position to test the effects of its products--fails to conduct the necessary tests and thus leaves the evidence based too small to draw conclusions, the burden of proof should shift from the plaintiff to the defendant: the industry now has to prove that the product is not harmful.

I don't know what the right answer is yet. But there is a clear tension between what tort requires of courts and what we demand of them. And I do not think we are ready--or even want--to strip courts of their vital regulatory and social insurance functions. At the same time, allowing juries to make "findings" about known unknowns is profoundly troubling. As our understanding of what we know and what we do not know grows stronger, we will have to think about and debate more thoroughly how we want courts to fulfill their implicit regulatory functions in the presence of uncertainty.

The technological and philosophical revolutions of the past few decades are exciting, but also humbling. Somewhat ironically, the epistemic superiority of EBP may force us to be more epistemically modest. An effective and legitimate legal system must adjust its procedures to reflect our changing understandings of what knowledge is and of what we know.