Friday, September 12, 2008

Empirical Legal Studies

In 1897, Oliver Wendell Holmes, Jr. wrote in the Harvard Law Review: "For the rational study of the law the blackletter man may be the man of the present, but the man of the future is the man of statistics and the master of economics." Well, it appears that the future is finally here. Today and tomorrow, Cornell Law School is hosting (and NYU and Texas Law School are joining Cornell in organizing) the third annual Empirical Legal Studies Conference. I'm attending and participating in the conference, even though my own work is not especially empirical (with an occasional exception). Only a fool of a law professor would be completely uninterested in whether the world as imagined by legislators and judges corresponds with the world as it actually is.

Still, I want to quibble slightly with both Holmes and the ELS Conference organizers for their tacit but unmistakable suggestion that doctrinal analysis is not itself empirical. Holmes has a better excuse. He was reacting to Christopher Columbus Langdell's pretension that law, as understood via the case method, was an inductive science. Holmes' chief complaint was that Langdellians were analyzing the wrong data. One can have a completely self-consistent understanding of existing cases, but the body of rules that produces can still be socially harmful. As Holmes put it in the very line following his prophesy of a statistics-based future: "It is revolting to have no better reason for a rule of law than that so it was laid down in the time of Henry IV."

The legal empiricists (at the ELS Conference and more broadly) who want to examine the real-world effect of various legal regimes follow admirably in Holmes's path. But a significant number of empiricists are Langdellians in an important sense: They aim to understand how cases are decided. What influence does ideology have, for instance? Do judges truly follow the rule of stare decisis? Etc. For many of these empiricists, the favored mode of analysis is the large-n study with statistical regressions. And much of what they discover is both real and interesting.

However, the fact that we can learn much about a phenomenon by large-n study does not mean that we can't also learn other things about that phenomenon by close observation. Traditional doctrinal work, if done well, is to large-n studies of judicial behavior, as participant-observer studies are to large-n studies of any social phenomenon. Thus, Billy Beane, Theo Epstein and Bill James can tell us a great deal about which pitchers are likely to perform well in a given season just by looking at their past records, even focusing on a small number of variables (for Beane per Moneyball, just strikeouts and home runs allowed). Still, additional interesting information can be gleaned from direct observation. E.g., that a pitcher will throw a curveball on a "fastball count," or that even though his fastball hasn't lost velocity, it seems to have less movement than in prior seasons. In principle, any of this information can be captured by an existing model or a by modified version of the model, but in practice, there are so many potentially relevant variables that no model can capture them all. Stats-minded baseball teams have been extraordinarily successful in recent seasons, but that doesn't mean that the game of baseball is fully captured by the statistical models.

And likewise for law. Sophisticated legal doctrinalists have insights about the law that the models don't capture, or at least that the existing models don't capture because no one had yet thought to look at them. Probably the most famous example in law is Learned Hand's claim that economic efficiency is the hidden logic of negligence. That was surely an overstatement, but it was an extraordinarily useful overstatement. Once Hand made the point, it could be tested empirically and subject to normative critique.

It's tempting to say that doctrinal work and empirical work complement each other, and they do, but my point here is that doctrinal work, if done well, itself has an empirical component, albeit a limited one. Langdell's mistake, in other words, was not in thinking that the common-law method of inducing general rules from particular cases is a kind of science; his mistake was in thinking that this is all that needs to be said or done (and it is not even clear that he made that mistake either).

11 comments:

The problem with your analogy (or perhaps the value of your analogy but problem with your argument - I really don't know which) is that, in baseball, more often than not, when attempting to predict future performance, those "participant-observer" "studies" lead to mistaken conclusions. They don't, in fact, introduce data into the equation, they just enter noise - from which errant conclusions are then drawn.

Whether the same is true of legal study I don't know, but either the analogy is imperfect or the analogy is perfect, but the argument is flawed in the same way it is flawed in baseball.

In response to Paul: No model is perfect. The problem with most of the baseball wisdom collected from direct observation is that the people collecting it make no attempt to integrate it with the statistical information. So, for example, Moneyball tells us that other things being equal, it's worth attempting to steal second base if there's a 70% or better chance of success. (I think I recall that but the exact number is not important.) An idiot conventional manager who sends runners with less than a 70% chance is ignoring, not supplementing, the statistical info. The same for a "conservative" manager who doesn't send runners who would have an 80% chance of success. But the 70% number is an average for the middle of the game. A team trailing by two or more runs with two outs in the bottom of the ninth inning should never send the runner. The statistical model doesn't say otherwise, of course, and this fine-grained knowledge could be worked into the model. For all I know, it already has been. (In football, coaches carry cards telling them when to go for two and when to kick the extra point. Maybe Terry Francona has a similar card about when to attempt a steal.) My point about doctrinal analysis in law is that without this kind of fine-grained analysis, the models will always be incomplete.