www.elsblog.org - Bringing Data and Methods to Our Legal Madness

29 March 2007

I am currently in the process of designing a survey on financial and billing information for the Indiana State Bar Association (ISBA). In preparation for this task, I have read and reviewed lots of surveys by bar associations in other jurisdictions. There are a surprising number, often conducted by local academics (I blogged about a really terrific survey of Pittsburgh lawyers here).

What I lack, however, is a good reference guide for interpreting survey data, particularly a methods book or article that discusses how to check for a representative sample and various weighting techniques for either correcting sample bias or imputing missing values based on known characteristics of the population (information I have because the survey is sponsored by the ISBA).

Any suggestions in the comments or by email would be greatly appreciated.

"... medical malpractice claims that were closed in Florida, Illinois, Maine, Massachusetts, Missouri, Nevada, and Texas from 2000 to 2004. In these states medical malpractice insurance providers are required by law to submit information on closed medical malpractice claims. These claims are typically submitted to the Department of Financial and Insurance Regulation, which provided the data to BJS. The report provides an in-depth analysis of medical malpractice claims, including the number of claims closed with or without payouts, types of medical providers named in claims, facilities where alleged injury occurred, severity of alleged injury, court disposition of claims, amounts paid to compensate claimants, costs of defense counsel, and claim processing time."

25 March 2007

Vault, Inc, which is a publisher that collects industry information on various professions, recently released its listing of the Top 25 Most Underrated Law Schools. (Hat tip: Paul Caron & Volokh.) The findings are derived from a survey of 512 law firm recruiting managers, hiring partners, and corporate counsel, who were asked "to name law schools that, based on their experience as hiring managers, are underrated."

As I read the list, I wondered, what is the practical implication of a law school being underrated by the people who make hiring decisions for entry level lawyers? Presumably, it means that graduates of certain law schools tend to perform better than their school's U.S. News ranking would suggest; thus, legal employers are more likely to hire them.

If this is true, what is the source of superior performance? Here are two possibilities:

Stronger Students. Some schools may enroll a stronger student body than their rank might suggest.

Better Education. Some law schools may equip graduates with more or better skills than other schools of comparable rank

Based on some preliminary statistical analysis, there is fairly clear quantitative evidence for the first hypothesis. There is also some qualitative evidence for the second--enough to warrant some additional research.

These research questions are really important to the ranking debate because their answers could reveal a mechanism by which some students will discount USN. If an "underrated" school offers a better entree to coveted legal employers, why go to the higher ranked school, especially if the underrated school costs significantly less? Law schools that understand these dynamics are in a better overall competitive position.

23 March 2007

Last April, we posted the abstract of Robert Rasmussen's (Vanderbilt) working paper, "Empirically Bankrupt," which critiques three empirical papers in the bankruptcy area. The abstract concluded that "[f]or empirical work to be credited, at a minimum, it has to look in the right place, ask the right question and draw the right inferences. When empirical work fails to cross this threshold, it conclusions must be rejected."

One of the articles being critiqued was Elizabeth Warren's (Harvard) & Jay Lawrence Westbrook's (Texas) recent study,
Contracting Out of Bankruptcy: An Empirical Intervention, 118 Harv. L. Rev.
1197 (2005). A few months ago, Warren & Westbrook posted a rejoinder, "The Dialogue Between Theoretical and Empirical Scholarship," that not only addresses Rasmussen's criticism, but also tries to articulate an optimal relationship between theory and empiricism. The entire exchange between Rasmussen and Warren & Westbrook is worth reading. Here is the abstract of the rejoinder essay:

In this essay we offer brief reflections on the best process for critiquing empirical work in law and sustaining an engagement
between theoretical and empirical approaches. We emphasize the importance of theoretical work in helping to shape the scholarly agenda, but we urge that theory should be more closely tied to fact. We illustrate our argument by responding to a recent critique of our own empirical work by Professor Rasmussen. His principal claim is that our work should be discounted because we reported on all business bankruptcies, both those of entrepreneurs and those in corporate form. In response, we reanalyze our data, separating the individuals
from the corporations; in every case the re-analyzed data support the conclusions of our original paper to the same extent or more strongly. Similarly, his other claims about our work are shown to be incorrect.

21 March 2007

Over at the Social Science Statistics Blog Jim Greiner helpfully reminds us about the salience of precedent for those endeavoring to model court decisions. Jim notes (here):

"It's tempting to think that one can code appellate decisions or
judicial opinions pursuant to some neutral criteria, then look for
trends, tease out inferences of causation, etc. Here's a note of
caution: they're not i.i.d. They're probably not i.i.d. given X
(whatever X is). Precedent matters."

"... at least for some types of
cases, judges' political orientations have significant effects on the
number of cases filed. Thus, the political orientations of even trial
court judges have important effects on the outcomes of the federal
justice system: fewer potential plaintiffs seek relief in federal
courts if they believe the judges are less likely to be sympathetic to
their cases. This finding also helps explain the surprising findings of
other researchers that a federal district judge's political orientation
generally does not affect the probability that a case ends in a
judgment for the plaintiff or a settlement. This paper's findings
suggest that those results are based on a selection bias: because
parties believe that judges' political orientations matter, they are
likely settling cases on more favorable terms for the plaintiffs if the
cases are assigned to Democratic judges rather than to Republican
judges."

19 March 2007

This week, the Colloquy published two
pieces as it begins a series of responses to an article forthcoming in the print Northwestern Law Review by Profs. Lee Epstein, Andrew
D. Martin, Kevin M. Quinn, and Jeffrey A. Segal. In it, Epstein et
al. perform an empirical analysis of Supreme Court Justices' voting patterns and
find that almost all Justices exhibit
some degree of shifting ideological preferences during their time on the Court.

Colloquy posts the introduction to the full piece
by Epstein et al (see here), and follows with a reflective piece by Linda
Greenhouse, the Supreme Court correspondent for the New York Times and author of
Becoming Justice Blackmun (see here). In her
piece, Ms. Greenhouse draws on her time covering the Court and her study of
Justice Blackmun, the exemplar of shifting Justice ideology, to provide
anecdotal support for the conclusions of Epstein et al.

Kicking off a recent friendly-but-frank
discussion about the relevance of contemporary law review articles,
Dean David Rudenstine of the Benjamin N. Cardozo School of Law offered
one description: "useless blather puffed up with self-indulgence" ...

Judge Sack returned to the not-so-funny problem under consideration:
today's highly theoretical articles are largely ignored and seldom
cited by judges, a dramatic turnabout from a generation ago when the
mostly practical content of law reviews was a significant element of
judicial decision-making. Judge Sack was sorry to say that the bench
now uses law reviews "like drunkards use lampposts, more for support
than for illumination." ...

"If the academy wants to change the world, it must decide if it wants to be a part of the world." ...

Although Judge Sotomayor agreed that brainy law professors should
not trouble themselves by contemplating reactions from the bench to
their writings, she leveled a sharp gaze at the Cardozo Law faculty and
declared, "If you think that judges are not as capable of creative
thought as you are, I beg to differ." She added, "My question to
academics: do you really think you're serving some function to
someone?"

A recent paper by Herbert Kritzer (Wisc.--poli sci) et al., The Business of State Supreme Courts, Revisited, sets out to update Kagan et al.'s earlier study of the docket composition of state supreme courts. I am partial to such undertakings (updating, revisiting, and/or replicating important past work) and welcome attention to the comparatively understudied state courts. Kritzer and his colleagues find:

"... many of the patterns of change described
by Kagan et al. continued through the 20th century: debt and real
property continued to decline and criminal continued to increase.
However, other patterns of change either reversed or halted.
Specifically, neither torts nor family cases have continued to
increase; torts have stabilized and family cases, rather than
increasing, have declined. The most surprising shift is the sharp
increase in "other contract", which had no particular pattern in the
earlier data, but which represented five percent or less of the courts'
business; in the 1990s, "other contracts" had grown to a level
approaching that of public law, and exceeding real property and family
and estate cases."

15 March 2007

14 March 2007

Last week, Justices Kennedy and Thomas visited Congress to present the annual budget request and, while there, discussed the need (as they see it) to split the Ninth. (Hat tip to How Appealing Extra.) They are not, of course, the first to discuss this, but I thought I'd use it as an opportunity to think about what ELS can provide to an important (and sometimes heated) policy question. Very often Congress (or state legislatures) make policy decisions without "good data." (States' choices of selection systems, for example, rarely seem to be informed by the ramifications of selection method on decision making or legitimacy, etc.) So, to start a discussion of what a split of the Ninth would accomplish v. what harm would it do, I call to your attention work by Kevin Scott of the Congressional Research Service (and I'm hoping others might call attention to other empirical work on this question as well). Here's his abstract:

Much attention has been focused on the Ninth Circuit Court of Appeals. To some degree, the attention has been self-created, with high profile decisions on three-strikes laws, punch-card ballots, and the constitutionality of the Pledge of Allegiance generating substantial media coverage and reaction from elected officials. This attention mirrors an ongoing debate about the wisdom of splitting the Ninth Circuit into two smaller circuits. Arguments about splitting the Circuit center on the source of the Circuit’s high reversal rate. There are two competing explanations for that reversal rate: the judges of the Ninth Circuit are more ideologically distant from the Supreme Court than judges on other Circuits, or the size of the Ninth Circuit prevents the circuit majority from correcting “wrong” decisions by panels, a problem augmented by the Ninth Circuit’s reliance on the limited en banc procedure.

This paper sorts out the cause of the Ninth Circuit’s reversal rate by looking at the Court’s relationship with the Supreme Court over the past twenty years. By looking at merits reversals, including unanimous reversals, and attempting a broader assessment of the Ninth Circuit’s status vis-à-vis the other circuits and the Supreme Court, I ultimately argue that both size (though indirectly) and ideological orientation influence the Ninth Circuit’s high reversal rate. These findings have broad implications for how we model the behavior of court of appeals judges and their relationship with the Supreme Court.

13 March 2007

Recent guest blogger Carolyn Shapiro
and her colleague at Chicago-Kent, Bernadette Atuahene, are proposing a new
listserv for junior scholars interested in empirical legal scholarship. The
purpose of the listserv would be to provide a forum for folks to seek advice
on methodological, statistical, and other types of questions. The idea is
modeled, very generally, on the successful Chicago-Area Junior Faculty Workshop,
which brings together junior scholars from the Chicago area to present
and discuss each other's work. Here, the idea would be to provide a
nonthreatening environment for people to ask questions and seek out resources.
(In this context, junior means non-tenured, whether pre-tenure or not
tenure-track.) If you are a junior faculty member interested in this, please
e-mail cshapiro1@kentlaw.edu with "ELSlistserv"
in the subject line. If there's sufficient interest, we'll
proceed.

One article I have been meaning to blog about is this profile of economist Kevin Murphy in the Nov/Dec 2006 issue of University of Chicago Magazine. Murphy has an impressive list of accomplishments, including the John Bates Clark Medal of the American Economic Association, which is given once every two years for the most outstanding American
economist under age 40 (1997), and a MacArthur Foundation "genius grant" (2005).

Murphy is quite a character, and the article provides some fascinating details on his career history. But this passage on co-authorship really caught my eye:

"[Regarding co-authorship,] I think most people get a better product that way,” he explains. “You
pool insights and eliminate oversights. First-rate coauthors make
working on projects more fun and rewarding. Plus, writing is a pain.”
Since earning his PhD and joining the faculty in 1986, he has written
every one of his 60-plus published papers with a former teacher, a
close colleague, or students. ...

As a shy but affable 25-year-old grad student, Murphy confessed to
[his advisor] Topel that his goal at that stage in life was “to become the world’s
best coauthor.” By the time he was 35, he was there.

Wow, that is quite a testament to collaboration. To the right is a picture of Murphy, who is an avid woodworker, in his workshop at home. Apparently, it is impossible to locate a picture of Murphy without his signature baseball cap.

12 March 2007

If replication was not already difficult in many instances, a recent post at the Social Science Statistics Blog reveals that, for studies using certain finance data (specifically, data on stock analyst recommendations), the degree of difficulty is materially higher than one might imagine. Evidently, certain data sources (notably in the finance sector) resemble "ever-shifting moving targets."

10 March 2007

Another useful methodology often used in psychology -- less in social, more in cognitive -- is one alluded to here before, "multidimensional scaling" (MDS). MDS has only been used in a small number of legal analyses, but can be quite helpful in the right context.

As I describe in a forthcoming paper, "MDS is a procedure that helps researchers uncover 'hidden structures' in data by graphically plotting respondents' perceptions of perceived similarities (or dissimilarities) among various stimuli. When these stimuli are located on a plot based on such perceptions, underlying dimensions that respondents may have used (consciously or not) can be inferred."

So, for instance, a bunch of stimuli are identified -- crimes, Supreme Court opinions -- and every pairwise combination is created. Each pair is rated on how similar or close they are (though the researcher usually does not articulate the scale to be used, just asks "how similar"?). Those similarity ratings are then plotted to give the best fit to the data, yielding a "map" of the ratings that also reflects the perceived structural relationship of all those stimuli. Finally, by examining the map, one might be able to identify the dimensions that, implicitly or explicitly, underlay subjects' ratings (and also, interestingly, look at individual differences, e.g., do men and women or Republicans and Democrats come up with different maps or weight those dimensions differently?).

As a specific, somewhat prosaic example: subjects might be asked to rate the similarity of countries in the world, and a plot made of the countries based on those similarity ratings. The resulting map might look like a map of the world, which could suggest that subjects were, implicitly or not, using dimensions involving distance, in particular, dimensions of north/south and east/west. But the plot might line up the countries on a single axis going from largest to smallest, or richest to poorest, or most to least economically developed, suggesting that subjects used that single dimension in their ratings. Inferring which dimensions are important to respondents helps understand the way they see the world (literally, here), and what factors are important to them.

In the legal context, I used it in that paper to map people's perceptions of the seriousness of crimes. MDS is broadly useful for looking at lay intuitions about legal institutions, punishments, or cases; or to assess whether different punishments may have been proportionate; or any context in which (a) similarities among stimuli may be interesting or (b) we are interested in understanding what the basis might be for people's judgments or perceptions.