www.elsblog.org - Bringing Data and Methods to Our Legal Madness

09 March 2006

The Trouble with Surveys

It’s interesting to me that, as Todd Peppers says, he “ha[s] worried less about low response rates than responses from law clerks who downplay their job responsibilities while exaggerating their employer’s intellectual abilities and moral virtues.” Certainly that’s a problem for survey research, not just surveys of Supreme Court law clerks, but in general. One challenge of surveys is that many people won’t participate. Another challenge is that people who do participate may not give objective responses.

It’s a problem faced both by academics and by journalists. Books like The Brethren and Closed Chambers were criticized as based on information from law clerks who allegedly exaggerated their roles. Now Todd expresses concern that law clerks may have understated their roles. Both cases lead me to question whether maybe we should be trying to find other ways to gather information than surveying law clerks.

Certainly
surveys can provide valuable information about topics that can’t be
studied any other way. And I’m sure Todd, Stephen Wasby, and Artemus
Ward do not rely solely on surveys of and interviews with law clerks in
their studies of court practices and procedures. Nonetheless, it seems
to me worth thinking about other ways in which information about the
inner workings of the Supreme Court (and other courts) might be
gathered.

One
obvious source of information is the Justices themselves. In a comment
to my earlier post, Todd explained that while he was able to meet with
Justices Stevens and Scalia, Justice Scalia
would only speak off the record and none of the other Justices would
speak with him. My reaction is that if the Justices won’t speak with
researchers, is it any surprise that the Justices’ law clerks won’t
either? That’s a pretty strong signal by the Justice that such matters
are in fact confidential (or at least might be perceived as such a
signal by a former clerk). But even if the Justices would regularly and
openly speak with researchers, there would seem to be the same concerns
about the objectivity of their responses. If in fact Todd’s Justice X
would be willing to be interviewed, we certainly wouldn’t expect him to
own up delegating all the work to his clerks (much less the problems in
his personal life).

(As
an aside: Todd suggests that it might be in the public interest for
this sort of information to come to light. That was the same argument
Eddie Lazarus, author of Closed Chambers, used to justify his
journalistic exposé on the Court. It’s not the law clerk’s place to “go
public”; it’s a matter for the Court as an institution to deal with.)

An
alternative is to try to develop objective ways of examining how the
chambers of Supreme Court Justices work. Textual analysis of opinions
might be a way of determining the extent to which opinions are drafted
by Justices or their clerks. The article by Timothy Johnson et al. on
“The Influence of Oral Arguments on the U.S. Supreme Court” (described
in the post below) uses data from Justice Blackmun’s papers to test
whether oral arguments influence the outcome of cases. Perhaps
information from bench memos might be used to similar effect (or
information from cert pool memos might provide a basis for testing
whether clerks influence agenda setting). Indeed, a team of political
scientists and law professors has used a statistical model to predict
the outcomes of Supreme Court cases more accurately than panels of
legal experts were able to do. Theodore W. Ruger et al., The Supreme Court Forecasting Project: Legal and Scientific Approaches to Predicting Supreme Court Decisionmaking,
104 Colum. L. Rev. 1150 (2004). (Embarrassingly, the experts predicted
fewer than 60% of the Court’s cases correctly, worse than simply
picking “reverse” every time!) Maybe a variable could be added to the
model based on the identity of the law clerk responsible for the case
in a particular chamber. Some of these approaches may already have been
tried, either with or without success. No doubt creative scholars could
come up with many others.

Now
don’t get me wrong: the research done by people like Todd, Stephen,
Artemus, H.W. Perry (whose book I own), and others based on interviews
with law clerks has been fascinating and invaluable. But that research
methodology, like all research methodologies, has limits, and those
limits aren’t unique to law clerks.

Comments

Indeed, Chris is right that I have used material other than survey/interview responses in my research. I have used opinions (as suggested), but, more important, and as hinted at in my original post, I have been able to see casefiles, which contain memoranda between clerks and judge; those are invaluable in determining what clerks actually do, and which suggestions by the clerk are adopted (on the influence by clerks point).
As to the difficulty of obtaining interviews with judges, I would suggest that some of the difficulty is trying to interview "the Supremes" -- it's like trying to interview members of Congress: there are so many demands on their time (and, in their case, so many others have already interviewed them), that one can't access to them. Now, if people would study other courts --e.g., the courts of appeals-- they would find judges much more willing to be interviewed, about PROCESS, not about specific cases, of course.
Steve Wasby