This week, I’ve been aided by
two third-year students at UCLA – Patrick Anderson (who will
coauthor my forthcoming book) and Matthew Butterick (whose independent work on
the legal side of the mismatch was mentioned in Post 7). I’m grateful to
them both.

My
previous posts laid out the story of what is happening to black law
students as a result of
affirmative action. There are grey areas and we need
better data to resolve several issues. But if this were a less
controversial
topic, I think many would agree that affirmative action in law schools
may not be achieving many of its core
goals, and that it is probably producing some extremely serious side
effects. If affirmative action were not an entrenched social
policy, but instead an experimental drug, could it possibly survive an
FDA
review? Not on the existing evidence.

As
I suggested at the beginning of the week (and Bill
Henderson suggested earlier), there has not yet been a complete
critical
engagement with these arguments and the underlying data. Critics have
raised specific questions about my analysis, which in some cases have
led me to new insights and improvements. But no critic of
the mismatch hypothesis has woven the data into a consistent alternative theory.

I have responded to most of the critiques; rather than rehash those, I’d
like to illustrate the problem by examining a recent
critique by Rothstein & Yoon.

Jesse Rothstein and Albert Yoon have produced by
far the strongest response to Systemic
Analysis,
and I know and admire Albert. Rothstein & Yoon
recognize the problem of selection bias, and attempt to
deal with it. They construct models for the data, and they report
evidence of significant mismatch effects, though they think other
factors
might explain their results. All this is good. But Rothstein &
Yoon also commit many of the common critical errors:

Rhetoric. Rothstein & Yoon repeatedly say they
find no evidence of mismatch effects in the top 80% of the credential
distribution of law school applicants. This is misleading. On closer inspection, one finds that
the “top 80%” of all students includes only the top
20% of black students, so their mismatch findings apply to 80% of black
students. Even if they were right that the
top 20% of black students don’t suffer mismatch (but see below), it
trivializes the problem to say it only affects the “bottom
quintile”.

Ignoring relevant data. Given Rothstein & Yoon’s concern with
selection bias, it would seem that a comparison of first- and second-choice
blacks would be called for; no one has identified any meaningful bias in the
first-choice / second-choice data. Moreover, one can split the
first-choice / second-choice sample to compare the elite blacks (who supposedly
do not suffer from mismatch effects) with other blacks. I’ve done this,
and the effects seem indistinguishable – that is, blacks with high
credentials seem to gain as much from going to second-choice
schools as other blacks do. In other words, Rothstein & Yoon don't reconcile their findings with relevant data.

Shrinking “significance”. Law
student outcomes (for all
racial groups) get much better with rising credentials. Relatively few
students of any race with an academic index above 700 fail to graduate
and pass
the bar. But that doesn’t mean the mismatch effect has declined, if by
"mismatch" we mean how much relative learning occurs. For example, it
could easily be that mismatched students with fairly high
credentials are scoring much lower on bar exams than comparable,
non-mismatched
students, but both groups have mean scores far above the passing
threshold. Indeed, mismatch effects in grades (a measure of learning)
are
not different at all for the most elite blacks.

Rothstein and Yoon compound this problem by focusing on “ultimate bar
passage” as a key outcome measure. Since first-time bar failure
rates are much higher than ultimate bar failure rates, looking only at a
bivariate measure of ultimate bar failure throws out most of the relevant
information. Ultimate bar passage is also not a sensible measure theoretically.
The mismatch effect occurs during law school. If a student passes the bar
on his fourth attempt, two years after law school, he may well have partially
offset the mismatch effect by hiring tutors, taking a variety of
bar-preparation courses, and other work aimed at learning what he didn’t
learn in law school. First-time bar attempts are the best measure of what
was actually learned in law school.

Critics, including Rothstein & Yoon, chronically favor measures that narrow
apparent black-white differences or shrink sample sizes, in what looks like a
quest to shrink the gaps to a point where one can proclaim "it's no longer
statistically significant, and therefore not a problem."

Measuring mismatch in the job market. Rothstein & Yoon use BPS data
to see if blacks are hurt in the job market by mismatch effects. This is
commendable, but they miss the obvious test -- they do not compare the relative
impact of school eliteness and law school grades on black earnings. If they
had, Rothstein & Yoon would have found results highly consistent with those
I found in Systemic Analysis. Rothstein & Yoon instead show that blacks
right out of law school receive large preferences. That's really not relevant
to whether mismatch effects hurt black chances in the job market. (And, as
I've argued elsewhere, the preferences in the job market often backfire on
black hires.)

Bill Henderson has done his own analyses of the BPS data on a few of these
points, with results compatible with mine; interested readers can ask him about
those.

Again, the point here is not to single out Rothstein & Yoon, whose paper
has many strengths. Rather, it is to highlight how even the better critiques too
often read like briefs on behalf of a sacred policy, rather then disinterested
investigations into the pros and cons of a controversial program.

As
I suggested at the outset, I see this debate as a sort of maturity test for the
ELS community. There are several things we, as an academic community, can do
to make progress on these issues, and to move away from what has become an
overly politicized area of research:

Pressure the LSAC to study the issue. LSAC has data on where most
applicants apply to law school, where they are accepted, where they
matriculate, and how they perform in the first year. It would not be
difficult to use this data as the foundation for a quasi-experimental dataset
similar (but far superior to) the data on first- and second-choice
students. Supervised by credible, independent researchers, a study of
this type could be truly definitive.

Debate and engagement. More candid discussion of specific aspects of
law school affirmative action, and greater involvement in the debate by those
with no ax to grind, will necessarily make the discussion healthier and more
robust.

My
thanks again to the ELS hosts for creating this wonderful forum and letting me
borrow the microphone.

TrackBack

Comments

Lempert says he has offered a theory to explain lower black outcomes. But my point is that no one's advanced a theory which attempts to explain the available data. Let's see why Lempert's theory fails that test.
Lempert first says that affirmative action bears some responsibility for the black-white bar passage gap, because it leads to the admission (into legal education as a whole) of low-credential blacks who wouldn’t be law students at all but for their race. In other words, the credentials gap accounts for part of the black-white graduation and bar passage gap. This is true, but it misses the point – Systemic Analysis shows that about half the bar passage gap is due to the differences in credentials. The issue is, what accounts for the other half?
Lempert suggests two possibilities: financial hardship among blacks, and black underperformance in the classroom. Both of these are contradicted by a wealth of data. If income or financial need were strongly correlated with race and a powerful predictor of either graduation or bar passage, then the race coefficients in Tables 5.6 (predicting graduation) and 6.1 (predicting bar passage) in Systemic Analysis would be negative and significant. They are not. Blacks in law school are generally from affluent, upper-middle-class families. At a given level of family SES, black law students usually receive substantially more grant assistance from law schools than do whites. Likewise, the “underperformance” arguments – that blacks get lower grades because of professor hostility, institutional racism, stereotype threat, etc., etc. – simply don’t wash. Black underperformance in law school has been shown by many studies to be somewhere between small and non-existent. It is not anywhere near enough to account for the dramatic differences in performance that amplify the black-white graduation and bar passage gaps. Lempert’s “theory” is a non-starter.

Sander says that no critic of the mismatch theory has offered a coherent alternative theory. First, the failure to be able to explain a complex result does not make a flawed explanation sounder. Second, I have offered my explanation for the BPS and other data in testimony to the United States Civil Rights Commission in a joint presentation with Professor Sander and I sent him a written copy of my testimony. I will be happy to send it to anyone upon request or to post it on this web site if the custodians would like that.