www.elsblog.org - Bringing Data and Methods to Our Legal Madness

11 March 2011

Yesterday, Bob Morse of U.S. News published a blog post in which he signaled a change in the law school rankings methodology, specifically with regard to employment. The prevailing view on the law school administrator list-serves (which nearly a dozen people have forwarded to us) is that U.S. News will be increasing the weighting of "employed at graduation," presumably because U.S. News Editor Brian Kelly sent a letter to law school deans--reprinted in Bob's blog post--discussing the importance of employed-at-graduation as a metric.

We have zero inside information, but we are willing to bet a substantial sum that any methodology change will be in a completely different direction. Here is why. Over the last decade, fewer and fewer schools have been supplying U.S. News with employed-at-graduation data. Employment at graduation is not a statistic required or collected by the ABA; as such, its accuracy cannot be checked through cross-reference to the annual ABA-LSAC Official Guide.

But much more significantly, when a school fails to provide this data, U.S. News has--up until now--imputed the figure based on employment at 9 months. (Kudos to Ted Seto for unraveling this mystery. See Understanding the U.S. News Law School Rankings.) Crudely speaking, the magazine applied a roughly 30% discount rate on the employed at 9 months figure. Earlier this year, Paul Caron suggested that if a school's employed-at-graduation rate is more than 30% lower than its employed at 9 month rate, it is "rankings malpractice" to supply U.S. News with the data.

As readers can see from the above chart (generated by Paul Caron in his rankings malpractice post), a large proportion of law schools have figured out the payoffs. Over the last decade, the percentage of non-reporting schools has skyrocketed. With this information in mind, Bob Morse's blog post may seem less cryptic:

In an effort to make our law school employment data more reflective of the current state of legal employment, U.S. News has modified how we calculate the employment rates that are used in the new law school rankings. ...

U.S. News agrees with the efforts of Law School Transparency to improve employment information from law schools and make the data more widely available.

If the goal is (a) to utilize data that better reflect reality, and (b) provide greater transparency and access to such data, it makes no sense to increase the weight of an input (employed-at-graduation) that is either withheld by law schools or is heavily gamed. This latter point is made by U.S. News Editor Brian Kelly in his letter to the deans:

[E]mployment after graduation is relevant data that prospective students and other consumers should be entitled to. Many graduate business schools are meticulous about collecting such data, even having it audited. The entire law school sector is perceived to be less than candid because it does not pursue a similar, disciplined approach to data collection and reporting.

At U.S. News, we work to make meaningful and fair comparisons, based on industry-accepted data. ...

To eliminate some of the gaming that seems to be taking place, we have changed the way we compute employment rates for the rankings due out March 15. In addition, we will also be publishing more career data than we have in the past in an effort to help students more completely understand the current state of legal employment. We think more still needs to be done.

So Kelly is saying that employed at graduation data are important, and the magazine is tired of being gamed. Therefore, we think two methodogical changes have a good chance of being implemented:

U.S. News is likely to heavily penalized schools that withhold the employed at graduation data. Going forward, the imputation may be far more negative than -30% off of employed at 9 months. A drop in rankings will stop in its tracks the non-response problem.

Regarding perceptions of gaming, it is possible that U.S. News has formulated a way to quantify how many jobs at graduation map onto full-time professional jobs that require a law degree. For example, the ABA Official Guide provides lots of comparable data by practice setting. Law firms, judicial clerks, and government jobs could be weighted more heavily than business or academic jobs. Unknown may also be treated as 100% unemployed rather than the current 25% presumption of employment. Such changes would have the law schools scrambling to report better numbers in higher weighted categories rather than just finding ways to goose up the employed-at-graduation and employed-at-9 months figures. Remember that Bob Morse explicitly endorsed the Law School Transparency movement.

We would like to suggest to our colleagues in the legal academy that we are approaching an endgame. Here is the reality: prospective students are not being given an accurate picture of their future employment prospects. Why? Because we are all focused on filling next year's class with as many high credential students as possible, thereby protecting our school's place in the pecking order. Our focus is so shockingly narrow that, from the outside looking in, it appears that our intent is to deceive incoming students. Brian Kelly's letter to the deans essentially makes that point--law schools fall short on candor and ethical behavior.

The numbers that get submitted to U.S. News include many graduates who are technically employed but often significantly underemployed, often at positions that don't require law degrees. Finer grained data get reported to NALP, but they are never published on a school-by-school basis. If these data were released, prospective students may not fully process the information--that is an argument that we often hear law professors make. But that does not alter our duty to provide "basic consumer information ... published in a fair and accurate manner reflective of actual practice." ABA Accreditation Standard 509.

At some point, all our lawyerly rationalizations will come to a bad end because a governmental agency or a court is going to challenge our right to self-regulation, thus ushering in a truly disgraceful chapter in the history of American legal education.

Now is one of the very few moments in our careers as academics where we have to make hard choices and demonstrate that we warrant the trust and respect of our tenured positions. Through our governance organizations (ABA, LSAC, NALP, AALS), we need to implement a system of complete transparency on employment outcomes. If the system has real teeth, it will force us all to work very hard to ensure we are delivering value commensurate with the tuition dollars we collect.

It's the end of the road. We likely have one last chance to get it right.

19 October 2010

Over at Balkinization Brian Tamanaha (Wash U) posted graphs endeavoring to describe law student enrollment and BLS data on legal employment trends (2001-09). Visually, the two graphs support Brian's assessment: "Law schools thus responded to the worst recession in the legal market in at least two decades by letting in more law students." As Brian Leiter (Chicago) notes, however, "One can't tell, though, from the second chart [and the underlying BLS data] what portion of the downturn in 'legal employment' is a reduction in the employment of attorneys as opposed to other law-related employees."

14 October 2010

Over at The Faculty Lounge Al Brophy (UNC) has fun with the recent spat of various rankings (here). As Al notes: "A couple of things stand out here -- there's a high correlation between the Leiter ranks and the Sisk ranks (.7) -- but there's a really high correlation (.96) between the Leiter ranks and the US News peer assessment data. The Leiter voters and the US News voters are coming to very similar conclusions...."

31 July 2010

In a recent
story in the San Jose Mercury News, the dean of Stanford Law
School, Larry Kramer, announced his intention to make Stanford the #1
law school in the country. As noted
by Brian Leiter, the story did not quote Kramer as saying that he
wanted Stanford to be #1 in the U.S. News rankings. The
reporter, nonetheless, inevitably gravitated to this dominant measure. As
noted by Dan Filler, most students, alumni, employers, and
bystanders, cannot get their heads around the meaning or usefulness of
alternative measures.

Fortunately, Larry Kramer's comments have
served up an opportunity to grasp some of the limitations and folly of
the current dominant method of ranking law schools. (And by the way, I
am certain that Larry Kramer knows most of what I am about to write--he
definitely understands the math. He was making comments for public
consumption. Like Kramer, I am trying to make Indiana Law #1 in the
nation, albeit not in US News sense. Every faculty member at
every law school ought to share this aspiration.)

The Research
Question: What will it take for Stanford Law School to be ranked #1
in the U.S. News Law School rankings?

Andy
Morriss and I annually construct a simulation model of the US
News rankings. It routinely explains 99.5% of the variance in the
actual rankings, and it enables us to test endless "what if" scenarios.
So let's load the dice and make some really extravagant assumptions:

Academic Reputation (25% of the input formula). Increase
Stanford's Academic Reputation from 4.7 to a perfect 5.0 (surpassing
Harvard and Yale, who languish at 4.8).

Lawyer/Judge Reputation (15%). Increase the Lawyer/Judge
Reputation from 4.8 to a perfect 5.0 (breaking their current 4.8 tie
with Harvard and Yale).

UGPA (10%). Increase median UGPA from 3.88 to 4.0 (Harvard
and Yale are at 3.89 and 3.9 respectively).

LSAT (12.5%). Increase the median LSAT from 170 to 180
(blowing past the 173 medians at Harvard and Yale). This assumption is
somewhat absurd because there are not enough perfect 180 scores to produce this median at any law school.

Surely, giving Stanford a perfect score on 62.5% of the US News weighting formula will make Stanford #1, right? It turns out, the answer is no. Yale
still beats out Stanford by a hair. Stanford would, however, finally
muscle ahead of Harvard for #2.

It is fair to ask, what is it
about the US News ranking formula that produces these peculiar
outcomes? The answer is standardization of the underlying inputs. In
order to make things like library books, LSAT scores, and employment
figures commensurable, US News converts the underlying arrays
into standard deviation units where the mean is roughly equal to zero and approximately 66% of the input scores fall within one standard deviation
above or below the mean. The US News weighting formula is then
applied to each USN input and the weighted inputs are summed to produce
the final US News score.

Many of the underlying input
distributions, however, do not reflect a typical normal distribution (i.e., bell curve). They are skewed. The reputation variables are roughly
normal, and therefore they are unimportant--or, more accurately,
unlikely to strongly influence school movement. Likewise, UGPA and LSAT
have a natural cap (4.0 and 180) that limits outlier scores. And
outliers are what skew distributions.

The biggest "uncapped" input
variable is direct per student expenses at 9.75% of the input formula.
In theory, there is no limit to how expensive legal education can be.
Yale, it turns out, spends over $100,000 per student, not counting
financial aid, which is a separate variable accounting for 1.5% of the
input formula. In contrast, Harvard and Stanford spend only $80K+ per
year per student. In standard deviation units, that translates into 5.8
for Yale versus 4.2 for Harvard and 4.5 for Stanford. So Stanford has a
1.3 standard deviation gap to close. Reputation, LSAT, and UGPA are
not enough to overcome this deficit.

Implications

My
back of the envelope calculations suggest that a check for $350 million
ought to be enough to produce enough endowment income to eclipse Yale in
the US News rankings. This assumes that the money is used for
things like books, more faculty, and higher salaries for everyone. If
the money is spent on student scholarships, however, Stanford would need
a check for roughly $1.8 billion to be #1. Again, these are the
idiosyncrasies of the dominant method of law school rankings.

Stanford
alumni are well heeled. And, like all of us, they want to be #1. But
here is my my advice: Before you write your check to give Stanford faculty a
big pay raise and a lower teaching load, or give some rich kid with a
180 LSAT a free legal education, including living expenses, consider
other investment opportunities. $350 million would go a long way to
solving the AIDS epidemic an Africa. $1.8 billion could provide
life-altering educational opportunities for children mired in poverty.

The
legal profession, especially our students, have some big problems at
the moment. And society's are even larger. The best law school is one
that prepares its students to solve these problems. This requires a
careful balance of innovative teaching and scholarship. The U.S.
News rankings don't capture these metrics. In fact, they obscure
them and create incentives for truly destructive behavior. By and large
the deans are trapped. From my own perspective, I don't think even one
law school in the US News Tier 1 has reached even 10% of its
potential to educate and solve problems. Too many one-professor silos.
Too much ego.

I am sorry to moralize. But someone needed to
say it. Let's focus on some problems worth solving. At the end of the
day, it will be worth it.

13 May 2010

David Hoffman alerted me to the recent news concerning the ELS Blog's very own David Stras. According to news reports, this morning Minnesota Governor Tim Pawlenty appointed David (all of 35 years-old) to the Minnesota Supreme Court as an Associate Justice. Congratulations to David.

20 April 2010

It is with great pleasure that I note that my friend, frequent coauthor, and Cornell colleague--Theodore Eisenberg--is among the eight law professors recently elected to the American Academy of Arts and Sciences. A well-deserved and long overdue honor. The complete list is here.

24 March 2010

Today's New York Timesreports that one of California's responses to its crushing fiscal mess is to implement a new (and much more aggressive) early release program for non-violent
prisoners who are unlikely to recidivate. As Laura Appleman notes over at The Faculty Lounge, California's new early release program is being closely watched by other cash-strapped states also looking to reduce spending. Assuming that at least some states do not experiment with dramatic changes to early release programs, however, California's program might generate helpful data for those seeking to test the efficacy of its new policy.

09 November 2009

The good folks over at the Computational Legal Studies blog have an interesting take on the recently-passed H.R. 3962 (health care). Befitting the blog's overall "computational or complex systems" approach, the post contains more than a few interesting factoids. As Dan Martin Katz notes, "The bill is 1990 pages as 'typeset' by the House. However, the 'substantive' words contained in bill are no longer than the length of a Harry Potter Novel."

"In Atkins vs. Virginia, the Supreme Court declared that evolving
standards of decency and the Eighth Amendment prohibit the death
penalty for individuals with intellectual disability (formerly, "mental
retardation"). Both supporters and opponents of the categorical
exemption, however, have criticized the Atkins opinion. The Atkins
dissent, for example, urged that the decision would open the gates of
litigation to a flood of frivolous claims. Another prominent criticism,
heard from those more supportive of the Court's ruling, has been that
the language the Court usedcommunicating that states must 'generally
conform' to the clinical definitions of mental retardationis ambiguous
enough to permit states to stray from the clinical definitions and,
consequently, for death eligibility to vary depending upon the
jurisdiction in which a defendant is charged. This Article presents
preliminary data responsive to these issues, and reports three basic
findings. First, Atkins has not opened floodgates of non-meritorious
litigation. Second, the success rates for Atkins claims vary
dramatically between states and state deviations from the clinical
definitions appear to have a palpable impact. Third, as compared to
their representation on death row, African-American defendants both
file and win a disproportionately high number of Atkins claims."

24 September 2009

Reaction to a recent post on the Wall Street Journal Law Blog (here) makes it clear that my colleague Ted Eisenberg's (Cornell) data-driven critique (forthcoming in JELS) of the US Chamber of Commerce's annual state "rankings" of perceived "lawsuit climate" struck something of a raw nerve. As the WSJ notes, "The Chamber’s Institute for Legal Reform publishes an annual survey,
conducted by the Harris Poll, which ranks states from best to worst,
according to how their judges treat class actions, the prevalence of
punitive damages, and juries’ “predictability and fairness” among other
factors." The WSJ's brief summary of Ted's critique of the survey follows:

"Eisenberg concludes that the survey itself is logically inconsistent.
Respondents, for example, he writes, make incorrect conclusion about
which states are the most favorable towards punitive damages due to
erroneous impressions among the respondents or even misrepresentations
of a state’s laws by the Chamber. He says that a state’s movement in
the ranking — such as New Jersey’s, which plummeted in one year — could
not be validated by a detailed analysis of legal events in the state.
Another problem: respondents are given the prior year’s rankings before
offering their opinions. This reduces the likelihood that a state could
shake an unfavorable image, even if state law there had changed."

29 June 2009

NALP has just posted its entry-level starting salary for class of 2008--i.e., the lawyers who started their jobs just as Bear Sterns and Lehman Bros unraveled and the credit markets completely froze up. Of the 22,305 law school graduates in NALP's sample (over half of all 2008 graduates), a remarkable 23% (5,130 '08 grads) reported an entry-level salary of $160,000. In contrast, 42% of entry level lawyers reported salaries in the $40,000 to $65,000 range. Once again, the central tendencies are a poor guide to the distribution as a whole: whereas the mean salary is a $92,000, the median salary was $72,000. Further, the two modes ($50,000 and $160,000) are separated by $110,000.

Amidst all the layoffs, deferrals, salary cuts, and apprenticeship programs announced in 2009, it is safe to venture that the bi-modal era has peaked. Every law school class for the foreseeable future will graduate to a much different economic landscape. Although many students will regret the opportunity to earn such a big payday upon graduation, it brought with it intense billing pressure, client resentment, heavy leverage, and very little substantive training for new hires. I would argue that profession as a whole (including current and future graduating classes) is better off with a lower entry level salary.

Admittedly that is a long-term view for the profession as a whole. In the short term, current students and recent graduates are in a world of hurt. Specifically, law school debt loads continue to climb. Thus, law schools are (rightfully) going to be under increased pressure to deliver value to our students. I don't think most law professors and law school administrators fully appreciate the difficult times ahead. For a provocative take on the current state of legal education, see Paul Lippe, Welcome to the Future: Time for Law School 4.0.

After the jump are the distributions from 1991, 2006, and 2007. The primary takeaway is that the bi-modal did not exist in the early 1990s. It first emerged in 2000 (with the dot.com salary wars) and became progressively more extreme starting as the decade unfolded. On Wednesday, I have an article coming out in the NALP Bulletin, entitled "The Bursting of the Pedigree Bubble," which will provide some additional analysis.

08 May 2009

As long as every other interested party is weighing in on Pres. Obama's first S.Ct. nomination, the least I can do is endorse Dave Hoffman's proposal for a nominee that, at a minimum, possesses a general familiarity and comfort with empirical methods. Although I can understand it when judges choose to simply ignore salient (and, I hope, well-crafted) empirical evidence brought before them (however much it might pain me), when judges (especially Justices) choose to engage with empirical work, however, they really should avoid botching things. In Exxon, Justice Souter's recent high-profile bungling (or "debacle" as Dave notes here; subsequent scholarly work on the issue is found here) falls below what I expect from my law students. Dave's conclusion warrants repetition:

"The point is that at least some exposure in statistics and social science techniques is quickly becoming part of a well-rounded legal education. It should also be part of what we look for in a Justice."