Tomorrow, the law school world will overreact to slightly-shuffled U.S. News rankings. Proud alumni and worried students will voice concerns. Provosts will threaten jobs. Prospective students will confuse the annual shuffle with genuine reputational change.

Law school administrators will react predictably. They’ll articulate methodological flaws and lament negative externalities, but will nevertheless commit to the rankings game through their statements and actions. Assuring stakeholders bearing pitchforks has become part of the job description.

If the rankings measured something useful, the entire charade would be much easier to stomach. The unfortunate irony is that these rankings adversely affect the decision-making process for law school administrators and prospective law students alike. The stakes are high. Our profession and society need law schools that don’t figure inefficient metrics into annual budgets. Dollars spent chasing U.S. News rankings diverts funds away from students’ education. It also stands in the way of reducing tuition.

In this post, I examine five U.S. News rankings failures. I consider the methodology and underlying rankings theory from the perspective of a student who features job prospects prominently in his application and enrollment decisions. Considering the near universal support for prioritizing job outcomes in the process, these failures demonstrate just some of the reasons annual consternation hardly seems worth it.

Word cloud of LSAT test-takers surveyed about the reasons they plan to obtain a J.D. degree. Source: Above the Law.

First, the rankings pay insufficient attention to what matters most to prospective students: job outcomes. In a survey of 600 students studying for the October 2012 LSAT, Breaking Media’s research arm found that the two most popular words associated with the students’ purpose of getting a J.D. were “career” and “work” (image above). These are not exactly shocking results. Despite the importance of job outcomes, they account for only 18% of the rank and credit schools for jobs few attend law school to pursue.

Second, the rankings use a national scope, which places schools on the same scale. Only a handful of schools have a truly national reach in job placement. The rest have a regional, in-state, or even just local reach. The relative positioning of California Western and West Virginia in the rankings is virtually meaningless. Graduates from these schools do not compete with one another.

Percentages in above graphic are of the entire class.

It turns out that 158 schools place at least half of their employed class of 2013 graduates in one state. The top state destination for each school accounts for 67% of employed graduates. A much smaller 8.2% of employed graduates go to a school’s second most popular destination, with just 4.5% of employed graduates working in the third most popular destination. Only 20.4% of employed graduates (16.7% of the entire class) end up in a state other than the top three. Comparing schools across the country just doesn’t make sense.

Third, U.S. News rankings follow an ordinal system that fails to show the degree of difference between schools. Are Columbia and NYU virtually tied? Or does the two-rank difference represent a wide gulf in quality? Is the so-called difference between Columbia and NYU the same as the difference between Cornell and Vanderbilt? Students weighing school prices need to know not just which school is better but how much better it is.

Fourth, performance changes over time but year-to-year comparisons are virtually impossible using the U.S. News rankings. U.S. News will tell you that Stanford knocked Harvard out of the #2 spot in 2012-13, but the swap in rankings does not provide any clues to your average reader as to why. Stanford may have improved while Harvard declined. Or, Stanford may have improved while Harvard’s quality stayed the same. Or perhaps both schools saw a decline in quality but Harvard’s decline was more severe. In fact, if every single school saw a marked decline in quality the U.S. News rankings would not indicate that this happened. Instead, students can know only relative performance.

Finally, U.S. News inexplicably places every ABA accredited law school on the list of "The Best." The best at what? U.S. News doesn’t say. But it implies that every school on the list is good. The truth is that once costs and employment outcomes are considered in comparison to personal career goals, many schools are bad choices. The U.S. News rankings provide no help in drawing the line.

Rankings are not inherently bad. In fact, they are conceptually quite useful. They order comparable things to help people sort through more information than they know how to or can weigh. However, ranking credibility may be lost when methodologies are unsound, through irrational weighting or meaningless metrics, or when the scope is too broad. The legal profession is worse off for elevating the importance of a publication that falls victim to these flaws each and every year.

I think your article was great, but I do wish there was a bit more detail about the actual "methodology" that US News uses. I was vaguely familiar with the methodology, but after looking at http://www.usnews.com/education/best-gr ... s-rankings , I now feel even more disgusted by how nonsensical their approach to ranking schools is, and how much weight 0Ls place on them.

jenesaislaw wrote:However, ranking credibility may be lost when methodologies are unsound, through irrational weighting or meaningless metrics, or when the scope is too broad. The legal profession is worse off for elevating the importance of a publication that falls victim to these flaws each and every year.

jenesaislaw wrote:However, ranking credibility may be lost when methodologies are unsound, through irrational weighting or meaningless metrics, or when the scope is too broad. The legal profession is worse off for elevating the importance of a publication that falls victim to these flaws each and every year.

I found this article to be very misleading, alarmist, populist, click-baity, etc. There are definitely problems with this set of rankings, and obviously noone should be relying entirely on them --but they're the gold standard and as all parties involved (students, deans, faculty, firms, judges) continue to rely upon them, the significance compounds. Of course people will like taking shots at them, because agreement rarely gets attention --but I think this article is a particularly bad account.

I wrote a criticism of this in the other thread that this was posted in back in March in the Choosing a law school forum (where this probably exclusively belongs)...but since this has been stickied, I'd like to also cross post my response (viewtopic.php?f=1&t=244746#p8461468) which unfortunately never got a rebuttal. Perhaps some of my concerns can be addressed now that this is stickied.

lacrossebrother wrote:Dumb article.1. the goal is to come up with some proxy to determine what the best law schools are. anyone can list last year's employment rankings. you'll find however that those numbers fluctuate. the fact that everyone says you want to be a lawyer --that's the point of the lsat --doesn't mean that the fucking purpose of legal education is to maximize employment outcomes only.

what's fucking even more insane is that you started an entire website called "law school transparency" whereby you challenge the notion that a single percentage of employment is an appropriate means to gauge employment . so you fucking say "only 18% of your ranking is based on this really important number..." even though you think it's a shitty number to begin. dope.

moreover, "only 18%"?? you give us absolutely no idea if that's actually high or low. do a sensititivy analysis with this number you hate so much and then offer your still super layman's opinion that "this is failure number 1."

2. i don't even get this point. it's impossible to try to determine if university of arizona is a better school than uconn because people who go to school in arizona won't want to work in uconn? No shit. Perhaps this is why they don't do a fucking 100% employment ranking?? Instead, they ask judges and employers to rate their perception of the products of these schools. They try to rate the faculty. And ya, they take a look at the fucking incoming product because it's pretty easy to see that if one school, on average, has a bunch of dopes attending and the other doesn't, the fact that they aren't in competitive markets doesn't preclude reaching a conclusion. They try to gauge the respective quality of the school's infrastructure and commitment to research by looking at the size of the library (volumes is an imprecise measure, but whatever). The fact that they attempt to do this for everyone, for their magazine, doesn't make the whole thing a failure.

3. There's a raw score you fucking moron4. it's a failure that the magazine doesn't do a more in-depth analysis of each 2-3 swap? Mr. "only 18%" is upset that the magazine doesn't explain with data graphics somewhere what the most heavily involved coefficient contributing to each swap. 5. ok ya i agree this is dumb. what failures.

lacrossebrother wrote:moreover, "only 18%"?? you give us absolutely no idea if that's actually high or low. do a sensititivy analysis with this number you hate so much and then offer your still super layman's opinion that "this is failure number 1."

[/quote]

So basically they did a survey accounting only for 18 percent of the rank and expected law students to come over to these schools due to authority and popularity. Obviously the idea is gold, and misleading, and it is funny. 18 percent is a bad number, do we need to argue with this? 18/100 isn’t even 1/2 of the 1/4 of 100. So tell me why it is a good number and i will still not believe.

lacrossebrother wrote:I found this article to be very misleading, alarmist, populist, click-baity, etc. There are definitely problems with this set of rankings, and obviously noone should be relying entirely on them --but they're the gold standard and as all parties involved (students, deans, faculty, firms, judges) continue to rely upon them, the significance compounds. Of course people will like taking shots at them, because agreement rarely gets attention --but I think this article is a particularly bad account.

I wrote a criticism of this in the other thread that this was posted in back in March in the Choosing a law school forum (where this probably exclusively belongs)...but since this has been stickied, I'd like to also cross post my response (viewtopic.php?f=1&t=244746#p8461468) which unfortunately never got a rebuttal. Perhaps some of my concerns can be addressed now that this is stickied.

lacrossebrother wrote:Dumb article.1. the goal is to come up with some proxy to determine what the best law schools are. anyone can list last year's employment rankings. you'll find however that those numbers fluctuate. the fact that everyone says you want to be a lawyer --that's the point of the lsat --doesn't mean that the fucking purpose of legal education is to maximize employment outcomes only.

what's fucking even more insane is that you started an entire website called "law school transparency" whereby you challenge the notion that a single percentage of employment is an appropriate means to gauge employment . so you fucking say "only 18% of your ranking is based on this really important number..." even though you think it's a shitty number to begin. dope.

moreover, "only 18%"?? you give us absolutely no idea if that's actually high or low. do a sensititivy analysis with this number you hate so much and then offer your still super layman's opinion that "this is failure number 1."

2. i don't even get this point. it's impossible to try to determine if university of arizona is a better school than uconn because people who go to school in arizona won't want to work in uconn? No shit. Perhaps this is why they don't do a fucking 100% employment ranking?? Instead, they ask judges and employers to rate their perception of the products of these schools. They try to rate the faculty. And ya, they take a look at the fucking incoming product because it's pretty easy to see that if one school, on average, has a bunch of dopes attending and the other doesn't, the fact that they aren't in competitive markets doesn't preclude reaching a conclusion. They try to gauge the respective quality of the school's infrastructure and commitment to research by looking at the size of the library (volumes is an imprecise measure, but whatever). The fact that they attempt to do this for everyone, for their magazine, doesn't make the whole thing a failure.

3. There's a raw score you fucking moron4. it's a failure that the magazine doesn't do a more in-depth analysis of each 2-3 swap? Mr. "only 18%" is upset that the magazine doesn't explain with data graphics somewhere what the most heavily involved coefficient contributing to each swap. 5. ok ya i agree this is dumb. what failures.

you mean you wrote a fucking criticism of this fucking thing in the other fucking thread way back in fucking March. Fuckkkkk. to be fair i fucking agree with you. bro

runinthefront wrote:I think your article was great, but I do wish there was a bit more detail about the actual "methodology" that US News uses. I was vaguely familiar with the methodology, but after looking at http://www.usnews.com/education/best-gr ... s-rankings , I now feel even more disgusted by how nonsensical their approach to ranking schools is, and how much weight 0Ls place on them.

Yeah, it's really interesting to see how much weight is put on factors that, in many cases, won't even matter to students. LST has a system where the applicant picks what is most important to them, and then it rates law schools based on those factors. Personally I prefer LST's system, but the different systems (USN, ATL, LST, etc.) for the most part don't end up showing much difference. T14s remain T14s.

artistar wrote:The point that schools are spending money to improve rankings when they could be putting it towards lowering the cost of tuition is a good one. I've read many of these articles, but few note that.

But it feels so much cooler to get a conditional 30k scholarship at a 50k/yr school than to get no scholarship at a 20k/yr school.

lacrossebrother wrote:I found this article to be very misleading, alarmist, populist, click-baity, etc. There are definitely problems with this set of rankings, and obviously noone should be relying entirely on them --but they're the gold standard and as all parties involved (students, deans, faculty, firms, judges) continue to rely upon them, the significance compounds. Of course people will like taking shots at them, because agreement rarely gets attention --but I think this article is a particularly bad account.

I wrote a criticism of this in the other thread that this was posted in back in March in the Choosing a law school forum (where this probably exclusively belongs)...but since this has been stickied, I'd like to also cross post my response (http://top-law-schools.com/forums/viewt ... 6#p8461468) which unfortunately never got a rebuttal. Perhaps some of my concerns can be addressed now that this is stickied.

I'm facing a bit of a crisis. So I took my LSAT for the first time in October 2015. I studied so hard and was in the 150s for my practice exams. Unfortunately, I only got a 149 on the exam. (My gpa is terrible its 3.3 - but I did work for Legal Aid for 2 years and also volunteered in Sudan)

I decided to apply see which schools I would get into and CUNY law was one of the schools that accepted me. At first I was excited about this because CUNY is cheap, and it focuses on public interest (Which is what I'd like to study). However, I am starting to feel anxious because I know that I could've done much better on the exam plus I have read so many negative comments about CUNY on this website and countless forums.

I already submitted my deposit to CUNY but I am not sure if I should retake the exam (is it too late?), go for the first year and transfer out, or write a letter of continued interest to St. John's University (waitlisted me).

I'm facing a bit of a crisis. So I took my LSAT for the first time in October 2015. I studied so hard and was in the 150s for my practice exams. Unfortunately, I only got a 149 on the exam. (My gpa is terrible its 3.3 - but I did work for Legal Aid for 2 years and also volunteered in Sudan)

I decided to apply see which schools I would get into and CUNY law was one of the schools that accepted me. At first I was excited about this because CUNY is cheap, and it focuses on public interest (Which is what I'd like to study). However, I am starting to feel anxious because I know that I could've done much better on the exam plus I have read so many negative comments about CUNY on this website and countless forums.

I already submitted my deposit to CUNY but I am not sure if I should retake the exam (is it too late?), go for the first year and transfer out, or write a letter of continued interest to St. John's University (waitlisted me).

lacrossebrother wrote:I found this article to be very misleading, alarmist, populist, click-baity, etc. There are definitely problems with this set of rankings, and obviously noone should be relying entirely on them --but they're the gold standard and as all parties involved (students, deans, faculty, firms, judges) continue to rely upon them, the significance compounds. Of course people will like taking shots at them, because agreement rarely gets attention --but I think this article is a particularly bad account.

I wrote a criticism of this in the other thread that this was posted in back in March in the Choosing a law school forum (where this probably exclusively belongs)...but since this has been stickied, I'd like to also cross post my response (viewtopic.php?f=1&t=244746#p8461468) which unfortunately never got a rebuttal. Perhaps some of my concerns can be addressed now that this is stickied.

Yet another year later, still no rebuttal. I feel bad for you.

Haha. Don't feel bad. LST has and will always remain a shittily run organization despite its noble intentions. They thrive on the law school is a "scam" concept, but can't see that the best way to fix "failures" is to work with the organizations, not against them.

Indeed, that a year has passed and US News remains the most important school ranker should make you pity LST. Not me. to be. Long live the king.

I'm facing a bit of a crisis. So I took my LSAT for the first time in October 2015. I studied so hard and was in the 150s for my practice exams. Unfortunately, I only got a 149 on the exam. (My gpa is terrible its 3.3 - but I did work for Legal Aid for 2 years and also volunteered in Sudan)

I decided to apply see which schools I would get into and CUNY law was one of the schools that accepted me. At first I was excited about this because CUNY is cheap, and it focuses on public interest (Which is what I'd like to study). However, I am starting to feel anxious because I know that I could've done much better on the exam plus I have read so many negative comments about CUNY on this website and countless forums.

I already submitted my deposit to CUNY but I am not sure if I should retake the exam (is it too late?), go for the first year and transfer out, or write a letter of continued interest to St. John's University (waitlisted me).

Please help! Any advice is helpful!

Screw it, I'll answer:

If your goal is public interest in NYC, CUNY might be a good option for you. It's certainly a cheaper and less risky option than other schools. If you're an NY resident, CUNY at sticker is still probably cheaper than most schools would be, even with a scholarship.

Job outcomes for lawyers are dismal. Anyone planning to go to law school these days is frankly, not that bright to begin with. If you want real, solid job opportunities, get a STEM degree. And grow up.

seanmitchell wrote:Job outcomes for lawyers are dismal. Anyone planning to go to law school these days is frankly, not that bright to begin with. If you want real, solid job opportunities, get a STEM degree. And grow up.