September 24, 2016

Wait a Minute! Let's Think About the Mathematics of Bar Passage

For most states, you must graduate from an ABA accredited law school to be eligible to take the bar exam at all. As Florida has been the main topic of conversation in the current discussion, I will use it as an example. Unless an applicant has been in practice for ten years in another jurisdiction, all applicants for the Florida bar must attend an ABA school. See Florida Rules of the Supreme Court Relating to Admissions to the Bar 4-13.1(a) & 4-13.2. The out-of-state practitioner rule seems to be new as passage rates in this category only exist since 2015.

Florida had a 68.2% success rate on the bar last July. SeeFlorida Results. (The success rate for practitioners is just slightly better at 68.9%). All of the non-practitioners—and probably most of the practitioners, too—are graduates from ABA schools.

It is not possible for all ABA schools to achieve an 85% (or even 75%) pass rate in Florida. Despite what is said on A Prairie Home Companion, all of our children cannot be above average. The reality is that if one ABA school gets a passage rate that is above the state average, another one will be below it.

Further, the passage rate trend on the Florida bar raises an important question. As the table and chart of July pass rates below indicates, the pass rate on the Florida bar has been declining fairly dramatically over the last eight years:

This declining passage rate can have two sources. One contributing factor, as much of the discussion about bar passage rates have assumed to be the sole factor, is that the quality of the people taking the bar exam has declined. The other possible factor that the discussion has ignored is that the bar exam itself has become more difficult.

There is some support for this second factor being important in the percentage of practitioners who pass. There is not a significant difference between that rate and the overall rate. All of the practitioners presumably graduated from law school at least a decade ago and before the current shrinkage of the law student applicant pool occurred. If the current admission practices of law schools is causing the bar passage decline, how do we explain that practitioners are doing no better?

It would be interesting in seeing a longer term trend on bar passage (and LSATs). It seems that everyone keeps comparing bar pass rates to their historic highs which is misleading to the conversation since it is an historic high. What is the 30 year average? Every year the news comes out that bar passage rates are at their lowest since Xyear. Were law school grads less competent in Xyear? Was there media outrage about the pass rate in Xyear? Or was it just accepted that the Bar is a tough test and that some people will fail? Where did this expectation that everyone will pass the bar come from? Is this expectation of a high pass rate just another form of a grade inflation for an already entitled generation? Why are we so paternalistic about allowing people to invest in themselves? I wish I had answers but all I have are questions.

"Twbb, the problem is that two variables change over the life of a standardized exam like the MBE: the difficulty of the exam (because there are new questions every time) and the competence of the test takers. It's hard to equate when two variables are in flux."

Hard, but not impossible; the equating process is one means to resolve it, and while it's not 100% effective it does at least lend some validity to the results. Statistics isn't just math; it's research design, and the
NCBEX has trained statisticians on staff, and I guarantee you they don't ignore these issues.

This actually cuts against your point; the test is equated and scaled to create a median of about 150, year in and year out. A steadily decreasing average test taker competence means that people who would have failed in past years are more likely to pass than the reverse. Your argument seems to be that despite this, a sudden influx of poor performers in a state will unfairly cause a certain group of otherwise passing but borderline students to fail. I suspect that threat is overblown, primarily because the correlation between academic predictors and bar passage appears to be non-linear, so there are probably less of these students than you think. Additionally, this can be empirically tested by looking at schools that maintained admissions standards at the same time their state saw such an influx. I am open to being proven wrong, but I suspect that if this group of borderline students exist it's not a large one.

As a side note, I don't think it's valid to assume the addition of civil procedure made the test more difficult in any significant way. 7 subjects instead of 6 isn't a big leap, the addition of Constitutional Law back in the day didn't seem to significantly impact bar passage rates, and in many cases students already have to learn civil procedure for the state portion anyway, so they've already studied for it.
SSstudents at other schools in the state to fail the bar--even if those schools have not changed their admissions standards or educational practices.

The opening post in this thread is a complaint that the Florida bar is too tough - and on its face seems to be special pleading by certain Florida law schools that their students are failing the bar. I am, as you know, well aware that you have pressed for tougher standards for law schools.

I made two points with respect to bar exams - the main one is as follows: assuming arguendo that the current bar exam is an imperfect test (an argument I agree with by the way) in that it allows people to pass who are, in reality, persons who should be rejected, while at the same time rejecting some who should be admitted. The rejection as Deborah and others have put it is a result both of the exam format and the "cutoff."

Even if one accepts that the exam is an imperfect exam, there has to be a score, a level of performance that is so low, that the examinee has no prospect of being a competent lawyer. We see this discussion duplicated with LSAT scores, where some scores are reach the level where randomly picking answers would deliver nearly the same result. How low is the score on the current bar exam that indicates that someone really has no business practicing law? Moreover, since the bar exam is a measure primarily intended to protect the public - how low should we let the score go, before we conclude that the public, the consumer's of legal services, would be at risk if the candidate was allowed to become a lawyer?

One could conclude that there will be no hurdle at all, no test of knowledge and ability, for entering the legal profession. If people think that would be a good thing, well.... But as long as there is going to be a test, a bar exam of sorts, and a need for people to get a legal education at a significant cost in time and money, the a priori question before they invest that time and money needs to be asked - is this person likely to clear this hurdle?

My basic objection to the debate about the bar exam - as it is presented by several commentators - is that it is seeking ex-post-facto justification for admitting students who, as David Frakt put it, are at very high risk or extreme risk of failing the bar. I am not accusing Deborah of that, but I making the point that at the end of the day what law schools "pitch" consists of is "borrow a lot of money, give it to us, and we'll get you through the bar."

I found some data from the NBCE statistics archive about Florida Bar overall pass rates over a longer period of time. From 1997-2006 the overall pass rate ranged from a high of 79% to a low of 60%. Six of those ten years had pass rates below 70%. When we look at the longer term trend it doesn't seem that the sky is falling so much as we had a bubble and it is now correcting itself.

You guys are really verbose. If there is even one lawyer who desires full time sustainable middle class legal employment and can't find it, it is called a failure. No amount of academic wrangling will cure that.

Deborah, you appear to think the skill level required in the practice of law is akin to something like RN level nursing or teaching 7th grade. Traditionally (i.e. before the profession was trashed by lower standards) becoming an attorney was thought to require a doctor's level mentality. I have been practicing in commercial litigation for 6 years now, and I can assure you that everything that I do of any importance is more difficult than the material on the bar exam. A lot school graduate should be able to write a judicial opinion of medium complexity that is publication worthy, just like an new engineer should be able design a product that works, a new doctor should be able to correctly diagnose a condition, or an aspiring professor should be able to write a publication worthy piece of scholarship. How many law schools grads could actually write a publication worthy judicial opinion that you could teach in class? Probably no more than 15%. You keep speaking of the requirements to practice law, but it is clear you have no clue.