Search Site

Monday, July 01, 2019

In-Class Exams Are Probably Wrong

I enjoy Malcolm Gladwell's podcast on Revisionist History. One reason why is that he sometimes talks about the law. Last year he interviewed Michael Stokes Paulsen for an episode. This year he's done one with Bill Henderson. The latter was a critique of the LSAT and law school exams with short time limits.

I've long been bothered by the way law school exams are administered. Why should there ever be a closed-book exam? And why should you only get three hours for an exam? Law is complicated and almost always involves research. Neither of these traits are reflected in many law school exams. (And legal questions do not present themselves in a multiple choice format either.) This is why I've always used take-home essay exams for 2L and 3L classes and made all of my exams open-book.

Reluctantly, I've kept my IL exams as three-hour essay exercises. Why? I had two not-so-great reasons. One is that there are some circumstances (especially in a trial) where lawyers must work on tight deadlines. The other is tradition, I suppose. But I've now concluded that I was wrong. From now on, I'm going to make my 1L exams take-home as well. Maybe I'll start a trend.

Comments

@ Atticus Grinch,

Sure but it’s also true that being able to advocate while feeling awful (eg having a cold or flu) makes one a better advocate as does dressing well. Yet we don’t think it’s appropriate to demand students take tests in a 10” degree room (to make them feel awful) or to grade them on their sartorial choice. We certainly don’t evaluate students looks even though that too makes a difference when advocating (people do listen more to the attractive). You can’t bring out this argument only to defend the traditional class mechanics but not accept it as persuasive elsewhere.

There are already plenty of aspects of school that test for memorization and working under time pressure adding that component to every last course grade is a mistake and doesn’t create the right incentives for learning.

Posted by: Peter Gerdes | Jul 9, 2019 12:19:14 AM

@ Atticus Grinch,

Sure but it’s also true that being able to advocate while feeling awful (eg having a cold or flu) makes one a better advocate as does dressing well. Yet we don’t think it’s appropriate to demand students take tests in a 10” degree room (to make them feel awful) or to grade them on their sartorial choice. We certainly don’t evaluate students looks even though that too makes a difference when advocating (people do listen more to the attractive). You can’t bring out this argument only to defend the traditional class mechanics but not accept it as persuasive elsewhere.

There are already plenty of aspects of school that test for memorization and working under time pressure adding that component to every last course grade is a mistake and doesn’t create the right incentives for learning.

Posted by: Peter Gerdes | Jul 9, 2019 12:19:12 AM

Lawyers who can complete a task effectively in 3 hours are much more valuable to their employers than those who need more time. A short-fuse open book exam tests this skill.

Posted by: Atticus Grinch | Jul 5, 2019 8:04:42 PM

Ok, so I can't find Derek's articles. I guess it was a joke? I think there could be strong arguments for a 100% final, but I haven't actually seen research on this.

Posted by: Profanon | Jul 5, 2019 8:49:35 AM

Derek -- thanks for this, but do you have anything from the past 20 years?

Posted by: Profanon | Jul 5, 2019 8:44:20 AM

I confess to having tried everything--from 24-hr take-home to in-class M/C to in-class essay to in-class short-answer, open-book, closed-book, open limited materials (such as a clean copy of the FRCP or FRE). I don't know what's right, although I do know answers are easier to read and understand when not written under a time-crunch.

I will just echo a couple points that were made: A 24-hr exam works best when it is really a 3-hr in-class, stretched to a longer period to remove the time crunch. And the students should be told that, so they can prep accordingly. I also tell them that the opportunity to look at materials (especially in a take-home format) does not mean they should not create and study an outline before beginning the exam. They must be familiar with the material before beginning the exam, using the materials as a crutch and a quick spot-check. If they are flipping through the outline and trying to figure things out during the exam, it is too late.

Some of the valid concerns Paul raises can be addressed by imposing a tight word or page limit on a take-home, which eliminates the possibility of using the entire period. We can tell them to write to the word limit, then edit (and probably find time to sleep).

Posted by: Howard Wasserman | Jul 4, 2019 12:37:35 PM

Paul,

There is in fact quite a bit of research suggesting that 100% finals (especially those where the only feedback is a single number or letter grade) are not only a good pedagogical tool, but perhaps the best for assessing and improving student learning. I've listed some of the leading research on this subject below:

Kingsfield, Charles. "That's How We Did It When I Was In School." 21 Harvard Law Review 87 (1937).

Perini, Rudolf. "That's Also How We Did It When I Was In School." 89 Harvard Law Review 114 (1991).

Posted by: Derek Tokaz | Jul 4, 2019 9:11:53 AM

When I started teaching, I refused to give closed book timed exams because I thought they were artificially stressful and unlike anything lawyers actually do. I no longer give open book exams or take-home exams because I have found that students don't study well for them and miss out on the critical integration of the material that exam prep provides. They are lulled into false security by the availability of resources, and that hurts their ability to do well on the exam. That said, I always use many evaluative devices in every class, some timed testing, some reflective writing, some scenario-based simulated advocacy.

Posted by: Rebecca Bratspies | Jul 3, 2019 3:26:24 PM

To be clear something like a 24 hour test only is a good idea when you are designing the test *not* to apply time pressure, e.g., design a test you think should be able to be completed in 3 hours. Assuming your goal is to encourage students to understand concepts not memorize the 21 extra hours available ensure students won't even suspect they will benefit from memorizing the right material but, unlike a similarly short test given with a week's time, don't have the time to learn only those concepts they see on the test.

Now it's quite plausible you do want to test memorization in some circumstances. But I do think it's fair to presume anyone capable of mastering the concepts will manage to remember whatever facts they end up using frequently and that concepts stick while mere memorization often vanishes pretty quickly rather than providing a long term skill.

Posted by: Peter Gerdes | Jul 3, 2019 8:26:55 AM

The concern about cheating is valid (although, in my view, insufficient; the goal should be to work incredibly hard, to the detriment of one's personal and research time as necessary, to craft the kind of take-home that is hard to cheat on). But the tail shouldn't wag the dog and cheating should not be what drives the how-to-test question. Even if we give in-class exams, there are still relevant questions, such as how many we should give over the course of the semester (okay, an in-class test--but why only one? Is there any research suggesting that 100 percent finals are a good teaching and evaluation practice?), what varied formats they should take, and what other forms of evaluation should also form part of the grade. These questions should be answered by the best available research on forms of evaluation and of learning and learning retention. Anecdotal views about practice and anecdotal fears about the bar exam are insufficient in themselves, not least because even if they are the goal to which we are striving, they don't tell us how best to achieve that goal for the largest number of students. I suspect that many law schools (but not all, and there are some more innovative schools and teachers out there) do way little to investigate and implement the best methods of teaching and evaluation. (I add that I'm certainly not at the head of the class here, although I've tried to make some changes over time.)

Posted by: Paul Horwitz | Jul 2, 2019 5:03:11 PM

“Why should there ever be a closed-book exam?“

Ever? Well, I can think of three reasons off the top of my head:

1. In part because there’s a bar exam, and if law students have *no* prior exposure preparing for time-limited closed-book testing, they’re going to struggle mightily to master that testing modality in the few weeks between graduation and the bar exam.

2. There are in fact scenarios in legal practice where one must do their best to recall and apply legal principles to a set of facts under time pressure and without materials in front of them.

3. Cheating.

That said: I don’t think that in-class closed-book exams should be the sole or even primary method of assessment, which is why I use multiple assessment methods in my courses.

Posted by: anon | Jul 2, 2019 3:38:52 PM

I don't ever give take home exams for this exact reason. I also never recycle used exams.

You should always assume that some percentage of the students are cheating with a take home. I think its more socially acceptable to give take home exams in the upper level because the grades aren't as important to the students, thus there is less incentive to cheat and less harm.

Posted by: Profanon | Jul 2, 2019 2:18:34 PM

If cheating is an issue, then why give take-home exams in large 2L or 3L classes? I think that objection proves too much.

Posted by: Gerard | Jul 2, 2019 1:16:48 PM

I completely agree with the last two comments. In class exams are needed to prevent cheating. I can guarantee you that some students will cheat. If you want to give students more time, simply include fewer issues. I cannot imagine giving a take-home test to 1Ls.

We really should only be testing what we are teaching. I try to teach doctrine and analysis, so I grade based on rule statements and application. Legal research is a valuable skill, but it should be tested only in legal writing or related classes.

For many of us, we also need to think about the bar exam. Some critical mass of professors need to give exams in a bar format. I think this is also the best argument for closed-book exams, though I am on the fence on this.

Posted by: Profanon | Jul 2, 2019 12:55:08 PM

Gerard writes: "Virtually all 1L exams are in-class and timed. So professors have reached a conclusion that that’s the best format. As you say, how can they know that?"

FWIW, I have always done in-class open-book exams for two reasons:

(1) They most closely match the kinds of problem-solving that I encountered in practice, which was at DOJ. My practice experience may be unusual, but I found that having 20 minutes to digest a hard problem and organize thoughts about it after having spent a lot of time learning an area beforehand -- what we call on students to do in exam -- was pretty representative of what lawyers in the criminal law area were called on to do.

(2) When I was a 1L, I heard stories about a group of students in another section cheating on a take-home exam. The rumor was that they picked up the exam, met together to confer on the best answers, and then went back to their rooms and wrote of their answers. I didn't know the students and didn't know if the rumors were true. But given how cutthroat Harvard was about 1L grades -- really, your entire existence and future was understood to be defined by that handful of grades -- I wasn't shocked that some students would try this if given the chance.

Posted by: Orin Kerr | Jul 2, 2019 12:24:50 PM

Sorry - meant to say "1L timed in-class exams," obvi

Posted by: Enrique Armijo | Jul 2, 2019 11:47:09 AM

1L timed exams have many flaws. Their primary benefit is that they can be proctored better than any alternative method of examination. This benefit outweighs most of the flaws. The stakes for 1L exams are much higher than other exams, in most ways one would define "stakes." Therefore, the reliability and consistency of the proctoring of 1L exams is much more important than the reliability and consistency of the proctoring of other exams, which in the take-home example isn't much more than student self-regulation. Which again, is more OK when the stakes of the results for the takers is relatively lower.

Signed, an associate dean

(I know that lawyers have to be honest and self-regulating, and part of law school is teaching students that lawyers have to be honest and to self-regulate. My comment describes the world that law schools and students in ultra-competitive environments live in, not one they aspire to.)

But I'm 100% with you on open vs. closed book!

Posted by: Enrique Armijo | Jul 2, 2019 11:46:09 AM

From my experience in practice, the primary obstacle to success does not generally come in finding case law, but rather in applying it. With enough effort, just about every associate is capable of finding the necessary case law to succeed; and although there are efficiency concerns that arise--nobody wants their associates over-billing for research--I don't think that research efficiency is something that profs are or should be teaching in substantive classes. In-class exams, and especially open-book in-class exams, test the latter--applying case law--but not the former--finding it. Besides being more important, applying case law is a much more difficult and much more important skill that IS and should be actually taught in substantive classes.

Posted by: a non | Jul 2, 2019 10:39:14 AM

I don’t think any exam can mimic practice all that well, but I don’t think it has to. Most law school classes just barely skim the surface of their subjects — the point of an essay exam, IMO, is to assess how much of those top-of-mind concepts the student has absorbed and is able to use to analyze a fact pattern. It’s stuff you shouldn’t have to look up if you practice in the area, or if you do, it’s just to get the exact wording right. At least, that’s how I design my exams, which are open book but without enough time to do research or re-read entire cases.

Separately I share Paul’s concern about 24-hour exams, which is why I make my take-homes either 8 hours or 10 days.

The doubt you express suggests that a mix of formats would be best. But virtually all 1L exams are in-class and timed. So professors have reached a conclusion that that’s the best format. As you say, how can they know that?

Posted by: Gerard | Jul 2, 2019 8:13:24 AM

While no exam format is perfect, let me suggest some reasons why a 24-hour take-home exam is a problem, at least unless the parameters are carefully set in advance. For those who are in good or even decent health, the 24-hour exam can become a competition over who has the most endurance and can pull an all-nighter. For those who are in poor health or have medical problems (including me, both now and when I was in law school), that’s either impossible or injurious. Leaving aside whether it’s bad for even though it was in decent health, and without any desire to be overly paternalistic, we should be testing only for learning, thinking, writing, and editing skills and not for extraneous (in the sense that they’re not what we’re setting out to test for) factors. I prefer a take-home exam length that either makes working all night impossible, like eight hours, or one that makes it less likely or necessary, like 48 or 72 hours—PLUS time and word limits, which are essential. I think (and hope) many of us agree that even a better 100-percent final is less than ideal based on pretty well all available research, and that law professors should test multiple times and in multiple formats over the semester. This has the added plus of being laborious and requiring schools and their administrators to change the typical calendar and format of the law school semester.

Posted by: Paul Horwitz | Jul 2, 2019 7:17:41 AM

This is exactly right and not just in the law but everywhere.

I think the ideal time limit is more something like 24 hours or the like because if you really give unlimited time you end up giving students who really have lots of time *that week* an advantage over those who have jobs, other classes etc. that week. I mean it does matter if you know how to look things up in a relatively short time as opposed to having to laboriously slog through them.

Having gone to a school where we had a working honor code that was taken seriously I can say that 24 hour exams are a bitch to do but a great way of testing ability. I still think totally untimed is better than an inclass exam since few people are able to do well but couldn’t manage to do that in a reasonable time but forgetting a key fact or two which happens to be on the exam is common and you can make sure the exam is simple enough not to benefit from time past the one day mark or whatever.

But if you can’t trust the students not to cheat you’ve got a bigger problem with take home exams.

Posted by: Peter Michael Gerdes | Jul 2, 2019 2:56:21 AM

"Law is complicated and almost always involves research. Neither of these traits are reflected in many law school exams."

Maybe that's right, but I don't know the right way to think about it. Say you have a month to study for a test -- to study the law, to master the material, to understand the scope and depth that is tested -- and then you have three hours to take the test. Is the three hour test just evaluating short-term reaction (the three hours), or is it evaluating long-term preparation (the month), or some mixture of both? And if it's both, is the mixture all that different from law practice? And how can professors of all people know?