Testing students during video lectures improves learning

In recent years, massive open online courses (MOOCs), through the likes of Coursera, have attracted hundreds of thousands of students from across the world. Many teachers are using a "flipped" classroom model, where students take lectures through videos at their convenience and spend the time in class delving into issues they don't understand.

But are they really improving learning? The evidence is not fully convincing. A 2010 meta-analysis of the literature on online teaching by the US Department of Education revealed that there are only "modest benefits" to online learning compared to classroom learning, but more rigorous studies were needed. After all, ease of access to the learning material comes with a bundle of distractions only a single click away. Not surprisingly, studies have shown that students suffer from attention lapses when learning through videos.

Given those findings, an improvement in students' attentiveness is bound to pay significant dividends. To that end, Karl Szpunar, a cognitive psychologist at Harvard University, might have a rather simple solution to rein in distractions, one that focuses attention in real-world classrooms: intersperse pop quizzes into the online lectures.

Szpunar got this idea from a 2008 study he conducted. In it, students learned five lists of words; half the students were tested after each list and the other half were tested only after the fifth list. Both groups were then tested on the cumulative list of words. Those tested after every list, not surprisingly, performed better on the cumulative test. But the results also suggested that being continually quizzed helped focus the students on the task at hand. Both groups had the same gap between seeing the fifth set of words and being tested on them, yet the group that was tested along the way performed twice as well on this set.

In the new study, he did something similar. In the first experiment, he divided a 21-minute lecture into four segments. After each segment, he asked half the students (the tested group) to perform arithmetic tests followed by tests on the subject matter of the lecture. The other half (the non-tested group) just performed arithmetic tests after each segment and took a subject matter test at the end of the lecture.

A second experiment was performed to get some data on mind-wandering during lectures. There was also an additional group (a restudy group) in which students were interrupted with questions and answers about each segment and tested only after the fourth segment.

His results, published in Proceedings of National Academy of Sciences, show that the tested group did better than both the non-tested and restudy groups. The interspersed tests on the subject matter of the lecture helped reduce mind-wandering by half, tripled note-taking, and improved retention of lecture materials. The tested group was also less anxious about the cumulative test at the end (possibly because they got accustomed to the idea of being quizzed).

Online learning is booming, and the sheer number of students involved makes it the responsibility of those in education to ensure that these courses are not a waste of time. While distractions may be a click away, student activity in these courses is also easier to monitor. This makes MOOCs an ideal place to experiment and improve educational techniques.

And Spzunar's experiments are continuing. The results of his study were obtained in tightly controlled settings and involved only 80 students. The hypothesis needs to be tested in an actual MOOC, and that's what Spzunar is now working on.

24 Reader Comments

From the article: "In the new study, he did something similar. In the first experiment, he divided a 21-minute lecture into four segments."

Never having taken single MOOC personally, perhaps someone who has can say how many of them offer only 21 minute lectures?

That aside, this looks an awful lot like stating the obvious: any teacher knows that you have to keep re-engaging students attention, no matter how large (or small) the class, or whether its on-line or in real life. It's just more obvious when you're physically in the classroom...

Colour me surprised. I've been through (UK) teacher training, and regular assessment of your pupils - both for their benefit and for yours, so you know how well you're doing - is very heavily emphasised.

Personally, I'm a little embarrassed for tertiary education that they're only just working this out.

This makes perfect sense. If only they could have done this when I was in high school. It would be difficult to apply this method to every student simultaneously in person, which is why the teacher tends to only pick on single students that appear to be drifting. It would benefit the class as a whole if public schools could put this into practice for whole classrooms.

The problem I have with this idea is that it ignores that some people learn better on their own instead of in a classroom setting being talked to as a group.

In my college time, I found it was often more productive for me to skip the lecture and read the textbook on my own time. That way, I could proceed at my own speed, visit points that I'm not yet clear on, and move on with points that I got relatively quickly. Since most lectures are just reproductions from the textbook anyway, I find the whole concept of lecture to be kind of outdated.

It's interesting that online courses have been shown to be more effective than in-class courses, despite the easier access to distractions... or so they assume.

Personally, I find that in 'online courses' distractions are less of an issue, as I can pause and rewind the video, and rewatch as many times as necessary. On the other hand, any distraction in an in-person course means I've missed something, and there's not really much I can do about it beyond interrupting the entire class and asking to go repeat it (which would become prohibitively expensive if every student did that every time they missed something).

In-person courses tend to focus on taking notes (i.e. cutting out information and then dumping the condensed version somewhere, then trying to learn what it means later), at the expense of actually thinking about what was said right away, and focusing on each point. Quite frankly, this is a horrible way to do things, and far inferior to having a video where you can think about every point and take as many/few notes as necessary for every point (and you won't have issues with accidentally cutting out important information).

I think the article at the start says the most important thing:

Quote:

Many teachers are using a "flipped" classroom model, where students take lectures through videos at their convenience and spend the time in class delving into issues they don't understand.

This, IMO, and IME, is the best way to do things. The only advantage lectures have over videos is the ability to ask questions, but that could easily be added in to online courses with similar results (have a set time of the day where you're encouraged to watch the lecture, while the teacher will be available for a live chat).

Beyond that, I see no advantages for in-person lectures, while huge disadvantages. On the other hand, being able to work through actual examples in class together is quite useful (replacing it with online would require more creativity in the solution, as compared to a video).

In my college time, I found it was often more productive for me to skip the lecture and read the textbook on my own time. That way, I could proceed at my own speed, visit points that I'm not yet clear on, and move on with points that I got relatively quickly.

You, my well-endowed friend, as well as most of the rest of us who read Ars, are in the minority of college students: you have Good Study Habits. You actually read the textbook, and could manage what you learned, and would go back and revisit what you didn't get the first time. Most students do not: if they read the book, it's the night before the exam, unless they are forced by assignments to do so. So basing teaching around your (and my!) learning experience might not help 90% of the class do any better.

I do, however, welcome these kinds of empirical studies that get at how to keep students on task so they learn something despite themselves.

Colour me surprised. I've been through (UK) teacher training, and regular assessment of your pupils - both for their benefit and for yours, so you know how well you're doing - is very heavily emphasised.

Personally, I'm a little embarrassed for tertiary education that they're only just working this out.

People with a professional interest in cognitive learning and dedication to their jobs already knew all of this for a very long time. Unfortunately, this likely covers fewer than 1 in 10 educators in the U.S. I always think of this West Wing / Sam Seaborn 6-figure salary for teachers remark. Then I realize, you know what, even if we started paying more educators that well, the number who would perform to match such pay scale would be pitiful (look at any university and consider the number of 6-figure tenured profs who educate poorly).

I have one physics professor who has grants for cognitive learning when he isn't doing space weather. He hands out a stapled bundle of cards numbered 1-6 at the beginning of each semester. Throughout the lesson he puts questions on the projector and has the class "vote" their response with the cards. Very effective and engaging educator.

In my college time, I found it was often more productive for me to skip the lecture and read the textbook on my own time. That way, I could proceed at my own speed, visit points that I'm not yet clear on, and move on with points that I got relatively quickly.

For the best students, a pace that is geared to the average student is going to aggravating and counter-productive. Some students will find that they don't need the lecture at all to do OK.

Then I realize, you know what, even if we started paying more educators that well, the number who would perform to match such pay scale would be pitiful (look at any university and consider the number of 6-figure tenured profs who educate poorly).

University Professors are typically academics/researchers first and educators second. Tenure can be achieved with average teaching reviews if the research is productive. Smaller colleges and community colleges tend to have better teachers because that is the primary job.Finally, starting salaries for professors are rarely in the 6 figures.

Then I realize, you know what, even if we started paying more educators that well, the number who would perform to match such pay scale would be pitiful (look at any university and consider the number of 6-figure tenured profs who educate poorly).

University Professors are typically academics/researchers first and educators second. Tenure can be achieved with average teaching reviews if the research is productive. Smaller colleges and community colleges tend to have better teachers because that is the primary job.Finally, starting salaries for professors are rarely in the 6 figures.

Plus, at the small/community college level, you're more likely to have adjunct (non-tenure stream) faculty teaching you. They aren't paid the same way, and earn a lot less. See http://adjunct.chronicle.com/

Never having taken single MOOC personally, perhaps someone who has can say how many of them offer only 21 minute lectures?

All of the Coursera courses I've taken have had at least a couple of hours of videos each week. They are, however, divided up into little five or ten minute chunks (annoying.)

Why? Is it because they never really get to the point in those chunks? What subject area was this in? I don't think I'd want to go over 15 minutes without something other than a talking head on screen.

The classic lecture is outmoded. The type of prof who just reads the notes aloud has been an anachronism since the printing press made textbooks cheap enough everyone could have a copy and just read it themselves.

I had a great professor who often interacted with the class by picking a victim (first those wearing a hat in class, then those sitting in the back, then bright colors, but by the end of the semester just those who look bored as everyone learned not to wear hats, bright colors, or sit in the back) asking a quick question, then picking another student and asking 'is he/she right?'. If they agreed but were both wrong, he would pick a third to answer (this once went on for 7 students until finally one replied 'no'. when asked to justify why, she said 'i have no idea, but you've asked 6 students who all agreed and are still asking more'). Fun times. Anyway, it was a good way to get students to pay attention and got people having the think not just about the questions he asked, but the answers others gave.

I like the way the youtube series CrashCourse does their videos - 15 minute chunks where they go at a pretty good pace, faster than advisable for a traditional lecture, but good for a video and keeps it dynamic enough to be engaging. During the end credits they provide an outline of the segment with each bullet point being a link back to the appropriate timestamp. If you missed anything or just want to review that bit, you just click and relisten.

I've learned more about chemistry and biology from their videos than I ever did in high school or university.

Monotony is the enemy of learning. Which means, when you find the perfect way to teach, it will not work for long. You have to vary your teaching methods. But, having said that, interaction is always necessary. The variation you need is changing the amount and direction of interaction between the students and teacher. To be honest, some of the best teaching I have ever done, is when I shut my mouth and let a couple of my students teach me something I already know (or something new, that is even more fun) in front of some other students.One last comment. Different students have different needs. You can't be all things to all people, but you can get out of their way and let them fulfill their needs when they are doing so. That is still good teaching, especially if you stand by to help catch them if they need it.

The results of this research may appear obvious, but that which seems obvious should still be empirically tested. This study seems to have two hypotheses: 1) do interspersed quizzes increase attention/focus and academically-relevant behaviors, and 2) do these increases yield greater performance. Research on online courses is actually more or less equivocal at the moment with other factors predicting the outcomes better than just "online vs. live courses".

With respect to online courses, I find this study interesting. Setting up an effective self-directed learning course is challenging. Studies that have found greater performance with online courses don't always use truly equivalent course structures, which is clearly a confound to the research. Some online courses are asynchronous, others are synchronous (and thus more analogous to live courses). Asynchronous may be largely self-directed (student is responsible for reading material, completing activities, and then takes a test), while others rely on frequent, direct feedback from an instructor. These differences may lead to very different outcomes for students from varied backgrounds (e.g., traditional early undergrad, non-traditional student, second career adult...). If interspersing quizzes during activities (video watching, reading notes...) could improve performance, that's fantastic. Though it's necessary to control for the practice effect of repeated quizzes with corrective feedback as well... Anyway, I think there's still plenty of work to be done to develop better online courses.

I had a great professor who often interacted with the class by picking a victim (first those wearing a hat in class, then those sitting in the back, then bright colors, but by the end of the semester just those who look bored as everyone learned not to wear hats, bright colors, or sit in the back) asking a quick question, then picking another student and asking 'is he/she right?'. If they agreed but were both wrong, he would pick a third to answer (this once went on for 7 students until finally one replied 'no'. when asked to justify why, she said 'i have no idea, but you've asked 6 students who all agreed and are still asking more'). Fun times. Anyway, it was a good way to get students to pay attention and got people having the think not just about the questions he asked, but the answers others gave..

With all due respect, that is a very dangerous method of teaching. Very few people like to be told they are wrong in front of a large group of peers. A large body of psychological research shows that most people will attempt to protect their ego, either by not listening to teacher, avoiding having to give an answer, or in some cases avoiding the classroom all together. I would prefer to have my students learn because they actually like my class, not because they are afraid that I will try to embarrass them.

People with a professional interest in cognitive learning and dedication to their jobs already knew all of this for a very long time. Unfortunately, this likely covers fewer than 1 in 10 educators in the U.S. I always think of this West Wing / Sam Seaborn 6-figure salary for teachers remark. Then I realize, you know what, even if we started paying more educators that well, the number who would perform to match such pay scale would be pitiful (look at any university and consider the number of 6-figure tenured profs who educate poorly).

Your comment about pay and ability to teach are silly. I doubt many students today have any idea if faculty are good teachers. I listen to students talk about faculty members and think they can't teach. I sometimes know these faculty members and I am disgusted at these students because I happen to know how much time and care they put into being great teachers. In college it is the students who are the problem in my experience - ill prepared for the class, lacking in skills, and with a crappy attitude of entitlement. You do not get a good grade for just showing up.

There can be mis-matches in the teaching style of a faculty member and the learning style of a student. Both matter and some are just incompatible to some degree. This doesn't make the faculty member a bad teacher and the student a bad student.

Finally, remember, faculty are not just paid to teach and most put in ridiculous hours.

I still remember my Electrodynamics course. The instructor was calculating the Electric field of a point charge in a rectangular conductor. We had already done spherical and cylindrical conductors so had the basics nailed. This however was also much simpler and it was easy to see the arithmetic aspects since it involved sines, and cosines instead of Legendre polynomials or Bessel functions.

Early on a thought occurred to me. The analysis would yield a solution with no singularities. Well OK there would be a singularity, but it would be different from the singularity that should appear in the solution. ( Something that seemed clearer to look at since we were only looking at sines and cosines. )

I spent the rest of the time resolving my problem, and seeing how the singularities I was seeing should transmute into the singularities I would see in the solution.

The end result is that I think I came out with a better grasp of the problem then a lot of the other students.

Moral of the story is that it is OK for the students mind to wander. ( So long as they don't wander too far. )

The points of education are that we want the students to think, and we want the students to think about the stuff they are being taught. Thinking means your mind will wander. To make sure that the student pays attention all the time is to teach the student to think the same way the teacher thinks. We have another word for that: indoctrination.

Colour me surprised. I've been through (UK) teacher training, and regular assessment of your pupils - both for their benefit and for yours, so you know how well you're doing - is very heavily emphasised.

Personally, I'm a little embarrassed for tertiary education that they're only just working this out.

People with a professional interest in cognitive learning and dedication to their jobs already knew all of this for a very long time. Unfortunately, this likely covers fewer than 1 in 10 educators in the U.S. I always think of this West Wing / Sam Seaborn 6-figure salary for teachers remark. Then I realize, you know what, even if we started paying more educators that well, the number who would perform to match such pay scale would be pitiful (look at any university and consider the number of 6-figure tenured profs who educate poorly).

I have one physics professor who has grants for cognitive learning when he isn't doing space weather. He hands out a stapled bundle of cards numbered 1-6 at the beginning of each semester. Throughout the lesson he puts questions on the projector and has the class "vote" their response with the cards. Very effective and engaging educator.

It would be a nice social experiment to try to build a small, isolated island-society of only talented and motivated individuals- to see what they would do when freed from the constraints that the incompetent people would otherwise fetter them with. It almost strikes me as Utopian, with the small exception that if you have kids and they aren't motivated/talented, you have to set them adrift in the Pacific...

I've now taken 4 MOOCs on Coursera. One of them did NOT involve interspersed questions throughout the lectures; this really bothered me, as it didn't let me know whether or not I was getting a full grasp of the content at hand. Truly, being at my home computer it was harder to focus (I have dual monitors and people were talking to me) and so even while taking notes, I didn't really get a grasp of it in my head.

But in the classes with interspersed questions, it was almost built into the lectures; not only did it seem as if the teacher was individually checking on all of the students to see how well they were doing, but it let me know that up ahead, there was a chance for me to check that I was really paying attention. Let me elaborate on that first part: In an ordinary class of say, 15-30 students, teachers often look around and make eye contact; if students appear to be confused, they will pause and try to make sure everyone is keeping up. Or at the very least, they'll do this after every major concept, and ask "does anyone have any questions?" - this is the part that you don't even get in LARGE college classes; this could result in 50 hands being raised! There's no time for that.

But in a MOOC video, that pause comes, and if you get the question wrong, an explanation is given. Not only does this check if I was paying attention (I know I got the question wrong) but if I did, I'm given the correct answer, not simply left behind.

The problem I have with this idea is that it ignores that some people learn better on their own instead of in a classroom setting being talked to as a group.

In my college time, I found it was often more productive for me to skip the lecture and read the textbook on my own time. That way, I could proceed at my own speed, visit points that I'm not yet clear on, and move on with points that I got relatively quickly. Since most lectures are just reproductions from the textbook anyway, I find the whole concept of lecture to be kind of outdated.

But the online method is encouraging your standard practice. The difference is merely that it has pop quizzes in the 'textbook'. (This screams they are using flash and I shudder; teachers and flash should never meet --- irrespective of the display's context.)

It is as if you had walked into a tute and had a pop quiz after spending the morning studying the text book. Slow and/or stupid students (me) can watch the video three or four times while avoiding the monotonous text.* Just have to do pop quizzes. Smart students can study the text and fast forward the video till the quiz which they pass with flying colours. Not once do you 'have' to go to a particular location to tentatively observe a monologue, that may or may not give relevant info concerning the module, interspersed with inane questions by students that can't grasp basic English.

* I'm currently studying a course where the text is less boring than the talking head video. I have no idea why they thought this bloke would be good on tv. And I've yet to refer back to a video for relevant info.

People with a professional interest in cognitive learning and dedication to their jobs already knew all of this for a very long time. Unfortunately, this likely covers fewer than 1 in 10 educators in the U.S. I always think of this West Wing / Sam Seaborn 6-figure salary for teachers remark. Then I realize, you know what, even if we started paying more educators that well, the number who would perform to match such pay scale would be pitiful (look at any university and consider the number of 6-figure tenured profs who educate poorly).

Your comment about pay and ability to teach are silly. I doubt many students today have any idea if faculty are good teachers. I listen to students talk about faculty members and think they can't teach. I sometimes know these faculty members and I am disgusted at these students because I happen to know how much time and care they put into being great teachers. In college it is the students who are the problem in my experience - ill prepared for the class, lacking in skills, and with a crappy attitude of entitlement. You do not get a good grade for just showing up.

There can be mis-matches in the teaching style of a faculty member and the learning style of a student. Both matter and some are just incompatible to some degree. This doesn't make the faculty member a bad teacher and the student a bad student.

Finally, remember, faculty are not just paid to teach and most put in ridiculous hours.

His comment didn't seem silly to me. It was just another perspective, same as yours.

As you said, the solution isn't a matter of fixing teachers or students. We need to fix the process. Paying more doesn't do that. Working harder doesn't do that. So, except for a few bold innovators, everybody continues playing along with the old song, knowing things aren't quite right.

This is exactly what has been done already on udacity, which works great for computer science and learning to code. I'm currently taking some classes there, and the way it works is that one unit is divided into around 30~35 small videos, of 5~10min each or less, and at the end of each video there is always a question and/or a programming exercise. I would say it's highly effective to be quizzed every couple minutes, and keeps you engaged.

In my college time, I found it was often more productive for me to skip the lecture and read the textbook on my own time. That way, I could proceed at my own speed, visit points that I'm not yet clear on, and move on with points that I got relatively quickly.

You, my well-endowed friend, as well as most of the rest of us who read Ars, are in the minority of college students: you have Good Study Habits. You actually read the textbook, and could manage what you learned, and would go back and revisit what you didn't get the first time. Most students do not: if they read the book, it's the night before the exam, unless they are forced by assignments to do so. So basing teaching around your (and my!) learning experience might not help 90% of the class do any better.

I do, however, welcome these kinds of empirical studies that get at how to keep students on task so they learn something despite themselves.

I'm the same way but with one key difference: Need of visual representation.

My hearing retention is god awful, but having the text in front of my face negates this effect and also has the advantages that you listed above.