Videos are a widely-used kind of resource for online learning. This paper presents an empirical study of how video production decisions affect student engagement in online educational videos. To our knowledge, ours is the largest-scale study of video engagement to date, using data from 6.9 million video watching sessions across four courses on the edX MOOC platform. We measure engagement by how long students are watching each video, and whether they attempt to answer post-video assessment problems.

Our main findings are that shorter videos are much more engaging, that informal talking-head videos are more engaging, that Khan-style tablet drawings are more engaging, that even high-quality pre-recorded classroom lectures might not make for engaging online videos, and that students engage differently with lecture and tutorial videos.

Based upon these quantitative findings and qualitative insights from interviews with edX staff, we developed a set of recommendations to help instructors and video producers take better advantage of the online video format.

The above figure shows four main kinds of videos on the edX platform:
a.) a recorded classroom lecture, b.) an instructor's talking head, c.)
a Khan-style digital tablet sketch drawing (popularized by Khan Academy), and
d.) a PowerPoint slideshow.

Since we're now in the early days of the widespread global popularity of
video-based MOOCs, we were curious about the following question: How
does MOOC video production style affect student engagement?

Why care about engagement? Because it's often a prerequisite for
learning. Although it's very hard to assess actual learning from MOOC
server logs without designing an experiment to measure learning
outcomes, engagement is something that can reasonably be measured at
scale on existing MOOC data.

For this study, we measured engagement by how long students watched each
video and also by whether they attempted to answer post-video assessment
problems. (Note: We can't measure true learner engagement at scale,
since we don't know how much learners were actually paying attention to
these videos while watching, but these metrics are a reasonable first
proxy.)

We took all 862 videos from four edX courses offered in Fall 2012
(Intro. CS, Statistics, Artificial Intelligence, and Solid State
Chemistry) and hand-classified each one based on its type (e.g.,
traditional lecture, problem-solving tutorial) and production style
(e.g., PowerPoint slides, Khan-style tablet sketch drawing, talking
head). We automatically extracted other features such as length and
speaking rate (words per minute). We then mined the edX server logs to
obtain over 6.9 million video watching sessions from almost 128,000
students. To our knowledge, this is the largest-scale study of video
engagement to date.

Findings and Recommendations

Here are our seven main findings and corresponding recommendations for
creators of online educational videos:

1. Shorter videos are much more engaging. Engagement drops sharply
after 6 minutes.

Recommendation: Invest heavily in pre-production lesson planning to
segment videos into chunks shorter than 6 minutes. This is the most
significant recommendation!

2. Videos that intersperse an instructor's talking head with
PowerPoint slides are more engaging than showing only slides.

Recommendation: Invest in post-production editing to display the
instructor's head at opportune times in the video. But don't go
overboard because sudden transitions can be jarring. Picture-in-picture
might also work well.

3. Videos produced with a more personal feel could be more engaging
than high-fidelity studio recordings.

Recommendation: Try filming in an informal setting such as an office
to emulate a one-on-one office hours experience. It might not be
necessary to invest in big-budget studio productions.

Recommendation: Introduce motion and continuous visual flow into
tutorials, along with extemporaneous speaking so that students can
follow along with the instructor's thought process.

5. Even high-quality prerecorded classroom lectures are not as
engaging when chopped up into short segments for a MOOC.

Recommendation: Some MOOC videos are created by simply chopping up old
classroom lecture videos (often recorded in prior years) and putting
them online. Those videos are less engaging than ones recorded
specifically with the MOOC format in mind. But if instructors still
insist on recording traditional classroom lectures, they should
purposefully plan lectures with the MOOC format in mind and work closely
with instructional designers who have experience in online education.

6. Videos where instructors speak fairly fast and with high enthusiasm
are more engaging.

Recommendation: Coach instructors to bring out their enthusiasm and
reassure them that they do not need to purposely slow down. Students can
always pause the video if they want a break.

7. Students engage differently with lecture and tutorial videos.

Recommendation: For traditional lectures, focus more on the first-time
watching experience. For step-by-step tutorials, add more support for
rewatching and skimming, such as inserting subgoal
labels in
large fonts throughout the video.

Videos are a widely-used kind of resource for online learning. This paper presents an empirical study of how video production decisions affect student engagement in online educational videos. To our knowledge, ours is the largest-scale study of video engagement to date, using data from 6.9 million video watching sessions across four courses on the edX MOOC platform. We measure engagement by how long students are watching each video, and whether they attempt to answer post-video assessment problems.

Our main findings are that shorter videos are much more engaging, that informal talking-head videos are more engaging, that Khan-style tablet drawings are more engaging, that even high-quality pre-recorded classroom lectures might not make for engaging online videos, and that students engage differently with lecture and tutorial videos.

Based upon these quantitative findings and qualitative insights from interviews with edX staff, we developed a set of recommendations to help instructors and video producers take better advantage of the online video format.