Unnecessary obstacles limit learning in cognitively-complex domains such as computer programming. With a lack of appropriate feedback mechanisms, novice programmers can experience frustration and disengage from the learning experience. In large-scale educational settings, the struggles of learners are often invisible to the learning infrastructure and learners have limited ability to seek help. In this paper, we perform a large-scale collection of code snippets from an online learn-to-code platform, Python Tutor, and collect a frustration rating through a light-weight learner feedback mechanism. We then devise a technique that can automatically identify sources of frustration based on participants labeling their frustration levels. We found 3 factors that best predicted novice programmers' frustration state: syntax errors, using niche language features, and understanding code with high complexity. Additionally, we found evidence that we could predict sources of frustration. Based on these results, we believe an embedded feedback mechanism can lead to future intervention systems.

Wouldn't it be awesome if a programming instructor or course designer
could magically peek into the minds of learners to see exactly
when they're getting frustrated? That way, they can know what kinds of
code are confusing for learners, when to best intervene to help out, and
how they can improve instructional materials in the future.

How might we go about achieving this goal? We could attach electrodes to
everyone's heads and monitor their brain activity. We could use 3D
sensors like a Kinect to detect their body posture and facial
expressions. We could attach eye trackers to everyone's computer monitor
and use that data to estimate frustration. We could imagine doing all
sorts of monitoring, but the problem with all of these ideas is that
they're expensive and thus don't scale. How are we ever going to attach
these devices onto hundreds of students (or tens of thousands, in the
case of MOOCs)? Who will calibrate or maintain them? What if they get
lost or stolen?

Instead of relying on high-tech gizmos, we decided to take the most
low-tech possible approach: What if we just asked learners to tell us
when they're frustrated? This method seems so simple and naive; does
it have any chance of working? We created a prototype called HappyFace
to find out. HappyFace is a five-level pain scale embedded within the
bottom of my Python Tutor code editor (click
screenshot to enlarge):

Whenever the learner wants, they can click one of the five faces
(ranging from really happy to really sad), and the system will capture
their selection, along with a snapshot of the code they're currently
working on. The learner can also optionally write in why they feel happy
or sad at the moment.

The design of HappyFace was directly inspired by medical pain
scales where
patients self-report their pain levels to their healthcare providers.
Here's an example from the UCSD urgent care clinic that I recently had
the pleasure of visiting!

The basic idea behind pain scales is that they provide a simple and
intuitive way for people to report how they're feeling at the moment.
One of the best ways to scale something is to have it be super-simple
with the fewest possible moving parts. That's what HappyFace tries to
provide for programming-related materials.

To test whether this straightforward approach can actually identify known
causes of programmer frustration, we deployed it to the Python
Tutor website for 2.5 months and collected 2,385 self-reported clicks
from learners around the world. We analyzed the associated code and
found three major factors that correlated with higher frustration
levels: syntax errors, using idiosyncratic Python language features, and
trying to understand code with high complexity. These three factors
are consistent with what prior studies have found to be frustrating for
novice programmers, which indicates that our simple approach can collect
useful data at a much lower cost than, say, using fancy physical
sensors.

In sum, HappyFace provides a simple and scalable foundation on which to
build follow-up systems that allow instructors to provide real-time
targeted help and to refine their instructional materials using
fine-grained data about common learner frustrations.

Unnecessary obstacles limit learning in cognitively-complex domains such as computer programming. With a lack of appropriate feedback mechanisms, novice programmers can experience frustration and disengage from the learning experience. In large-scale educational settings, the struggles of learners are often invisible to the learning infrastructure and learners have limited ability to seek help. In this paper, we perform a large-scale collection of code snippets from an online learn-to-code platform, Python Tutor, and collect a frustration rating through a light-weight learner feedback mechanism. We then devise a technique that can automatically identify sources of frustration based on participants labeling their frustration levels. We found 3 factors that best predicted novice programmers' frustration state: syntax errors, using niche language features, and understanding code with high complexity. Additionally, we found evidence that we could predict sources of frustration. Based on these results, we believe an embedded feedback mechanism can lead to future intervention systems.