Although the study is now a couple of years old, Paul’s tweet gave the study a lot more industry-related attention. In this post, I thought it would be interesting to take a closer look at the study’s findings with some details on the methods used. I’ll also let you know what progress has been made since the study was conducted and how this can help beginners learning HTML and CSS.

The Study’s Participants and Methods

We all know typos and syntax misunderstandings are a part of the learning process for all of us, so it’s always interesting to see what kinds of mistakes different developers are making.

This study was not an extensive one, but was done similar to how a company might do usability testing. In this case, 20 participants were video recorded as they completed assigned coding tasks (more on the tasks in a moment) using what they call a “think-aloud protocol”, which encourages participants to vocalize their thought processes while completing tasks.

Although the study was limited to 20 participants, the individuals had differing levels of experience in HTML and CSS. The study also claims to “not exclude any background or profession”, so we can appreciate that they tried to ensure that a very diverse group took part.

The 20 participants were 7 females and 13 males, ranging in age from 18 to 47. It’s also noteworthy that only 2 participants indicated that web design is their profession.

The Purpose of the Study

As noted, most of the participants were not professional developers. But why focus the study on people with little or no experience with HTML and CSS? Participants used the first iteration of openHTML Editor, a text editor that’s part of openHTML, a “collaborative research project aimed at designing better tools and practices for learning web development.”

Further, here’s a quote from the study’s introductory sections:

Beginners who learn HTML and CSS encounter many opportunities to learn from the process of authoring code for a computer to interpret, making mistakes along the way, and recovering from those mistakes.

And further:

In this study, we seek to identify the errors people make while writing code in HTML and CSS, and examine the cognitive origins of these errors. These findings are instrumental in our continuing efforts to design a web editor for beginners that is a pedagogically superior alternative to existing tools.

As stated, the purpose of the study was to see what are some of the most common errors made when first learning HTML and CSS. They explain that the intent was to use the findings to design a better web editor (text editor). The study’s findings have value for anyone involved in creating or maintaining a text editor, to decide what kinds of errors to look for to assist beginners to make progress more rapidly.

The study aimed to answer the following questions:

What errors do people make when constructing web pages in HTML and CSS?

What are the sources of these errors?

Once made, how well do people recover from such errors?

The editor the participants used was intentionally simple and unfamiliar to all participants. This way they could observe the mistakes made in direct relation to HTML and CSS syntax, rather than allowing participants to be sidetracked by editor-related complexities.

Tasks Performed in the Study

The 20 participants were permitted 30 minutes per task, using whatever resources they normally used (including web searches). The tasks were:

Create and horizontally center a container div, and add a right sidebar in the container.

As you can see, these are as simple as it gets when it comes to HTML/CSS. But this study wasn’t just about the final results that students produced. In most cases, the results were probably fine. It was the process of completing the tasks that was studied.

For example, hundreds of errors were tracked during the video recordings, but most of those were caught and corrected by the participants. Although these end up as corrected errors, they still constitute part of what was slowing down the learners. And so, by studying these common, temporary errors, they can figure out ways to help beginners learn faster and with fewer roadblocks.

How Errors Were Categorized

The errors made were placed under one of three types. And again, it should be emphasized here that these errors were not errors that were observed when observing tasks that had been completed; these are errors that were observed during the process of completing the tasks, whether the errors were corrected or not:

Skill-based Errors

These are unintended actions, true “typos” that could be caused by a number of factors, but are not caused by a lack of understanding. An example given was one participant who was told to add alt text to an image. The error looked something like this:

<img src="image.gif" alt=My Image"/>

Note the missing opening quotes. This was simply a typo, and was fixed shortly after being made.

Rule-based Errors

These errors are unintentional errors, but are caused by misunderstandings. They could be bad rules being applied, or misapplying a previously understood syntax rule. The example given for this category is an interesting one. The participant did the following in the HTML:

Knowledge-based Errors

Also called “breakdowns”, these errors occur at higher conceptual levels where unfamiliar situations are presented when a problem needs to be solved. They result from an incomplete understanding of the situation and often lead to experimentation and web searches (we’ve all been there!).

The example given in this case is a lengthy one, and kind of funny (hey, we’ve all been there too!). The task was to align text in a table cell to the right. The participant attempts numerous steps to get this right, including:

Copying and pasting deprecated code in an incorrect manner (<align=right>).

Moving the incorrect code into the table cell and between the <tr> and <td> tags.

Searching the web multiple times.

Copying and pasting HTML into his CSS.

Putting dummy text into the editor to see if the editor is working.

The participant eventually gave up after 6 minutes, moving on to something else.

Results of the Study

In the end, the number of errors under the three aforementioned categories broke down as follows:

70.9% of all errors were skill-based errors.

16.9% were rule-based errors.

12.1% were knowledge-based errors.

As mentioned, most of the errors were resolved during the task completion process, but some were not, and they broke down like this:

4.3% of all skill-based errors were unresolved.

39.6% of rule-based errors were unresolved.

52.1% of knowledge-based errors were unresolved.

The skill-based errors included, among others, the following:

Typographical (the majority)

Unclosed tags

Missing delimiter (e.g. semi-colon for CSS lines)

Mixing CSS syntax with HTML, or vice versa

The rule-based errors included:

Using obsolete elements

Invalid or non-existent elements, attributes, or properties

Using the wrong name (e.g. font-color instead of just color)

Errors in HTML lists (this was surprising, but many participants put things like <p> elements in place of <li> elements; others added numbers to ordered lists unnecessarily)

Incorrectly written inline CSS (which is a double-whammy of sorts)

Invisible elements (this is when an element doesn’t appear on the page because there’s no content, no height, etc., and the participant temporarily can’t figure out why)

What Progress Has Been Made as a Result?

Seeing as this is a two-year-old study, I contacted two of the study’s authors and asked if any progress had been made in building tools based on these or other findings. Thomas Park, one of the authors, had some encouraging progress to report:

We have a few design efforts that have come out of our study. Right now we’re working on a tool called Snowball for building modern, interactive articles that aims at reducing errors for beginners through the use of scoped (but accessible) HTML/CSS/JS components.

He also explained:

[W]e have ongoing collaboration with Mozilla Webmaker. We’ve sketched out an error feedback system designed around the cognitive source of HTML/CSS errors based on the 2013 study (e.g., whether a particular misconception or attentional/typographical slip is at the root). We intend to build it as a Brackets extension but don’t have much that is concrete yet.

Final Thoughts

The 2013 study wasn’t perfect, but it’s a good start and similar studies for HTML or CSS don’t seem to exist (at least I haven’t seen any). Additionally, Thomas Park pointed me to a follow-up study that I’ll try to cover in a future article.

As already mentioned, most errors get caught and corrected. But these slow down progress for learners especially when the correction isn’t conceptually sound but only gives the appearance of being corrected. Validation and code linters can help, but that’s usually a separate step. Even when coding errors are indicated in text editors via colors or error indicators, sometimes that’s not enough. I think text editors designed for beginners should give clear indications of errors along with an optional inline text explanation of what is wrong and how to fix it.

Although such a process might somewhat slow down the completion of tasks, I think with more detailed error explanations the larger concepts will be made clearer to beginners at an earlier stage, resulting in stronger developers when new skills are learned — because the skills will be learned correctly.

I think the following statement from the study’s concluding remarks nicely sums up my view on this:

When a beginner writes code that has many errors but still renders as desired, they receive positive feedback in the form of the properly formatted web page. These errors are latent, remain unresolved, and reinforce faulty understandings that can become difficult to overcome.

Many more details are made available in the study itself, including tables detailing the numbers of each kind of error made, so check it out if you want to delve more into the topic.

Louis has been involved in the web dev industry since 2000, and is a co-editor of the HTML/CSS content on SitePoint. He blogs at Impressive Webs and curates Web Tools Weekly, a newsletter for front-end developers primarily focused on tools.