The Mozilla support project

SUMO now helping an additional 7.3 million visitors

About 3 weeks ago we made the switch to a new information architecture and new design. The goal was to improve the browsability of the site and help people find the articles that they were looking for. 3 Weeks later we can now take a look at our key performance indicators to determine whether the whole project was worth the effort

Methodology

Since this project was primarily concerned with the Knowledge Base, we can focus on the helpfulness rating in this channel. Also, we know from our exit surveys that about 80% of our visitors use the KB. The KB helpfulness rating is based on the survey that accompanies each article in each language. We ask the question “Was this article helpful?”, which can be answered with yes or no. Of course this metric is not perfect, articles that describe features have higher ratings than articles explaining how to fix a problem, English articles are generally higher rated than localized articles, despite having the same content, and the rating is also influenced by the path people took to get to the page. However, in this case we are not interested in the absolute ratings, we are particularly interested in the change since we moved to the new iA and design.

So, what happened?

We knew from previous tests that making the site browsable would be beneficial for that segment of our users who would rather browse than search for their article. People rate an article down, when it’s not the one that they were looking for. We know so much from our article surveys, and assuming we did our homework we should help more people find the right article. That being the case we expected the helpfulness of articles to rise, but it was hard to tell by how much it would rise. Considering that we have over 500,000 visitors per day and 80% use the KB, even a change by one percentage point would help an additional 1.46 million visitors per year. Without further ado, here are the results:

The results are phenomenal, we raised the helpfulness by 10 percent on average. That’s an increase by 5 percentage points and means an additional 7.3 million visitors per year stating that they found a SUMO article helpful. This is across all languages and across all incoming channels. It means that in 7.3 million cases where people might have decided to drop Firefox or be miserable because they couldn’t get Firefox to do what they wanted, they will now leave SUMO satisfied with their browser.

It’s hard to overstate the significance of this, and we are extremely happy with the results. The improvements to the site were the result of month of hard work by many people on the SUMO team, from SUMOdev, our creative team, and UX designers. We knew we were able to offer our users a better service, and the work has finally paid off. Continually thinking about how to serve our users better is what’s driving this team, and we will take these results as motivation to work even harder on improving our services.

Today, I’m very proud of what this team made possible, and I’d like to extend my thanks to each and everyone involved in the process: You made these results possible!