Support/Q2HelpfulnessTests

This quarter we gave ourselves the goal of increasing the average helpfulness score of knowledge base articles by 2%. There are two basic ways to approach that – make articles more helpful (obviously) and get more people to an article that helps them on the first shot (thereby decreasing the likelihood of rating an article unhelpful).

Our plan is to run multivariate tests on the support home page (getting people to the right article) and on 2-3 of the top support articles (making things more helpful) is to run A/B tests on the start page and one article that incorporate many changes. Once we get results from these tests we may create new tests to refine our findings.

Article tests

Changing KB Article language

The current KB articles do a good job of speaking directly to the user and answering their questions. We'd like to test what impact changes to the language of the article have. For this test we'll work on rewriting each article, making these changes:

Remove unnecessary information - There are almost always lots of tangential pieces of information related to a topic that could be presented. We should look to remove or not include anything that isn't necessary to solve the problem and get the user on their way.

Use non-technical terms where possible - As SUMO contributors we're familiar with many technical terms and abbreviations. We shouldn't assume that people using the KB are familiar with any of these terms. We should explain things in as plain and simple language as possible.

Use a conversational or friendly style - The articles should read more like your friend is explaining something to you and less like you are sitting in a lecture.

Here's the opening paragraph of the Profiles article:

Firefox saves your personal information such as bookmarks, passwords, and user preferences in a set of files called your profile, which is stored in a separate location from the Firefox program files.

Note: This opening paragraph doesn't include any information about what a user should expect to find in this article. That info is included in the rewrite below.

Here's an example of the kind of rewrite we propose to do:

Did you know that all of the settings you make in Firefox, like your home page, what toolbars you use, saved passwords and bookmarks are all stored in a special profile folder? It's true. This article will tell you why this is a good thing, where you can find your profile folder, and give you links to related articles.

San-serif vs serif type

We'd like to test whether changing the body text of articles to a sans-serif font makes them easier to read.

Embedding screencasts inline

Screencasts can be very helpful. We'd like to test if embedding them inline will encourage more people to watch them when available. None of the articles we're considering for the tests currently have screencasts so they'll need some created before the test can be run.

Changing KB Article layout

Currently, KB articles have an "Actions" box at the top giving people the option to edit or translate an article. While this is nice option that helps to reinforce that the KB is a community effort, it may be a distraction for someone trying to solve a problem. Instead we'd like to test whether swapping the related articles and search sections with the actions box will help people get to the correct article faster.

Start page tests

Improving performance

Improving page load times should make using and navigating the site easier and therefore help prevent people from becoming frustrated while searching for a solution. After looking at the size and loading times for various elements of the home page, it appears that removing the large background image (54k) at the top of the page will make the biggest impact. The screenshot examples below are very subtle because there is actually a tiled image (4k) behind the large background image.

Removing the background image changes the character of the SUMO home page quite a bit. Another option might be to use a more compressed version of the image. Here's an example of the image compressed to about 18k.

Start page:

Articles:

Simplifying the header and footer

Another test we'd like to run is simplifying the page header and footer. This would get rid of a lot of unnecessary information (the links to the rest of Mozilla aren't crucial since the site is about getting help) on each page, making it easier to find what's important.

Browsing Categories

Another idea that might help get people to the correct article faster would be to offer a better browsing option. Currently the home page is designed around searching with an additional list of popular articles for users who prefer to browse. One idea we'd like to test is to replace that list of popular articles with a few category sections. They could have headings like Crashes, Troubleshooting, or Tutorials with a short description of what other things are included. These would then link to a category page. We might also test whether categories work better above search (mockup on right).

Proposed/Possible Categories

Bookmarks

How to set the home page

Bookmarks

Awesome bar

Using Firefox

Bookmarks

Tabs

Awesome bar

Private Browsing

Privacy

Cookies

Cache

Private Browsing

Clear Recent History

Customizing Firefox

Customizing Fx with add-ons

Unable to install add-ons

Add-ons Blocklist

Personas

Personas do not work with Themes

Troubleshooting

Basic troubleshooting

Firefox crashes

Pop-up blocker

Website doesn't load

Problems with Facebook

Edit options to add Adobe to the list of allowed sites

Article titles as questions

People often form their search queries as questions. We'd like to test whether listing articles titles as questions (that correlate with the questions that people search for) gets people to the right article faster. We would test this using the current start page layout, changing the wording of the list of popular support articles.

Another variation of this would be to add article titles as questions to the category pages that will need to be created for the Browsing Categories test.