Getting Help when Using Firefox – Part I

A few months ago, David Tenser from our Support team came to the Metrics team with some ideas for analysis concerning user experience with our support.mozilla.com site (SUMO). At the time, we did not have the functionality set up to get the data we wanted, but thanks to some hard work from Jeremy Orem and multiple members of our WebDev team, all of this was made possible about three weeks ago! Now we can see a user’s search term, which article is clicked on after searching, and whether or not the user thought the article was helpful:

The first bit of data we wanted to extract was our Bounce and Refine Search rates for our top search terms in SUMO. For this, I defined a bounce as a SUMO visitor searching a particular term and then immediately leaving the site. Bounces are viewed as failures to give the user results they were looking for. A refined search is defined as a user that searches one term and without clicking an article link or navigating anywhere else searches for a refined version of the same term. Again, we look at a refined search as evidence of a user not receiving the appropriate information with their original search. Here’s what these numbers look like for our top search keywords:

Highlighted in red are the particularly problematic search terms for SUMO in terms of bounce and refine search rates. Private browsing is the most alarming — not only is it our third most popular search term, but it also is in the highest bounce rate and refine search rate groups. From a user standpoint, searching private browsing is most likely an attempt to find out how to turn on private browsing. We can see this in the data; the top refined search for this phrase is start private browsing. Looking at the search results a visitor sees, the only one relevant to private browsing is the top result:

This article does explain how to start private browsing, but the description gives off more of a “what is private browsing” vibe. Either a “Start Private Browsing” article should be started or the phrase should be included in this article’s description.

The bounce rate of home page is a little more perplexing. It seems the main motivation behind this search is to change the default home page in Firefox. Fortunately, the top search result is:

So no issues right? Looking at the refined searches reveals a surprising anomaly: the top two refined searches are “set home page” and “change home page”. I’m baffled. Could users just be skipping past this first result without realizing it? Not likely, but possible. We need to explore other reasons for the high bounce rate — maybe users are searching home page for other reasons.

Next up is history. Users searching this term are (on assumption) likely to be looking to clear their history. Unfortunately, for Firefox 3.0 users there is not a clear result to navigate to. “Clearing Location bar History”, “How to clear Search bar History”, and “Clearing Private Data” are the appropriate articles to continue with, but for inexperienced users who might not know what these terms mean, there is not a definitive how-to-clear-your-history result.

For the mass migration from the downloads search results page (44% bounce rate) I think an article on where to find downloaded files would work well. Right now, there is no result pointing users in this direction, which is probably contributing to such a high bounce rate.

Import bookmarks is up last. Looking at the refined searches, import bookmarks from internet explorer tops the list. Comparing this with the search results, this article is not listed on the front page. I believe moving it up in the line of results could quell the refine search rate.

Each of these changes represents a small change to our database of support articles that could taken together greatly increase the effectiveness of search in the Knowledge Base. In part II of the SUMO analysis, I’ll be taking a look at pathing after searches — the articles visitors are choosing to view after a specific search and their satisfaction with each one.

7 responses

I wonder if it would help to have certain vague searches (eg “cookies”, “private browsing”, etc) go directly to a specialized page which offers link and a brief description of common tasks or observed refinements (eg “how to turn on private browsing”, “what is private browsing”, etc).

That would help if people are just feeling lost when searching for stuff because they get a page full of not-so-relevant results and they’re not sure how to refine the search.

Eric: I just want to say what a pleasure it has been to work with you and Ken on this. The insights gained by this analysis has been invaluable for us and we are already working hard on implementing changes as a result of our new findings.

Another thing it would be good to fix is the fact that the first bit of text for all those search results is stuff about which version of Firefox it applies to, rather than actual article content. The summary extractor needs to be smarter

Maybe it is just me but judging by the results shown here it must not be. Anytime I have search for anthing on SUMO, the results I am presented with were always forum posts. I did not want to read through a hundred links of forums posts in hope of fiding the one that was asking the exact question I had and then hoping it had an answer. KB articles should always be shown first!

A question about the data gathered. Since some of these terms are our most common problems when performing support ourselves, were there steps taken to exclude contributors for the info?

Here’s what I’m thinking. If I’m searching for an article myself to help someone, I would almost always end up as a “bounce.” I would see the article I need in the list, then right click on it to copy the link, rather than clicking through and letting the page load.