How we learn from users, what we learn from users, and how we use what we learn

Posts tagged "ratings"

In November 2010, a ratings badge was added to most of our online help and learning content. Visitors are asked on each page, “Was this helpful?”:

We actually use these page ratings quite extensively – we are continually monitoring changes in the scores and trying to improve our content.
We have found that the vast majority of visitors don’t leave a rating. This is sometimes troublesome on our less trafficked documents since this leaves us little guidance for what we can improve.

Matt Horn, a Senior Content and Community Lead, decided to do some investigating into the reasons why people might not leave a rating. He wrote a blog post over on the Flex Doc Team blog asking people to comment on when they do and do not rate our content. Although it is difficult to draw general conclusions based on such a small sample, the results he obtained are still pretty interesting. Here is a summary of his results, provided by Matt:

Why they don’t rate

The biggest reason folks don’t rate is because they’re too busy. 6 people said they either ignore the widget or are too busy to click on it. They just want the help content and ignore everything else: “Don’t even see it. It’s like when you go to a website and have to click away the “research” popups.” They may notice it, but just want to move on: “When I solve the problem, I am relieved and just want to get back to work.”

A couple of people didn’t even realize there was a ratings widget. 2 people mentioned that it loads slower than the rest of the page so they are usually already off and scrolling by the time it finishes loading. My tests bear this out: it loads after the content.

2 people felt the question was imprecise but didn’t elaborate on what question we should ask that would be better. “I perceive it as a question about overall experience and every time I find the information I feel frustrated a bit, so that I don’t feel like pressing “Yes” because I didn’t like the way I reached the info, and pressing “No” is also not an option, because info was indeed helpful.” Similarly, another person said he might go to a page but doesn’t rate it because he often searches for and finds the wrong information. Perhaps rewording to something like “Was this page helpful?” would be a small step.

1 person said maybe people only rate when the pages are really bad or really good. “Remember a non-answer can still be an answer. I think only the extreme ends want to be vocal.”

Some misperceptions persist about the ratings widget. 1 person said he doesn’t want to log in to rate anything, and 2 people said they don’t rate because they feel the comments are ignored. In both cases, they were confusing the way the commenting system at the bottom of the page either works now or used to work.

Specific suggestions

Some users suggested specific ways to improve the number of ratings.

1 person suggested having the widget stick with the user on the side of the page as they navigate. He specifically mentioned the Oracle feedback widget. “Its not annoying but it is also hard to ignore.”

1 guy just doesn’t like radio buttons, but didn’t appear to dislike the idea of rating pages: “Do something better and I’ll rate more.”

Don’t bother

3 people said they wouldn’t rate the pages regardless of how or where we put the widget. Instead, they suggested we collect analytics in other ways.

1 person suggested that we track the number of times users copy code blocks by adding a “Copy” button that we could track.

Another person said he would be fine if Adobe contacted him and asked him specific questions about help page usage.

1 person mentioned that it would be interesting to generate reports of help usage: “Maybe you could find a way to track user’s usage and then present it back to them as a report which they could comment on”. This seems a little big brotherish to me.

Off topic

As usual, users took the opportunity to make a few points that were not exactly related to the issue of collecting ratings:

2 people want Eclipse help.

2 people want more sample code.

2 people wanted links to the Adobe forums from the help pages.

1 person wanted the ability to rate comments like the StackOverflow ask/answer system.

Actions

There are some steps we could take that might improve the ratings, although most of these might not move the needle much:

Reword the question to say “Was this page helpful?” from “Was this helpful?”

Add “No login required” to assuage users of the need to log in to rate.

Load widget earlier in the process.

Change to a 5-star rating system rather than a YES/NO question.

Have ratings widget move with the user while they scroll the page.

How about adding the current rating to the widget? Something like:

“Was this page helpful? YES/NO (45% of users found this page helpful)”