Back in my usability days, I talked often about measurement error, the idea that something throws a monkey wrench into an otherwise careful attempt at accurate observation. Biases, for example, pop up in all sorts of interesting and confounding ways—I’ve seen users struggle with a site but say they loved their user experience. If not an issue with the Web design itself, users bring expectations with them as they use Web sites; one man I interviewed for the Social Security Administration consistently used an interface wrong because he didn’t personally identify himself as disabled, even though it took him 10 minutes to cross the room with his walker before taking his seat.

Usability scientists who aim to measure the success and efficiency of online systems have created an arsenal of tools for gathering more accurate information that can stem the effect of whatever measurement error is in play. One of those tools is a Likert Scale. We’re all familiar with them, those are the “rate this from 1 to 5, 1 being least and 5 being most,” items that float around opinion surveys and rating systems like Yelp. In truth, a scale can go from 1 to 3, 1 to 4, or 1 to 10, or whatever the designer thinks the range should cover.

But Likert Scales are notorious in the world of usability data collection, because very few people design them correctly, and very few respondents react to them appropriately. Problem number 1 with the scales is the discrete distance between each option: the scale demands that the difference between 1 and 2 is equivalent to the distance between 2 and 3. But emotionally, if we are judging our own satisfaction with something, can we parse out our feelings that way? Is happiness always even across a continuum? In terms of satisfaction, there is a lot of evidence that scores tend to drift toward the extremes of the scale, no matter how many markers there are in between. And for items that aren’t controversial, many people will select the middlemost answer. Some scale designers set up even-numbered scales to eliminate the lazy neutral response, but that doesn’t address the problem of pole attraction for respondents.

Another issue with Likert Scales stems from the individual differences among people making selections. When a nurse asks us, “What is your pain level,” and points to the familiar happy or sad face scale, what is a 3 for me may be a 6 for someone else. This particular scale is also problematic for the fact that while most of us no what the “no pain” rank feels like, we have very different (or no experience) with the “worst possible pain” point on the scale. Thus that range of 1 to 10 can be markedly different for different people. Perhaps the medical practitioner is only looking to see how someone feels generally, but this particular problem persists in other applications of the scale.

Which brings me to the now-standard 5-point scale for rating books. Amazon, Goodreads, Barnes & Nobel all have a kind of satisfaction rating available so that readers can make their assessment quickly, and this widget is typically separate from a text box where a written review would go. In granting access to such quick or snap judgments, one would think that users would be careful about where on the scale they made their mark. Instead, most responses are selected in the 4 and 5 slots, a strong preponderance for the top of the scale. Here are some examples I pulled from Amazon and Goodreads:

First, it seems to me that there are different readers out there—readers who like to rank books they’ve read, and casual rankers who will do it if the scale is presented to them. Obviously more popular books generate more responses, but it would appear that there is often a bias toward positive ranking over neutral or negative ranking. I will wonder out loud if the active book rankers, the first group, have a different style of ranking; that is, are people who look forward to rating books more inclined to rate in the middle than the casual reader who sometimes rates books?

That said, when a poor book hits the scene readers will respond with very negative numbers. And all of the history on Amazon of fake ratings going up for some titles can’t mask a very crappy book, as the 1- and 2-star ratings will start to climb despite an author or publisher’s attempts to fluff up the average rating.

All of this leads me to thinking that the scale ratings scheme just doesn’t work well for books. On the one hand, some people try to manipulate the numbers and results, and on the other, measurement error seems to be firmly entrenched in this design of recommending or rating titles. Even just splitting out that distinction: “Did you like this book” versus “Would you recommend this book” may garner different results.

What I would rather see is a couple of yes/no widgets like those above, with a short box for users to say why or why not. I can understand the impetus behind the ratings game, but I’d rather use more traditional sales markers and qualitative-based reviews from readers to gauge a book’s interestingness or quality. And I have high hopes that someday, others will agree with me.

3 Comments on “The Uselessness of Likert Scales in Rating Books”

The problem inherent with the Amazon or Goodreads reviews is that the vast majority of the reviews are going to come from people who really liked the books and want to suggest that more people read them. I would imagine that most people do not post reviews for every book that they read (I post them all on my blog, but not on Amazon or Goodreads).

Generally if you’re looking at Amazon for ideas of books to read I would pay more attention to the negative reviews than the positive reviews. If a book has mostly positive reviews that isn’t necessarily something that will push me towards that book if I was on the fence about buying it. (I’ve seen books on Amazon that had a 4 or 4 1/2 overall rating that I hated.) On the other hand, if I was on the fence about buying a book and it had mostly low ratings on Amazon that would probably be enough to keep me away from the book.

Well, I suppose books with more reviews may be somewhat more helpful in that there ought to be more negative reviews out there, but books are so subjectively experienced that I’d rather just see whether a reader liked it or not, rather than trying to parse out what a rating vote means. Like you say, books I’ll hate may have high scores, or vice versa. There are helpful ways of structuring data on Web sites, and sadly, unhelpful ways. I put Likert Scales in the latter camp when it comes to reviewing novels.

I find it hilarious that I was asked to rate your post at the end! I had the same problem trying to find an app to track my moods. Does dysphoric count as bad? What about crampy? The things that I wanted to measure…all the gray areas between depressed an manic…were lost amid smiley/frowney faces!

BUY THE UNINTENTIONAL TIME TRAVELER!

Buy Bumbling into Body Hair!

Endorsements

"Everett's work is luminous, brilliant, thoughtful and brave. I am so proud to be on this earth with him. He is who we all fight for, and he is the future of our community."
—Margaret Cho, comedian and author of I Have Chosen to Stay and Fight

"Everett Maroon has written a book that is brave, funny, smart, and true. I admire his courage, his wit, his unflinching eye, and most of all, his persistance and determination to be himself. This book is a gift from a very generous writer. The story of Everett's journey will stay with me for a very long time."
—Lesléa Newman, author of October Mourning: A Song for Matthew Shepard and Heather Has Two Mommies

"Reading Bumbling into Body Hair by Everett Maroon—LOVE IT! Great message for trans folk on being our whole selves—AND it's funny."
—Kate Bornstein, author of My New Gender Workbook and A Queer and Pleasant Danger

Email Subscription

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 4,195 other followers

Top Rated

Archives

Archives

Disclaimer

All of the writing on this blog, unless I have otherwise noted or cited it, is the creation and sole property of Everett Maroon. No writing from Transplantportation.com may be reproduced or used without my express written permission. For permission to reproduce or use my writing, please email me at ev.maroon at gmail dot com.