The companion blog by the authors (Randy Farmer and Bryce Glass) of the O'Reilly book: Building Web Reputation Systems.

Leaderboards Considered Harmful

Note: Randy and I both had the good fortune to be asked by our (ex- and current) Yahoo! colleagues Christian Crumlish and Erin Malone to provide essays for their upcoming book, Designing Social Interfaces. Randy's essay, The Tripartite Identity Pattern is available on his own blog. This essay appears in draft form here for the very first time. Public comment is welcome and encouraged! Help me improve it in time for inclusion in Erin and Christian's (sure-to-be-awesome) book.

It's still too early to speak in absolutes about the design of social-media sites, but one fact is becoming abundantly clear: ranking the members of your community—and pitting them one-against-the-other in a competitive fashion—is typically a bad idea. Like the fabled djinni of yore, leaderboards on your site promise riches (comparisons! incentives! user engagement!!) but often lead to undesired consequences.

So why do we use them? The typical thought-process goes something like this: there's an activity on your site that you'd like to promote; a number of people engaged in that activity who should be recognized; and a whole buncha other people who need a kick in the pants to jump in. Leaderboards seem like the perfect solution. Active contributors will get their recognition: placement at the top of the ranks. The also-rans will find incentive: to emulate leaders and climb the boards.

And that activity you're trying to promote? Usage should swell with all those earnest, motivated users plugging away, right? It's the classic win-win-win scenario! In practice, employing this pattern has rarely been this straightforward. Here are but a few reasons why leaderboards are hard to get right.

What do you measure?

Many leaderboards make the mistake of basing standings on only what is easy to measure. Unfortunately, what's easy to measure oftentimes tells you nothing at all about what is good. Leaderboards tend to fare well in very competitive contexts, because there's a convenient correlation between measurability and quality. (It's called "performance"—number of wins versus losses within overall attempts.)

But how do you measure quality in a user-generated video community? Or a site for ratings and reviews? It should have very little to do with the quantities of simple activity that a person generates (the number of times an action is repeated, a comment given or a review posted.) But these types of things—discrete, countable and objective—are exactly what leaderboards excel at.

Whatever you do measure will be taken way too seriously

Even if you succeed in leavening your leaderboard with metrics for quality (perhaps you weigh community votes, or count 'send-to-a-friend' actions), be aware that—because the leaderboard singles these factors out for praise and reward—your community will hold these things in high esteem as well. Leaderboards have this amazing 'Code of Hammurabi' effect on community values: what's written becomes the law of the land. And you'll likely notice this effect in the things that people do—and won't do—on your site. So tread carefully—are you really that much smarter than your community, that you alone should dictate the makeup of its character?

If it looks like a leaderboard, and quacks like a leaderboard…

Even sites that don't display overt leaderboards may veer too closely into the 'comparative statistics' realm. Consider Twitter, and its prominent display of community members' stats.

The problem may not lie with the existence of the stats but—perhaps—in the prominence of their display. They give Twitter the appearance of a community that values popularity and the sheer size of your social network. Is it any wonder, then, that a whole host of community-created leaderboards have sprung up to automate just such comparisons? Twitterholic, Twitterank, Favrd and a whole host of others are the natural extension of this value-by-numbers approach.

Leaderboards are powerful and capricious

In the earliest days of Orkut (Google's also-ran entry into social networking), the property managers featured a fun little widget at the top of the site: a country-counter, showing members' geographical origins. Cute, right? Harmless, certainly. Google had no way of knowing, however, that the entire population of Brazil would make it a point of national pride to push their country to the top of that list! Brazilian blogger Naitze Teng writes: "Communities dedicated to raising the number of Brazilians on Orkut were following the numbers closely, planning gatherings and flash mobs to coincide with the inevitable. When it was reported that Brazilians had outnumbered Americans registered on Orkut, parties [...] were thrown in celebration."

Today, Brazil maintains its number one position on Orkut (51% of Orkut users are Brazilian as of this writing—the US and India are tied for a distant second with 17% apiece.) Orkut is—basically—a Brazilian social network. Which is not a bad "problem" for Google to have, but probably never an outcome they would have expected from such a simple, small and insignificant thing as a leaderboard widget.

Cui bono?

This may be the most insidious artifact of a leaderboard community: the very presence of a leaderboard changes the community dynamic and calls into question the motivations of everyone for any action they might take. If that sounds a bit extreme, consider Twitter: friend counts and followers have become the coins of that realm, and when you get a notification of a new follower...? Aren't you just a little more likely to believe that it's just someone fishing around for a reciprocal 'follow'? Sad, but true. And this is a site that itself has never officially featured a leaderboard. Twitter merely made the statistics known and provided an API to get at them: in doing so, they may have let the genie out of the bottle.

TrackBack

TrackBack URL for this entry:http://buildingreputation.com/writings-mt/mt-tb.fcgi/4

Leaderboards Considered Harmful

Note: Randy and I both had the good fortune to be asked by our (ex- and current) Yahoo! colleagues Christian Crumlish and Erin Malone to provide essays for their upcoming book, Designing Social Interfaces. Randy's essay, The Tripartite Identity Pattern is available on his own blog. This essay appears in draft form here for the very first time. Public comment is welcome and encouraged! Help me improve it in time for inclusion in Erin and Christian's (sure-to-be-awesome) book.
It's still too early to speak in absolutes about the design of social-media sites, but one fact is becoming abundantly clear: ranking the members of your community—and pitting them one-against-the-other in a competitive fashion—is typically a bad idea. Like the fabled djinni of yore, leaderboards on your site promise riches (comparisons! incentives! user engagement!!) but often lead to undesired consequences.