In the book, Farmer and Glass categorize the motivations for user participation as altruistic, commercial, and egocentric. Commercial motives are clearly the most problematic: a review site loses credibility if commercially motivated reviews are disguised to make their commercial motives. Most review site scandals arise from this kind of deception (e.g., this one, this one, and this one).

Sincerity is a necessary but insufficient condition for a review to be valuable to the person who reads it. There is still the “people like me” problem: sincere reviewers may still be uninformed, unreasonably biased, or may simply not share our tastes. User-generated content is an inherently subjective medium.

Given these challenges, it’s a wonder that online review sites work at all! And yet there are real success stories. My personal favorite is Amazon.com. While it has has its hiccups, Amazon nonetheless serves as a poster child for creating value by aggregating user opinions about products.

Amazon has a well-designed review policy that gets many key elements right:

Reviewers have identities tied to purchasing history. That encourages disclosure (people use their real names) and discourages abuse.

The reviews themselves–and even comments in discussion threads about individual reviews–are themselves reviewed as helpful or not. That may seem overly meta, but it does a lot to mitigate information overload.

But products aren’t the only reputable entities, to use Farmer and Glass’s term. What about service businesses, such as restaurants, gyms, etc. Or people?

If Amazon exemplifies online product reviews, then Yelp is the canonical example of a review site for service businesses. And, despite its own share of controversy, it is quite successful. But I dare say not quite as successful as Amazon. Part of the problem is that is demographics are less representative of the general online population (here’s what Quantcast says about Yelp and Amazon demographics for their US users). Also, there’s more variance in experiencing a service than in experiencing a product.

But Yelp has also has had a credibility problem regarding which reviews they allow to be published. Perhaps the root of this problem is that Yelp’s business model depends on paid advertising from the businesses reviewed on the site, while businesses would much rather have unpaid positive reviews. In contrast, Amazon makes its money buy selling products–which at least makes it perceived to be more evenhanded.

But neither Amazon and Yelp have touched the third rail of online reputation: people. LinkedIn dabbles in this space by allowing its members to review one another, but reviewees have veto power over reviews–making the review graph more of a mutual admiration society.

A recent startup, Unvarnished, is trying to create a review site with teeth. Farmer argues on his blog that Unvarnished is breaking some major rules:

It displays negative karma–that is, it allows people to write negative reviews of one another and displays those reviews.

The reviews are not clearly tied to context (e.g., were the reviewer and reviewee co-workers?).

The anonymity of reviewers does not incent altruistic or even egocentric behavior, and is thus a recipe for abuse.

I’m not as down on Unvarnished as Farmer, but I agree it will have an uphill battle to succeed. Ironically, for all of the public concern about Unvarnished becoming a trollfest, the reviews skew strongly positive. This is probably an artifact of how Unvarnished is growing its membership: current users ask friends to review them.

I agree most with Farmer that Unvarnished’s incentive structure seems problematic. A person’s friends will probably be inclined to write positive reviews, and may even be annoyed at having to write them anonymously. A person’s enemies may be inclined to write negative reviews as a form of attack or revenge. But it’s less clear what will incent people to write accurate reviews–or what will signal to readers that a review is trustworthy.

All in all, I think that these are early days in the online reputation space, and that there is ample room for innovation. Facebook’s recent release of “like buttons” is an ambitious attempt to boil the ocean of “social objects”. A best poster award at the recent WWW 2010 conference went to Paul Dütting, Monika Henzinger, and Ingmar Weber’s “How much is your Personal Recommendation Worth”. Hopefully all of these attempts to research and innovate will lead to a world where we can derive real value from others’ opinions and feel incented to contribute our own.

10 responses so far ↓

I see FB “like” button as a distributed fan page system. Fans around the web can cast their “like” by clicking an FB “like” button. Let just say we have 5 “likes” on a “Beginner Guide to Python Programming” book. We do not have enough information to judge how relevant this “likes” to our need. We do not have information on the background of these people who like this book. We do not know why they like the book. Nevertheless FB “like” button has created more meaning towards our gesture on the web. It is one step to create a more meaningful semantic on the web.

First of all — thanks for highlighting a good book for my to-read pile!

This was a great post and fodder for a longer discussion but two quick thoughts come to mind. Sorry for the long comment (I am waiting on a testing run to complete so I had time)

1.
Reputation should be some combination of reliability (proof that reviewer has the expertise to give the review), recency (how fresh is that reviewer’s knowledge of the reviewed content) and rating (the actual review content).
I think the recency attribute is not sufficiently exploited by many systems. The related side effect of frequency (i.e., did you rate item in isolation or as part of a cluster) is equally interesting. As the Netflix solution(http://pragmatictheory.googlepages.com/NetflixPragmaticTheoryV6.pdf) showed, user’s opinions do change with time — particularly as they acquire more knowledge on the topic and more items to correlate (balance) reviews across. Users may be harsher or more pragmatic in reviewing items later than they would be when caught up in the emotion of the moment.
It would be interesting to see systems instrumented to allow users to periodically re-evaluate or adjust ratings and study how this changes overall reviews. For instance, when I review a book as 4 stars on Amazon, if it then popped up samples of books that I had rated as 3, 4 and 5 stars for comparison – would I change my rating? Or would I alter a previous rating?

2.
The second comment was regarding Facebook. Right now their “Like” button simply allows you to state your opinion without necessarily targeting it at any user — however, it’s a short step to targeted likes where the user, in addition to clicking the Like button, can now select from a pop-up of his social graph, specific friends to send that recommendation to. More user effort but invariably cleaner (more valuable) intent data. We’re already seeing some behaviors like this on Twitter where folks “RT” a link but then add a “/cc” at the end (with a list of other twitter IDs) to indicate they want this brought to the attention of those specific users.

Zainul (and Nitya): I agree that the “like” button is a pretty minimal signal. If anything, the more interest potential consquence of FB rolling it out is that more site authors with adopt RDFa.

Nitya: Great point about recency. Stale reviews are a real problem for service businesses, and Farmer and Glass discuss strategies for aging out reviews. But I also like your point about how time gives us (or changes our) perspective. Makes me think about Kahneman‘s research on experienced vs. remembered utility.