Yelp’s Review Filter Sometimes Does More Harm than Good

People like that — hit-and-run reviewers — aren’t welcome at Yelp, no matter what they have to say.

That’s all fine and good when Yelp’s review filter suppresses spammy, untrue, 5-star reviews of businesses that don’t deserve it. But, as Justin Vincent wrote yesterday, it’s not all fine and good when the filter suppresses legitimate one-star reviews.

Vincent says that he chose a moving company based on the preponderance of 5-star reviews it had. But, after a terrible experience that cost him $ 2,000, he found 10 negative reviews that Yelp had filtered because they came from non-power users.

Your algorithm typically hides entries by people who only post one review and who don’t otherwise engage in Yelp. Your assumption is that if a user only posts one review, posts no comments, has no friends etc. then most likely they are fake and trying to game the system.

Let’s call this “Assumption X”.

In the case of the company that I mention above (the one that ripped me off) Assumption X is exactly wrong at least 10 out of 14 times. Just to be clear, 10 honest one star reviews have been hidden from public view. That’s a 71% false positive hit rate.

So why did Yelp get it wrong 10 times?

In each case the one star review was left by someone who would never normally leave a review… they were simply so outraged that they were motivated to signup to Yelp and try to warn others how bad this company is. None of them ever used Yelp again. Furthermore, they didn’t have the knowledge or inclination to try to make their Yelp profile look acceptable to Yelp’s automated suppression systems.

Whether you love Yelp or hate it (and I don’t have a strong opinion one way or another, so please, fanboys, save your venom for someone else), that’s a blog post worth reading and thinking about. I’m not sure I agree with the solution that Vincent proposes, but I do agree that Yelp has to come up with some improvements that allow real reviews to show up on a business listing, whether they come from active Yelpers or not.