Tag: reputation-systems yelp

(4000 words on why arranging for fake reviews can be a legitimate response to reputation systems such as Yelp, where businesses cannot opt out, and where the site owner is neither accountable nor widely trusted.)

The story of Botto Bistro

You may have seen the campaign by the owners of Botto Bistro to become the worst-reviewed restaurant on Yelp; it’s brilliant, it’s successful, and it has significance way beyond Richmond just north-east of San Francisco, where Botto Bistro has been serving customers for years.

Radio Free Richmond was the first outlet to mention the campaign, and that was on September 12. Three days later the story was picked up by Mike Aidax in the Richmond Standard, and two days after that by Paolo Lucchesi of the San Francisco Chronicle. Then it went national, with appearances on CNN and CNBC. The most comprehensive account I’ve seen was by Cyrus Faravar in Ars Technica on September 21, which I’ll draw on again and again in what follows.

But let’s go back to the beginning. It was five years ago that Davide Cerretini and Michele Massimo quit their jobs to set up a restaurant in a “worse location—a ghetto—and serve amazing food without expectation of customer service or tablecloths.” And they seem to have been true to their word, adopting a take-it-or-leave-it attitude to their customers: “I work 12 hours a day, six days a week. But if you come here to try to make us change whatever it is that is in our place, just shut up and go somewhere else.”

So “Botto Bistro serves all of its food at its order counter—there’s no table service—and crams 10 tables amidst giant flat-screen TVs” and their website FAQ is awesome, with gems like this:

Q. Do you serve pizza by the slice?
A. No.
Q. Why?
A. Still no.

It sounds like something from Fawlty Towers, and you can imagine it’s not for everyone, but Davide and Michele have built a clientele and are apparently doing fine.

Enter Yelp

So why the trouble with Yelp? Well, according to Cerretini, Yelp has been continually pestering the restaurant to advertise on the web site “30 times a week, sometimes five times a day”. This may be an exaggeration. Still, for six months, Botto Bistro did advertise. Then they cancelled their ads but Yelp kept calling. “I insulted them and I called them bad names”, says Cerretini, “Yelp asked us to pay them as advertising so they can criticize me better.”

In frustration, the Botto Bistro owners flung up their hands (metaphorically, I wouldn’t want to stereotype Italians) and started their Hate us on Yelp campaign to become the worst-reviewed restaurant in the San Francisco area. Thirteen hundred reviews later, they proudly announced a one-star aggregate rating on Yelp, the lowest rating possible.1

Their Yelp page is a lot of fun. Some reviewers compete for the most outrageous complaint (“Their pizza ruined my marriage”, “I’m sitting here in my office in Hamburg/Germany and are in need of some food and learn that you are not able to deliver???”, “WORST place EVER!” and so on). Others are more obviously supportive (“Had this same idea when Yelp screwed over my business… just never went about doing it! Very glad to see it working out for you.” “Fuck you, Yelp. Long live Botto.”) Everyone is having a good time.

But Yelp has, stupidly enough, responded, as Faravar tells us. On September 15 “Marvin” from Yelp sent one of the co-owners an email that started this way:

Hi Davide,

I ‘m contacting you from the Yelp User Support Team because we’ve received complaints from the community that you may be offering incentives in exchange for reviews.

To be clear, this violates our Terms of Service…

After accusing Davide of violating Terms of Service to which he has not agreed, the email goes on to threaten “a Consumer Alert being placed on your listing”.

So three cheers for Botto Bistro. Its campaign is many things: it’s a gimmick, a gesture of frustration, a declaration of individuality and quirkiness, and an act of rebellion against the creeping tyranny of a world defined by software and Terms of Service.

But the campaign relies on novelty for its success, so it’s not a tactic that can be easily reproduced by others. What, then, can other restaurants do if they are frustrated by Yelp? It’s clear that some resort to “fake reviews”, with less panache than Botto Bistro. It’s an act that’s widely seen as both ethically and legally wrong, but maybe there is a place for it…

What’s wrong with fake reviews?

We all know what’s wrong with fake reviews. Crowd-sourced sites such as Amazon’s book reviews, TripAdvisor’s hotel ratings, and Yelp’s restaurant recommendations, depend for their success on being the honest and authentic opinion of those who have read, stayed at, or eaten at the subject of their review. If restaurant owners reward people, or pay people, to submit positive reviews (sometimes people who have not even eaten at the restaurant) then those reviews are “frauds”: Yelp talks about “fake” or “phony” reviews, and calls them “attempts to deceive consumers”.

Some law enforcement agencies agree that paying for reviews qualifies legally as fraud: last year New York regulators carried out a sting operation that got “19 companies to cease their misleading practices and pay a total of $350,000 in penalties.”

“What we’ve found is even worse than old-fashioned false advertising,” said Eric T. Schneiderman, the New York attorney general. “When you look at a billboard, you can tell it’s a paid advertisement – but on Yelp or Citysearch, you assume you’re reading authentic consumer opinions, making this practice even more deceiving.”

The finding of “false advertising” was not based on whether the content of the reviews matched reality, but on the fact that the reviewers were posing as authentic customers, while actually being paid. But of course there is nothing unusual in a business paying somebody to speak positively about their offering. We’ve all seen television advertisements featuring celebrities who are paid large amounts of money to endorse a product or service.

The claim that it’s worse when it’s not on an obvious billboard is also outdated. We all know that TV and movie product placements are endorsements inserted precisely to seem natural – where you can’t “tell it’s a paid advertisement” – by actors who may never have used the product they are endorsing. We’ve heard about “viral marketing” stunts in which people are paid to act as genuine enthusiasts for a product. This may be a bit sleazy, but it is also just business as usual.

There are many places where advertisement and “authentic” comment is getting mixed up. It’s of real concern where the venue is a public venue: the practice of drug companies promoting their own brand of painkillers in university lectures is an obvious one, but I’m sure you can think of more.

But why, at a time when ads and “authentic” comment mix in Facebook, Twitter and YouTube, is the Yelp reputation system off-limits as an advertising medium? Why is it “false” to pay people to say nice things about your business on Yelp’s review listings but not “false” to solicit endorsements on LinkedIn (tip: my LinkedIn profile is almost entirely fake), in the middle of a prime time television show? Why should Yelp reviews not be just another venue for marketing and promotional activities?

Yelp and restaurants: incentives

Just as Google’s search results pages hold both paid-for advertising listings and “organic” listings, so Yelp’s pages contain both advertising and “real reviews by real people”. Yelp contends that the two are kept separate, but the potential for conflict of interest is obvious and some business owners believe that Yelp manipulates its listings to benefit businesses that advertise with them. And I’ll come back to that later.

But Botto Bistro’s frustration with Yelp is different. The owners are not interested in promoting themselves on the site. They want nothing to do with Yelp. The catch is, they can’t opt out.

Consumers have the right to talk about what they like (and don’t like) about a meal they ate, a plumber they hired, or a car wash they visited. We don’t remove business listings, so your best bet is to engage with your fans and critics alike, and hear what they have to say.

(Aside: what a patronizing tone Yelp adopts as it tells business owners how to run their own enterprises, and how unsurprising that Yelp recommends actions that benefit—who else?—Yelp.)

If you’re a restaurant owner, Yelp wants three things from you. First, it wants reviews of your business in order to draw people to its site when they are looking for a place to eat. Second, they want to make money from you by selling you ad space. Third, they want to profit from the reviews of your business and your ads by striking partnerships with Microsoft, Yahoo, and maybe other search engine providers. It’s going to get two of those whether you like it or not.

The restaurant’s compulsory appearance on Yelp is both a promise and a threat. Yelp is the steward of the restaurant’s presentation on the site and paying Yelp is a way to improve that presentation. Regardless of whether Yelp actually manipulates its listings, there is an obvious conflict of interest in the roles.

If it’s going to be compulsory for restaurants to appear on Yelp, then we have to ask questions about the company’s accountability before we judge those who pay for reviews or (even worse) charge them with fraud. And the picture is not pretty…

While restaurants are held to strict standards when it comes to how they behave on Yelp, it turns out that Yelp itself is not accountable to the law at all. (I’ll look at American law here: if you’re a restaurant outside the US then holding Yelp accountable will be even more challenging).

It’s well known that Section 230 of the Communications Decency Act protects web site owners from being responsible for user-provided content on their sites, even when that content is the source of profits. The protection extends to Yelp, according to a judge in 2011 and reaffirmed in September 2014. Yelp has no legal responsibility to verify any comments made by reviewers, for factual accuracy, reasonableness, or anything else. So if a restaurant complains that a reviewer’s opinion is not based on fact, that’s not Yelp’s problem. Opinions can be both “authentic” and wrong.

But of course Yelp does not just assemble user content, it also arranges it and presents it. Yelp’s sorting algorithms can make a big difference to a restaurant’s business: is the company bound to be fair in its presentation of restaurants? Business owners recently took Yelp to court claiming that the company “extorted advertising payments from them by manipulating reviews on its site”. A US Federal Appeals Court found that—whether or not Yelp is manipulating reviews—it can do what it wants.

…even if small business owners could prove that Yelp manipulates their pages to persuade them to pay up for advertising, that doesn’t mean they are being extorted, or that Yelp is violating federal law… the law does not require Yelp to publish positive reviews in the first place, Judge Berzon wrote.

“By withholding the benefit of these positive reviews,” the judge wrote, “Yelp is withholding a benefit that Yelp makes possible and maintains. It has no obligation to do so, however.”

So Yelp is fundamentally unaccountable for the fairness of its reputation system.

The usual argument is that it is in Yelp’s own interest to maintain a trustworthy site—that market forces will keep it honest. But while it’s clear that a completely spam-filled site would lose its value, it is not clear that the optimal amount of “review fraud” from Yelp’s point of view is zero. Yelp is interested in volume as well as accuracy: it encourages participation by reviewers, restaurants, and users through a whole gamut of techniques. There is always going to be a balance between this encouragement and dissuasion of so-called “biased” reviewers. We cannot trust loosely-defined market forces to preserve fairness.

Yelp as a protection racket

So Yelp is not accountable, but what about its actual history? Does it act as a steward or is it running a protection racket?

It is known that Yelp asks for and gets free services in exchange for reviews. Each year Yelp rewards some of its reviewers by anointing them part of the “Yelp Elite Squad”, and among the benefits are some “epicparties“. So who pays? Here is what Yelp’s official blog says:

Sometimes Yelp foots the bill directly for the venue, food or drink; oftentimes we look for sponsors to help defray those costs (e.g., Peroni Beer comped the beer I drank last week in New York). The venues themselves typically provide the space, some staff and sometimes food or drink. This arrangement is great for yelpers because we can visit a new place and meet each other affordably. And it works well for the venues and sponsors because they get exposure to a group of local adventurers who, if they like what they see, may become future customers.

There’s a clear quid-pro-quo here: free stuff for reviews. Not that it’s explicit, but of course it doesn’t have to be. Ironically, Yelp understands how the implicit quid-pro-quo works when it comes to the small businesses being reviewed: here is an excerpt from Yelp’s advice pages for businesses:

Why does Yelp discourage businesses from asking for reviews?

Would-be customers might not trust you. Let’s face it, most business owners are only going to ask for reviews from their happy customers, not the unhappy ones. Over time, these self-selected reviews create bias in the business listing – a bias that savvy consumers can smell from a mile away. No business is perfect, and it’s impossible to please 100% of your customers 100% of the time.

Business owners obviously feel the implicit promise when approached by Yelp. According to the Chicago Tribune, back in 2009 Chicago cupcake shop owner Patty Rothman was approached by a Yelp staffer who “walked into her Gold Coast shop and ‘guaranteed us good reviews on the site if we catered one of their parties for free’. Offended but resigned, Rothman complied. And just as promised, positive reviews bloomed for the business right after the party, Rothman said.”

Or Doug Quint and Bryan Petroff of Big Gay Ice Cream in New York, which opened in 2011:

As Quint and Petroff explain, last summer before their New York City shop even opened its doors for the first time, they got an email from a so-called “Elite” Yelper that read, “I am going to bring a team of other Elite Yelpers on an ice cream crawl and we want to come to your store. Can you open early?” And then, in addition to asking the Big Gay Ice Cream guys to come up with special flavors just for them and sample-size portion of the whole menu, this Elite Yelper went in for the kill: “Think of all the great reviews you’ll get.” (link)

Beyond the solicitation of free services, accusations of the Yelp advertisement sales team employing protection racket tactics over the ratings posted on its site are widespread. Here are just a few:

…when the town hall opened up for Q&A an hour and a half later, business owners were quick to vent their frustrations.

Many slammed the company for allowing review posters to write inflammatory comments – one restaurant manager said she cried for three days after a Yelper said her restaurant was filled with Nazis, another said he’d received negative reviews from Yelpers who admitted they’d never tried the food – while others said they had been subjected to aggressive advertising calls from Yelp.

“I have one-star reviews for my diner from people that have never walked into the place. They’ve never stepped foot, they’ve never tried the food, but they give me one-star reviews. That’s insane. Why would you let someone like that stay on the site?” demanded Craig Martin, owner of Cafe 50s. “I spoke to your office, I called your guys, I emailed, I talked to your salespeople.”

Another business owner, vintage clothing shop owner Reiko Roberts, said the advertising pressure amounted to extortion. She said that when she declined to buy ads, “the lower reviews go to the top and the higher reviews go to the bottom.”

My anecdotal experience: I told the Yelp sales rep to get lost and to stop calling me after about 6 months of hounding. He finally relented… but guess what ALL of my positive reviews are now filtered. Up until that conversation, only about 25% of my reviews were filtered. Now EVERY SINGLE ONE.

I sat in on a sales call with a Yelp rep, and after questioning her on the finer details we decided Yelp wasn’t for us and told her so. She then became noticeably irritated and said “Good luck getting any positive reviews in the future.”

And for more, you can look (where else), on Yelp’s page on Yelp, where it has a rating of 2.5 and 3105 one-star reviews.

You might wonder if these claims are true. Is it fair of me to post these allegations without checking into their accuracy?

Well, one reason restaurant owners get angry at Yelp is that Yelp does not care whether factual claims made by reviews are true or verifiable: their rankings of businesses are based on opinion, regardless of veracity.

So I’m happy to tell you that my sophisticated, but unfortunately proprietary, algorithms have identified all these reports of Yelp as honest opinions of the people involved. I’m sure you can find Yelp’s response to these allegations elsewhere, but they haven’t created an account here on Whimsley, so I’m not going to post a link. What’s sauce for the goose is sauce for the gander.

I don’t see that a business owner has any ethical responsibility to maintain the integrity of a system that they feel is threatening their livelihood, is run like a protection racket, and which is unresponsive to their appeals. Seems to me like treating Yelp as a venue for marketing is a perfectly reasonable course of action.

Why trust reputation: bigger questions

So that’s the point of the essay, but before I finish let’s explore another issue. Part of the problem is Yelp itself, but if Yelp were better, would there still be a case for fake reviews? Sometimes yes.

There are problems that come from the design of the reputation system itself, even if it’s run “fairly”.

Yelp aggregates user ratings to show an overall “score” for each restaurant and then it ranks the restaurants in order of this score; Yelp uses this “aggregate and rank” approach to decide which restaurants to highlight and which to regulate to the bottom of a list.

Yelp is proud of the influence of its model. When Cyrus Faravar asked Yelp’s vice president of corporate communications Vince Sollitto about Botto Bistro, Sollitto replied: “I think that you have to wonder if this is the smartest strategy—there’s a Harvard Business School study that was done showing that for every [star] increase, restaurants see an uplift in revenue between five and nine percent.”

Yelp would maintain that a good restaurant will get good ratings from its customers and so will benefit from the site, but the story is not so simple.

While the net effects of reputation systems are complex and varied, there is a homogenizing, winner-take-all component that comes from this practice of ranking preferences and displaying them in order of average value. Yelp heavily favours those businesses that appear in the top few of the listings.

Yelp’s decision to rank based on average reviews favours restaurants with a broad appeal, and acts against more polarizing venues. It’s the same reason that many challenging or controversial movies that divide audiences (“cult movies” belong in this class) score lower on Rotten Tomatoes than those that are less divisive. Cult movies may attract a more intense following among their core audience, but there is nowhere in the Rotten Tomatoes system for such intensity to be reflected.

Like a cult movie, Botto Bistro’s owners set out to deliver their own vision in an uncompromising fashion. Appeasing those who don’t like it is not high on their list of priorities. As they told Faravar: “We specifically, aggressively inform people that if you’re looking for customer service, this is the last place you should be.”

There is a place for many types of restaurant, including those with Botto’s take-it-or-leave-it style. But it’s clear why Botto Bistro will never get far on Yelp even if they wanted to: Yelp’s whole model of reputation works against the idiosyncratic and difficult.

The homogenizing force of “average and rank” systems may become more severe as whole industries seek to increase the role of digital ratings in our daily lives. There are many in the technology world who dream of reputation as currency: the idea that a numerical reputation score on one system (such as Yelp) should be portable to other environments (TripAdvisor? Credit Card companies? Landlords?) A good score on eBay may indicate a “good reputation” that you could parlay into a better interest rate on a bank loan. It’s already happening to some extent, as Airbnb posts the number of Facebook Friends of each host as evidence of their trustworthiness. This kind of naive scoring system is a terrible idea. The only thing worse would be a sophisticated scoring system.

It’s well known that early reviews can have a big effect on many ratings systems by “setting the tone” for reviews that follow. If you know that early reviewers have given a restaurant five stars but you are underwhelmed, you may give it three stars rather than two.

The aggregation process itself can also affect your business. A couple of years ago Rebecca Rosen reported in The Atlantic on a study by Michael Anderson and Jeremy Magruder: the researchers seized on the observation that two restaurants can have very similar average ratings (say 3.24 and 3.26) and yet when Yelp rounds them into half-star categories one will be rounded down to 3.0, while the other is rounded up to 3.5. Looking at 400 restaurants in the San Francisco area, they discovered that “moving from 3 to 3.5 stars is associated with being 21 percentage points more likely to have sold out all 7:00 PM tables and moving from 3.5 to 4 stars makes restaurants an additional 19 percentage points more likely to have sold out all tables.” That is, Yelp’s own choices about how to present ratings, independent of business “quality” reflected in its average rating had a significant effect on some businesses.

In short, the impact of almost any reputation system on a restaurant’s business is not just the natural effect of that business’s quality, but is also a result of the system designer’s choices, which cannot be influenced by the business owner. Business owners are dependent, to some extent, on the design choices of an unaccountable company for their success or failure. That’s not a pleasant position to be in and it’s made worse by the “winner-take-all” structure of many social media marketplaces. If a few fake reviews are needed to limit the impact of large reputation systems, that’s fine with me.

The philosophy of ratings

Let’s finish up with what philosophers call “the question of meaning.” If we are to believe in a “fair reputation system” we have to ask: what, grasshopper, does a one-star (or a five-star) review really mean?

Yelp’s view (and TripAdvisor’s, and Netflix’s, and Amazon’s, and Airbnb’s, and eBay’s … you get the picture) is that a rating is an indication of individual preference, and that aggregating these preferences can be used to recommend the experience to others. If “reputation” is the social distillation of others’ opinions about ourselves, then these rankings are a measure of “reputation”, distilled multiple times into a single number.

But just because that’s what the aggregators would like a rating to mean doesn’t make it so. There is a world of context behind each click that may not fit into any algorithm’s assumptions. In Botto Bistro’s case, a one-star review is a gesture of support, a way to thumb your nose at an impersonal system, and a quick path to cheap pizza. It’s a dramatic example of how one “one-star review” may have a completely different meaning from another “one-star review” even though they are recorded in Yelp’s systems in the same way.

It’s tempting to think that if only “fake” reviews could be removed, then the remaining ones would indicate preference, but there’s little to back up such a claim. In other contexts (like eBay or Airbnb) a five-star review is more often a courtesy than a rating. Less dramatically, in the case of Netflix, one outcome of the Netflix prize was that reviews of one movie are heavily primed by what other movies we have recently reviewed (so that “objectivity” is tainted).

And maybe a “fake review” should be seen as a commitment on the part of the restaurant owner, an investment in branding that indicates an intent to stay in the marketplace, a sign that the restaurant can, in fact, be trusted — just like any other investment in branding.

Ratings do not always mean what data scientists would like them to mean. Human expressions are protean, and no attempt by Yelp or others to pin them down will succeed completely. There is something postmodern about ratings, just as any other human expression. We imbue them with meaning and intent outside of what the system wants, and there is no objective fact behind it all that is the “true” rating. Il n’y a pas de hors-click.

Reputation is a multi-faceted, qualitative concept. It has been pushed through a meat-grinder by digital reputation systems and has come out the other side homogenized, devoid of texture, but easier to digest. There’s nothing inherently wrong with reputation systems by themselves, but giving them too much authority or influence will inevitably throw up bad incentives on the part of the system owner and those taking part in the reviews. Before we declare “fake reviews” to be a crime or an obvious case of bad morals, we at least need to demand some accountability from reputation site owners.

Footnotes:

1As of this writing, Botto’s rating on Yelp has gone back up to 1.5, because the company has relegated over 600 reviews to the “not recommended” bin and removed over 800 for violating content guidelines, leaving only 560 reviews on the front page.