Algorithmic search is sinking

There's a fascinating story in the New York Times today about an online retailer who actually increased sales and profits by insulting, threatening and even cheating his customers because the more online complaints he got, the better he ranked on Google.

A woman who purchased eyeglasses on one of this online retailer’s sites was harassed and stalked for weeks because she tried to return a purchase. At one point the online retailer told her, "you put your hand in fire. Now it’s time to get burned." The woman told the New York Times, "This might sound like an exaggeration, but I feared for my life. I was actually looking over my shoulder when I left my apartment."

It turns out that the hundreds of online complaints being written about this bad actor were perversely fooling search algorithms into believing this was a quality site because it had a large number of inbound links. In fact, this retailer would intentionally begin battles with customers when he needed to drive an increase in traffic.

Unfortunately this is just a single appalling story in a huge trend we're seeing. There are a finite set of decent retailers you might want to buy stuff online from. But there is an ever-increasing number of spam sites on the web. We're at the point now where there are far more fake retailers than real ones online. The bad sites are getting ever more sophisticated in appearing to be legitimate, to both consumers as well as search engines.

Algorithmic search is sinking.

The only way to combat this and return trust and quality to search is by taking an editorial stand and having humans identify the best sites for every category. The algorithm can't find its way through the web's growing hall of mirrors anymore. And it's only going to get worse.

indeed, timing on that article could not have possibly been better, in the context of where you guys are going. i hadn't heard about blekko until i ran into tim c. last month, but i'm rapidly becoming a fan.

This does seem like a problem, but people are always going to be gaming the system. Do you really think humans would be perfect in filtering the results?

It seems like the problem is not that we are using algorithms to find things on the web, the problem is the algorithms being used. Hasn't there always been a cat and mouse with search with people trying to game the results to get better placing and the search engines working on better ways to provide quality results? This seems like a hard, but not impossible problem for companies like Google to solve to find a way for their algorithms to prevent people like this from gaming their way to the top.

Throwing out the algorithms in place of human work wreaks of a "throwing the baby out with the bathwater" type of issue.

Rich - I'm actually a big fan of adding some editorial discretion into an algorithmic mix (have a blog post on a very similar topic queued up), but I don't think the claim "algorithmic search is sinking" is accurate - at least not yet.

Until we see a year in which fewer searches are performed or fewer searches per searcher are made at Google/Bing, we're just "technologists" and "early adopters" making claims the data doesn't back up. To my knowledge, the search engines' internal tests have yet to show two consecutive periods when searcher satisfaction went down either (a good leading indicator).

I'm not saying that you're definitely wrong, just that so far, it's only a few voices (albeit attention-worthy ones who post in the tech sphere) declaring this phenomenon so. Given that we've been hearing pronouncements of this sort since 2004 (maybe even earlier), it's hard to convince a wider group, particularly when the data's against us.

All that said, I "feel" the same way you do. Google's webspam team appears to be falling down on the job more than they have in the past, and Bing is still a ways behind them. I also think the introduction of image previews, Google Places, and some (though not all) of the rich snippets and vertical results integrations are taking away from the beauty, simplicity and elegance of Google's SERPs.

Maybe Blekko can capitalize on that weakness. I certainly wish you luck!

Certainly it cannot be an accident that we see, in Google's blended results, a great deal of curated content.

Ten blue links were dead/dying some time ago, at least according to some of our best industry pundits!

It puts Google in a spot, though. They may agree and be taking actions that demonstrate their concern for the inadequacy of TBL, but they can't say so in so many words because their mythology rests on it.

In any case, there sure are a lot of "star ratings" all over Google, Google Maps, etc. wherever your mouse may roam, aren't there? And good old real world human review sites are the underlying data -- TripAdvisor, Yelp, etc.

You are of course only referring to one set of algos that are sinking. On the paid side algos are certainly rising, not sinking. As you know a great number of transactional queries end up clicking the ads - that are often more relevant (due to a/b testing of headlines and the like) and more useful than the organic results.

The bigger issue (IMO) is that there are a large number of really great mid/small retailers out there that people should not be afraid to buy from. This has been the case since day 1 of the web. The current algo changes (May Day et al) have skewed the SERPs (once again) towards the Amazon's and Wal-marts of the world. This is the gentrification of the SERPs and I think it's a bad thing for Search. At its core Search is about discovery. Let's not place limits on our ability to discover. Curation can be just as harmful and limiting in this regard as any algo. In fact I'd argue it can be moreso.

The rise of the semantic web will help out with this. Google will be able to tell the difference between good feedback and bad feedback in the favored sources that publish good semantic data. The network of trust will rise again, just don't count on Google to write a reliable algorithm for the wild west we live in today.

@ R. J. Steinbert - I agree that differentiation between positive and negative reviews is necessary and likely to become reality. Maybe the review sites might react in the way they present their scoring (so that Google can interpret them better) otherwise the review sites are threatened with being undermined by abuse.