Analysis

Google has been the king of the search-engine castle for so long we’ve grown used to thinking of it as a force of nature, something that has always been there and always will be. The company is responsible for close to 65 percent of all online searches, and its related keyword-advertising business continues to bring in vast quantities of cash, since Google controls about 80 percent of that business. But despite its success — or rather, because of it — the company also has a growing battle on its hands, one that threatens to not only jeopardize its core search business, but also the advertising that flows from it. The battle pits the web giant and its legions of programmers against search-engine spam artists and content “farms” who regularly game the company’s algorithms.

Search-result spam is not a new problem for Google. The company has always fought to keep its results clean, continually tweaking its algorithms so that unhelpful or spammy pages don’t show up as high as sites that actually provide value. How it does this is a closely guarded secret (although periodically innocent websites get caught in a spam sweep, and it becomes obvious that Google engineers have been fiddling with the levers). But while the desire to keep Google’s results spam-free is not new, there have been signs that the spammers may be winning — at least, that is the allegation made by some fairly high-profile critics, including entrepreneur-turned-academic Vivek Wadhwa, influential programmer Jeff Atwood and web developer Marco Arment. And in a recent response to these complaints, Matt Cutts, the head of Google’s spam team, admitted that the company could be doing more to improve its results.

A “Tropical Paradise for Spammers”

All of these critics have written polemics recently about the decline of Google’s search results, and how the first page of results for many searches is often useless because it is filled with pages that offer no useful information, just keywords strung together and content that has been cut and pasted from other sources. This is particularly true for those sites involving products that might be the target of advertising, and therefore appeal to search-engine optimization or SEO firms. Financial blogger Paul Kedrosky of Bloomberg has also written about the phenomenon, describing how useless a Google search was when looking for information about dishwashers: Virtually every result for the first couple of pages was put together by sites hoping to get a click on their ads, not anyone with actual useful information.

Vivek Wadhwa said that Google has become a “tropical paradise for spammers and marketers,” while Marco Arment complained that whenever he searches for anything, the results are filled with content “generated by penny-hungry affiliate marketers and sleazy web ‘content’ startups to target long-tail Google queries en masse, scraping content from others or paying low-wage workers to churn out formulaic, minimally nutritious pages to answer them.” He said that searching for products now is like “asking a question in a crowded flea market of hungry, desperate, sleazy salesmen who all claim to have the answer,” thanks to sites such as About.com and eHow.com.

A Problem of Perception?

But is the problem becoming worse or are people just complaining more? Some search-industry watchers argue that all search engines suffer from the spam problem, that Google is no worse off than anyone else, and likely better than most. But surveys show it isn’t just a few high-profile complainers who think Google’s results are filled with “noise.” And while the company may have been able to rest on its laurels before, it now faces more competition from Microsoft and its Bing search engine, which took over Yahoo’s search function last year, as well as from newer startups such as Blekko (which created a “spam clock” to show how many malicious pages are being created every minute). Bing’s search share has been creeping up recently and is currently around 11.5 percent.

Even more than that, Google faces competition on the advertising side of its business, primarily from Facebook, which is rapidly becoming the favorite of many advertisers due to its social focus, and is expected to pull in close to $5 billion in ad revenue this year. That may not seem like much next to Google, but Facebook’s rate of growth is much higher.

Content Farms Are Bad, But Also Good

Google’s problem is that over the past five years or so, it has effectively trained an entire industry of SEO consultants. This includes both the reputable kind and the “black hat” kind, who will use any trick to get their results higher in a search, including stacking a site with irrelevant but keyword-rich and link-heavy documents, in order to try and boost its “page rank” with the search engine.

There are also some very large companies that now specialize in creating that kind of content, including Associated Content (which was bought by Yahoo last year for $100 million) and Demand Media, which recently went public and is now valued at more than $1.5 billion. Demand has thousands of people writing articles about a staggering variety of topics for its eHow division, including things as mundane as how to change a tire or diaper a child — anything that the company’s own algorithms, which monitor Google ad-keyword trends, predict might be the subject of a web search and might bring in good revenue from advertising clicks.

In many ways, Demand Media and Associated Content are businesses whose sole purpose is to create content that will show up high in a Google search, and therefore can be easily monetized through keyword ads. And this relationship is a two-way street: Google also makes money from those ads, and so — as many critics have pointed out — it has a vested interest in having Demand articles show up high in search results, even if they aren’t really that helpful from a search standpoint (although our GigaOM Pro analyst David Card argues that the money Google makes isn’t enough to make much of an impact on the company).

Short-Term Gain, But Long-Term Pain

But while this keyword-advertising revenue might hold short-term benefit for Google, it risks alienating users over the longer term, and thus tarnishing the search engine’s reputation for useful results. And those useful results were the very claim to fame that gave Google a leg up when it first arrived on the web, since its results were noticeably better (and faster) than those from AltaVista and other leading search engines. If enough people decide that the company’s results are no longer useful, it could lead to an exodus of users, and that could be a nasty one-two punch for the web giant if such issues lead to more gains for Facebook and its “social search.”

In his blog post about spam and search results, which was posted following the criticisms from Vivek Wadhwa and others, Matt Cutts said that the search company admits there is more it could do when it comes to dealing with content “scrapers” and “farms” that clog up search results. Cutts also made a point of saying that the ad revenue Google gains from such sites has no influence on whether the web giant takes action against them or not.

Google is obviously still a massive company with huge resources at its disposal. And it has been trying to improve its search engine in a number of ways, including rewriting its search index to make it faster and adding real-time results from Twitter as way of bringing in some of the social element that it is currently lacking. Cleaning the spam content from its results might be a better place to devote its energies, even if it does lead to lower income in the short-term from content producers like Demand Media.