Whither the Long Tail in SEO?

Google is changing the way it deals with long tail search. In short, as it struggles to monetise lengthier and more specific search queries, it is increasingly promoting big brands and aggressively targeting common search queries through AdWords. As a result, the head of the demand curve has become bulkier and the long tail is gradually disappearing.

In this paper, we’re taking a closer look at the “long tail” phenomenon. We’ll illustrate how, in many ways, the digital boom democratised the consumer market by making niche “long tail” products and services more accessible. We’ll examine how big brands have fought back and, also, how Google has helped to facilitate the change.

What are the origins of the long tail?

Long tail search engine marketing refers to the practice of drawing custom to a website from less frequent search queries. The term was coined by author and editor of Wired, Chris Anderson, who argued that our consumer culture had shifted away from prioritising a small number of mainstream markets and products – the head of the demand curve – towards a much larger number of niches – known as the long tail.

In 2006, Anderson published “The Long Tail: Why the Future of Business is Selling Less of More”, which expanded upon a set of observations he penned two years previously.

In 1997, Jon Krakauer released Into Thin Air, a book about the 1996 Mount Everest disaster, which claimed the lives of eight climbers. It sold very well. A decade previously, British mountaineer, Joe Simpson, published Touching the Void, recounting his nearly fatal climb of Siula Grande in the Peruvian Andes – sales had been fairly modest. Following the success of Into Thin Air, however, sales for Simpson’s book curiously, and dramatically, picked up.

So “what happened?”, Anderson asked. Amazon recommendations happened. “The online bookseller’s software noted patterns in buying behaviour and suggested that readers who liked Into Thin Air would also likeTouching the Void”, he wrote. “People took the suggestion, agreed wholeheartedly, wrote rhapsodic reviews. More sales, more algorithm-fuelled recommendations, and the positive feedback loop kicked in.”

Amazon fuelled a new phenomenon “by combining infinite shelf space with real-time information about buying trends and public opinion.” As a consequence, the demand for a previously obscure book rocketed. In 2003 it was turned into a critically and commercially successful documentary of the same name, winning the BAFTA for Best British film. And, in 2014, the book was introduced as part of AQA’s GCSE English Literature course.

So, it was quite the unexpected turn around – all thanks to Amazon’s recommendation algorithm and the long tail.

In the world of online search the “tail” is indefinite and the combined total of less known search queries easily surpasses the sum of popular searches. This trend is otherwise known as the “Pareto principle” or the 80-20 rule, stating that about 80% of effects derive from 20% of causes.

In 1906, the Italian economist, Vilfredo Pareto, observed that 80% of all the land in Italy was owned by just 20% of the population and, similarly, 20% of the pea pods in his garden contained 80% of his peas.

The Pareto Principle is applicable to a surprising number of business and economic matters. In most countries, 20% of people earn 80% of the national income. And, with regards to online sales, 20% of customers purchase 80% of sales, meaning that the majority of keyword traffic lies in the long tail of search.

Bricks-and-mortar stores tend to only stock the most popular consumer goods. However, as online retail became more popular, and as production and distribution costs decreased, niche goods and services became more accessible and economically viable. Retailers like Amazon and iTunes can stock virtually anything, resulting in sales for niche goods outstripping short head (mainstream or “hit”) products. “The potential aggregate size of the many small markets in goods that don’t individually sell well enough for traditional retail and broadcast distribution may someday rival that of the existing large market in goods that do cross that economic bar”, explained Anderson.

Over the past decade consumers have been offered more and more choice. The nature of supply and demand has changed as a result. “It turns out to be less hit-centric than we thought”, Anderson wrote. “People gravitate towards niches because they satisfy narrow interests better, and in one aspect of our life or another we all have some narrow interest.” In other words, the online retail boom, coupled with new distribution mechanisms such as online downloading, streaming and peer-to-peer markets (like eBay), shifted our consumer mass market to an indefinite number of niches.

Anderson’s much vaunted notion that profit lies in long tail has proven to be short lived, however.

Indeed, in September last year, the Professor of Business Administration at Harvard Business School, Anita Elberse, addressed an audience about the decline of the long tail. Elberse compared the business strategies of two giants of the entertainment world: the head of NBC, Jeff Zucker and president of Warner Bros., Alan Horn. Zucker’s business model focused on cutting spending and managing for maximum profits across all its programmes. Conversely, Horn’s strategy involved taking more risk and hedging bets with a few big potential outcomes.

The results? Well, over the past decade, NBC has struggled commercially, while Warner Bros. has acquired huge profits.

“The notion of smaller bets being safer is a myth,” says Elberse, author of the new book Blockbusters: Hit-making, Risk-taking, and the Big Business of Entertainment. “It is safer to make bigger bets because they are likely to have bigger outcomes.”

So, larger scale brings marketing advantages and is more likely to be commercially successful in the longer run; even if, for instance, some big movies bomb at the box office.

Digital technology has significantly reduced the costs of buying and selling. Anderson’s long tail “was a wonderful egalitarian theory that would have us supporting small publishers, young authors, fringe artists and macramé jean makers”, writes Ginny Marvin, “pop culture and mass market would no longer rule commerce.” However, digital technology did not spell the end for the blockbuster, argues Elberse. The tail is now getting thinner, while the short head is getting bigger as big brands become more dominant.

Elberse uses the example of music sales: only 102 music titles accounted for 15% of total music track sales in 2011 – only 0.0001% of around eight million artists. Conversely, 74% of music titles – around five million different artists – accounted for only 1% of sales that same year.

The former Google CEO, Eric Schmidt, initially anticipated that AdWords would make most its money from the “tail”. However, he later discovered that the major advertisers generate the most revenue – a 90-10 type model. Elberse underlines how despite having over twenty million available tracks, Spotify users have never played one fifth of them. YouTube is investing heavily in the big brand channels. Meanwhile, Netflix continues to curate more of its content, rather than carrying every title.

In March 2014, MIDia Consulting published a new report entitled “The Death of the Long Tail: The Superstar Music Economy”, which argues that the music industry is dominated by a few best-selling artists rather than a long tail of niche performers. “The music industry is a Superstar economy, that is to say a very small share of the total artists and works account for a disproportionately large share of all revenues”, reports the Music Industry Blog. “This is not a Pareto’s Law type 80/20 distribution but something much more dramatic: the top 1% account for 77% of all artist recorded music income.”

The digital boom democratised access to music, which theoretically should have diluted the dominance of superstar names. However, the proliferation of digital music services has, if anything, further entrenched the concentration of big names. For example, the top 1% of artists account for 75% of CD sales and 79% of subscription revenue.

Why? Firstly, consumers are overwhelmed by a “tyranny of choice” – too much choice makes it more difficult to discover smaller niche artists. Secondly, there is a smaller amount of “front end” display for digital services, particularly on mobile devices. And, thirdly, the arms race between music catalogue providers is handing popular artists even more exposure and places the majority under “a pervasive shroud of obscurity.” Discovery mechanisms will evolve, but big name artists will likely still reign supreme.

“We used to operate on the 80-20 rule”, Jonny Geller, joint chief executive of the literacy agency Curtis Brown, told the Financial Times. Geller refers to the Pareto principle that 80% of sales derive from 20% of contributors. “Now, it’s more like 96 to four.”

The release of J.K Rowling’s novel, The Cuckoo’s Calling – under the pseudonym of “Robert Galbraith” – typifies the move away from long tail to mainstream sales. The moment word got out about the true identity of the author, sales of the novel went from modest to stratospheric. The novel surged from 4,709th on Amazon to number-one bestseller and signed first editions sold for as much as $6000. “It’s long been known that the publishing industry works through sheer numbers”, wrote Adam Taylor in Business Insider. “Publishers throw everything at the wall and see what sticks, and the mega-hits, of which there are few, pay for the flops, of which there are many. What Rowling’s experiment reveals, however, is that even a great book by an unknown author can simply get lost.”

Radiohead’s Thom Yorke pulled the new album of Atoms for Peace from the Spotify streaming service, protesting how smaller artists are at a huge disadvantage because the business model favours big names too much. “Small labels and artists can’t even keep their lights on”, wrote his producer Nigel Godrich on Twitter. “The big guys still make all the money, and the little guys have an even worse time than they used to.”

How responsible is Google for the decline of the long tail?

The long tail has become far less profitable because Google continues to consolidate its search volume against fewer keywords. For example, SEOs created webpages specifically targeting misspelled search requests, which equated to around 10% of all queries. The introduction of Google’s auto-correct and Instant features, however, made these pages virtually ineffective. Moreover, the “search mutation” feature also seeks to actively affect search behaviour by changing some queries to more popular requests.

So, Google is actively guiding users down fewer and more popular keyword paths.

“Google struggles from an advertising perspective because they’d like to be able to serve up great ads targeting those long-tail phrases, but inside of AdWords, Google’s Keyword Tool, for self-service advertising, it’s tough to choose those, says Rand Fishkin of Moz. “Google doesn’t often show volume around them. Google themselves might have a tough time figuring out, “hey, is this query relevant to these types of results,” especially if it’s in the long tail.”

A number of updates to Google’s algorithm have curbed on the long tail. In 2010, the May Day update targeted websites with poor content and necessitated more domain authority for sites to rank tail keywords. Since 2011, Google Panda has penalised sites with too many pages; so now the cost of being penalised outweighs the benefit of creating multiple webpages. The Freshness update in November 2011 promoted new content. Smaller sites with fewer resources struggle to roll out topical and authoritative content and, if Google doesn’t recognise them as a big brand, then they might be penalised for creating too many pages. Google Places and the Venice update integrated localised results more tightly into SERPs, which further splits online traffic. Furthermore, the introduction of the Hummingbird platform in August 2013 integrated more sophisticated semantic search tools – favouring big brands.

Long tail results are being either eaten up, or downgraded, in Google’s search results. Major sites have larger listings, AdWords sections are larger and the incorporation of new Google platforms such as YouTube, Google Books and News stories pushes long tail brand pages down the rankings. What’s more, Google’s promotion of auto-generated scraper sites, answer spam sites and sites that use third party content means that even if a small brand creates pages targeting the long tail, Google directs traffic to sites with copied content. It also hides keywords for users that are signed into their account. Therefore, site owners often do not know what search terms browsers used to find their content (even though relative themes can still be extrapolated from “keyword not provided” data).

Mass markets are becoming ever more competitive and big brand names will benefit as a result. Major companies can afford to take risks, command huge deals and outflank the competition – just look at the likes of Apple and Red Bull says Elberse. Conversely, spreading risk around is now a less effective way of making money.

Google has sought to kill the long tail because it is much more difficult to monetise and police. Nowadays, there are far fewer unique search requests. Big brands dominate the top of SERPs, meaning Google acquires huge profits because small businesses need to purchase advertising space to gain exposure. Targeting long tail keywords no longer guarantees traffic to your site, which only serves to underline the importance of bespoke organic SEO. As Google prioritises user-experience the demand curve will flatten and, if smaller brands play to their strengths, they will still be able to compete with their larger counterparts.