search

The best way to think of Facebook’s stock is as the sum of two businesses: the existing display ad businesses, and a probability-weighted option on a new line of business. This is how Wall Street views it. For example, here is a section of a recent Goldman Sachs analyst report on Facebook:

Optionality not in the model: further potential upside

While not in our model, as [Facebook] has not publicly expressed pursuit of these areas, we believe there are three obvious opportunities that the company could leverage its platform to capitalize on:

– Developing an external ad network

– Monetizing paid search

– Entering China

Of the three options, search is clearly the most interesting. An external ad network is inevitable. Google proved this model with Adsense. With an already huge base of advertisers bidding on CPCs, it is impossible for most other ad networks to compete on publisher payouts. But Facebook’s traffic is so great now that an external ad network might increase their revenues by 2x or so. The same goes for entering China. They might get another half a billion users who monetize at lower ad rates than US users. Neither move would put them in Google’s revenue range. They need a better business model for that. The only (known) models that deliver RPMs high enough to compete with Google are search, payments, and e-commerce.

At TechCrunch Disrupt last week, Mark Zuckerberg talked about possibly entering the search business. Investors had been concerned that maybe Zuckerberg really meant what he said in his IPO letter – that he just didn’t care that much about making money. By expressing an interest in search, Zuckerberg signaled that he understood Facebook’s immensely valuable embedded option and was thinking about ways to exercise it.

Most websites spend massive amounts of time and money to get any of their pages index and ranked by Google’s search engine. Indeed, there is a entire billion dollar industry (SEO) devoted to helping companies get their content indexed and ranked.

Twitter and Facebook have decided to disallow Google from indexing 99.9% of their content. Twitter won’t let Google index tweets and Facebook won’t let Google index status updates and most other user and brand generated content. In Facebook’s case this makes sense for content that users have designated as non-public. In Twitter’s case, the vast majority of the blocked content is designated by users as public. Furthermore, Twitter’s own search function rarely works for tweets older than a week (from Twitter’s search documentation, they return “6-9 days of Tweets”).

There is a debate going today in the tech world: Facebook and Twitter are upset that Google won’t highly rank the 0.1% of their content they make indexable. Facebook and Twitter even created something they call the “Don’t be evil” toolbar that reranks Google search results the way they’d like them to be ranked. The clear implication is that Google is violating their famous credo and acting “evil”.

The vast majority of websites would dream of having the problem of being able to block Google from 99.9% of their content and have the remaining 0.1% rank at the top of results. What would be best for users – and least “evil” – would be to let all public content get indexed and have Google rank that content “fairly” without favoring their own content. Facebook and Twitter are right about Google’s rankings, but Google is right about Facebook and Twitter blocking public content from being indexed.

Update: after posting this I got a bunch of emails, tweets and comments telling me that Twitter does in fact allow Google to index all their tweets, and that any missing tweets are the fault of Google, not Twitter. A few people suggested that without firehose access Google can’t be expected to index all tweets. At any rate, I think the “Why aren’t all tweets indexed?” issue is more nuanced than I argued above.

When Google released its search engine in 1998, its search results were significantly better than its competitors’. Many people attribute Google’s success to this breakthrough technology. But there was another key reason: a stubborn refusal to accept the orthodox view at the time that “stickiness” was crucial to a website’s success. Here’s what happened when they tried to sell their technology to Excite (a leading portal/search engine in the late 90s):

[Google] was too good. If Excite were to host a search engine that instantly gave people information they sought, [Excite’s CEO] explained, the users would leave the site instantly. Since his ad revenue came from people staying on the site—“stickiness” was the most desired metric in websites at the time—using Google’s technology would be counterproductive. “He told us he wanted Excite’s search engine to be 80 percent as good as the other search engines,” … and we were like, “Wow, these guys don’t know what they’re talking about.” – Steven Levy, In The Plex (p. 30)

Famed investor/entrepreneur Reid Hoffman says world-changing startups need to be premised on “accurate contrarian theories.” In Google’s case, it was true but non-contrarian to think users would prefer a better search engine. What was true and contrarian was to think it made business sense to get users off their site as quickly as possible. The business model to support this contrarian theory wouldn’t emerge until years later, and by then Google would already have become the world’s most popular search engine.

There is an amazing amount of useful, free information available on tech blogs for fledgling tech entrepreneurs (this list is a great place to start). I think sometimes we techies forget that this wealth of content is unknown to the non-startup world. I was reminded of this recently when I met a first-time entrepreneur who said when he was first starting out he tried finding books on Amazon, Googling for stuff etc. He described it as an epiphany the first time he stumbled upon Fred Wilson’s blog, which then led him to Brad Feld, Mark Suster, Eric Ries, Venture Hacks, etc.

So this weekend I thought I’d try an experiment. I took about 100 of my blog posts (the ones that I thought were most “evergreen”), bundled them as a PDF and submitted them to the Kindle Store. The Kindle submission process was surprisingly easy. You give your book a name and upload the PDF and then choose pricing. They force you to charge a minimum of $0.99. Also, strangely, if you charge less than $2.99, Amazon takes 70% of the revenue, but if you charge between $2.99-$10 they only keep 30%.

I decided to price my book at $2.99 and donate all of the proceeds (~$2 book) to HackNY, a non-profit that “keeps the kids off the Street” (encourages college students to join/start tech startups instead of working on Wall Street). All of the content in the book is available for free on cdixon.org. The only reason to buy the book is to get this blog in a different format and to support a good charity. It is available in the Kindle Store here.

I don’t expect many people to buy the book but maybe some first-time entrepreneurs will stumble on it and from there discover more tech blogs. Think of it as “Kindle SEO” for tech blogs.

Finally, I am having trouble getting the links to work on the Kindle version. I’m not sure if this is an Amazon policy or if I am just doing something wrong (the links work fine in the PDF I uploaded to Amazon). So here is an alternative version on Scribd that has working links.

Many of the today’s most successful informational sites such as Yelp, Wikipedia and TripAdvisor relied heavily on SEO for their initial growth. Their marketing strategy (whether deliberate or not) was roughly: 1) build a community of contributors that created high-quality content, 2) become the definitive place to link to for the topics they covered, 3) rank highly in organic search results. This led to a virtuous cycle where SEO drew more users, leading to more contributors and more inbound links, leading to more SEO, and so on. From roughly 2001-2008, SEO was the most effective marketing channel for high-quality informational sites.

I talk to lots of startups and almost none that I know of post-2008 have gained significant traction through SEO (the rare exceptions tend to be focused on content areas that were previously un-monetizable). Google keeps its ranking algorithms secret, but it is widely believed that inbound links are the preeminent ranking factor. This ends up rewarding sites that are 1) older and have built up years of inbound links 2) willing to engage in aggressive link building, or what is known as black-hat SEO. (It is also very likely that Google rewards sites for the simple fact that they are older. For educated guesses on which factors matter most for SEO, see SEOMoz’s excellent search engine ranking factors survey).

(TechCrunch had a very good article on the TripAdvisor’s decline in quality).

In contrast, this cleaner and more informative page from the relatively new website Oyster ranks much lower in Google results:

As a result, web users have a worse experience and startups are incentivized to clutter their pages with ads and use aggressive tactics to increase their SEO when they should just be focused on creating great user experiences.

The web economy (ecommerce + advertising) is a multi-hundred billion dollar market. Much of this revenue comes from traffic that comes from SEO. This has led to a multibillion-dollar SEO industry. Some of the SEO industry is “white hat,” which generally means consultants giving benign advice for making websites search-engine friendly. But there is also a huge industry of black-hat SEO consultants who trade and sell links, along with companies like content farms that promote their own low-quality content through aggressive SEO tactics.

Google seems to be doing everything it can to improve its algorithms so that the best content rises to the top (the recent “panda” update seems to be a step forward). But there are many billions of dollars and tens of thousands of people working to game SEO. And for now, at least, high-quality content seems to be losing. Until that changes, startups – who generally have small teams, small budgets, and the scruples to avoid black-hat tactics – should no longer consider SEO a viable marketing strategy.