Search Memories

Veterans of AltaVista, Excite and Infoseek reminisce about the early days of web search, and opine about Google, then and now.

A special report from the Search Engine Strategies conference, August 2-5, 2004, San Jose, CA.

In January 1962, Dick Rowe earned his place in popular music history, by becoming "the guy who turned down the Beatles."

As head of Decca records at the time, he had final say on whether they should sign up the Liverpool based combo, or London based outfit Brian Poole and the Tremeloes. He chose the latter and the rest, as they say... is history!

That story came flashing back to me as I sat in the audience for the Search Memories session at SES San Jose in August. Doug Cutting, Primary Developer, Nutch (formerly Senior Architect, Excite), Steve Kirsch, founder, Propel Software (formerly founder, Infoseek) and Louis Monier, head of R&D, eBay (formerly chief technical officer, AltaVista) together, reminisced about the early years of search engines.

And one by one, they each openly admitted that they had been approached by two students with a new idea for a search engine. "Go pound sand I told them" said Steve Kirsh. "I wasn't impressed with their demo at all" said Doug Cutting. "I didn't have the authority to sign a check anyway" said Louis Monier.

The two students which they all turned away, of course, were Larry Page and Sergey Brin. Their new idea for search was called PageRank, and would eventually become the power behind Google.

It was good to see that they could all share a laugh about it. Although I'm certain that, when they saw the expected valuation of Google for the impending IPO (as it was at the time of the conference), a little weeping and gnashing of teeth may have occurred behind closed doors.

This was a most revealing session for me. Not about the inner workings of early search engines, more about the similarities in the stories they each had to tell.

Doug Cutting explained that they were thrilled in 1995 to boast about having access to an index of a whopping 1.2 million pages. Eventually this would extend to 2 million pages. But only to find themselves overshadowed by the launch of the mighty AltaVista, hitting the ground running with super power and an index of 16 million pages. As moderator Chris Sherman commented, this must have been the start of the "size wars" at search engines.

And so it must have been, with Doug Cutting then receiving instructions to get Excite up to 50 million pages! For a while, Excite was the biggest thing in search (if size really does matter). Only to be eclipsed once again by the mighty AltaVista.

Yes, the mighty AltaVista with the huge power of Digital behind them...

I have to tell you, there was nobody in the room more surprised than me to discover that "the mighty AltaVista" at the time consisted of Louis Monier and about a half dozen other people that he had access to - and that was it. In fact, there was so little support, it was his wife who designed the company logo. The company was "just the bare bones".

However, judging by the press around the launch of AltaVista it seemed as if there must have been hundreds of people involved. "AltaVista was lost inside a dying hardware company that was in complete free fall" Louis explained. Digital research labs were recovering from what he called "a couple of black eyes" and had just been sued twice losing $100 million!

But none of this prevented the press guys trying to squeeze the maximum PR possible out of the launch of AltaVista. Louis lamented the fact that the press guys at Digital at the time, had such little knowledge of what AltaVista was all about. "You enter your query and within 24 hours you get a response they wrote!"

So what were the innovations in search during those early days?

We had enhanced relevance with a basic Google approach at Infoseek said Steve Kirsch. It was kind of like a poor man's Google back then. We'd look at the number of web pages that had hyperlinks containing the query term and try and rank the page with the most links at the top. And Infoseek was the first to have phrase searching.

Meanwhile at AltaVista Louis was in perplexed state once again as the strategy shifted from search to portal. "The attempt to turn AltaVista into a portal was just a freakin' disaster" but speed and universal access were top of his own list of priorities for search. "I once spent a week trying to iron out a bug in Lynx 0.9 even though there were only about ten thousand people using it."

Doug Cutting was proud of a "very cool" feature which Excite added. "It was like an automatic thesaurus. But it was really a technique from statistics called discrimination analysis. Where we found words that weren't in your query, but could help to disambiguate it, so we'd put three or four of these words at the top of the results page to help."

And what was it, did they think, that Google brought to the table?

Steve Kirsh seemed to think that, beyond technology, building the Google brand was exceptionally important. But also added that, when he'd put this to Google cofounder Larry Page some years earlier, he'd dismissed brand and put it down to operations. Just as Coke has a brand, Larry Page said that their success was due more to their operation i.e. you can get Coke anywhere in the world. And the same applies to Google, which manages to copy a multi-terabyte index around the world on a regular basis. Also noting that in the early days of Infoseek, they had trouble just trying to copy 50 gigabytes and get it over to AOL!

Doug believed that they had all missed "the big trick" with Google and that was creating the link graph and using it for ranking purposes. But he also pointed out that since Google entered the space with a clean, uncluttered white interface and a link graph, not a lot had changed since then.

And whereas AltaVista had been founded upon huge powerful machines, Louis Monier commented on the more flexible and low cost scaling method which Google uses in its hardware set up. "They've built something that scales like you wouldn't believe and it's really, really cheap."

AltaVista had also built a link table, but it was more for estimating the size of the web. "We could tell from new links discovered just how fast and big the web was growing."

Steve Kirsch couldn't resist telling the audience that they knew about this at Infoseek so they had placed a web page with the words "This is the last page on the web - there's no more after this" so the AltaVista crawler would find it and just stop there!

Finally Chris Sherman rounded up the session by asking the panel to take a look into their crystal balls and talk about the future of search.

Doug took the opportunity to talk about his new project Nutch, an open source search engine similar to Google in its operation. "It's good for niche web search and also for searching intranets and university campus when you have ten servers and you want to index them all. But we've demonstrated it on hundreds of millions of web pages and we think it will go as far as billions."

"Search is not sticky," said Louis. "That's the direction Google is going right now though. Look at how quickly they just turned Gmail on. That's the next thing for the future. Getting people to stick with you... Until two guys working in a garage come out with something new and..."

The final word went to Steve Kirsch. "I'm not qualified to predict the future really. But one thing I could certainly predict is that, the next time some Stanford student comes to me with a new idea... I may not be so quick to tell him to go pound sand again!

About the author

Mike Grehan is currently chief marketing officer and managing director at Acronym, where he is responsible for directing thought leadership programs and cross-platform marketing initiatives, as well as developing new, innovative content marketing campaigns.

Prior to joining Acronym, Grehan was group publishing director at Incisive Media, publisher of Search Engine Watch and ClickZ, and producer of the SES international conference series. Previously, he worked as a search marketing consultant with a number of international agencies handling global clients such as SAP and Motorola. Recognized as a leading search marketing expert, Grehan came online in 1995 and is the author of numerous books and white papers on the subject and is currently in the process of writing his new book From Search to Social: Marketing to the Connected Consumer to be published by Wiley later in 2014.

In March 2010 he was elected to SEMPO's board of directors and after a year as vice president he then served two years as president and is now the current chairman.