2/19/1999 @ 12:00AM

Searching in Dolby

Search engine web sites were some of the first to populate the Internet and have become some of the most heavily trafficked, like the hugely popular Yahoo!, for instance. Despite their seniority in cyberspace, search engines are among the least effective online tools around. Just think about how many remarkably unrelated sites your last web search returned. Given the immense task of organizing 4 million web sites consisting of 500 million web pages as well as the distraction of furiously adding features such as news and e-commerce to achieve portal supremacy, this is not surprising.

No doubt all the engines are constantly looking for ways to refine how they function. But for the major players, offering a better search function is no longer viewed as a means to increased profits. They’re focusing on how to get users to hang around their sites longer, not move them out faster. This has left the niche for new search technology wide open, and not surprisingly a second generation of “search” is upon us.

Already there are a handful of non-household name sites–Google, Ask Jeeves, Dogpile, Metacrawler–vying to offer better search results and become destination sites in their own right. Their search results are often superior to traditional engines’. The trouble is that the proposition of luring the masses away from extensively equipped, well-branded portals like Excite and Lycos, which already have 20 million or 30 million registered users, is a long shot.

However, there is one search up-and-comer that’s got a better strategy. The clearly and succinctly named firm Direct Hit has developed not only a superior technology but a much smarter business plan, putting it way ahead of the pack. Direct Hit, based in Wellesley, Mass., is selling its technology to established engines like HotBot for use in conjunction with HotBot’s search program. In return, Direct Hit gets a cut of the ad revenue on the new results page it generates, as well as a payment per search. “We like that because the more searches performed, the more money we make,” says Direct Hit CEO Michael Cassidy. “And every day on the net people are doing more and more searches.”

“Direct Hit is trying to become the Dolby of search engines.”

Presently there is no one competing with Direct Hit’s business model to sell more sophisticated search programs to the big guys, and the marketplace doesn’t offer any comparable technology. San Mateo, Calif.-based Inktomi–the creator of the search engine currently used by HotBot, Snap, and Yahoo!–is the search market leader in terms of the proportion of web searches handled by its technology, but it is this very technology that Direct Hit is building upon. Indeed, it’s possible that Direct Hit could soon replace programs like Inktomi’s entirely.

Other distinguishing characteristics include the fact that last May cofounders Cassidy and Gary Culliss won MIT’s prestigious $50,000 Entrepreneurship Competition, $30,000 of which goes to the top dog. What’s more, because Direct Hit had just received $1.4 million in backing from the venture capital firm Draper Fisher Jurvetson, the pair graciously used the awards ceremony to give their winnings to the other finalists. In December the company got $2 million more in a second round of financing from Draper Fisher and Mosaic Venture Partners.

Culliss and Cassidy’s flagship product, developed last summer, is a search technology called the Popularity Engine. Neither this nor their subsequent innovations are meant to be stand-alone products. What Direct Hit does is create search mechanisms intended to be add-ons for the traditional players. When the results of a traditional search are returned on HotBot–Direct Hit’s first customer as well as what’s generally considered to be one of the best search sites by industry experts–there’s an option to see the ten most popular sites from Direct Hit. A click takes users to another page to view them. For example, a HotBot search for California wine calls up some 19,000 listings, the first couple of which include a Mexican wine wholesaler and the site for Shug Carneros Estate Winery. Click on Direct Hit’s “most popular” button and up come a site that reports California wine news, one that lists and links California wineries alphabetically and another that features the California Wine Maker’s Guild–selections that on the face of things seem far more promising.

“The strategy they’ve chosen is not to be absorbed by a large portal,” says Gartner Group Internet analyst Ray Valdes. “They’re trying to be the Dolby of search, a brand-name enhancement to existing technology. And I think they’ve already done that to a certain extent.”

As a testament to its compelling product, Direct Hit took just one week to cut a deal with HotBot last June, and the cobranded service went live in August. In October Direct Hit launched its Popularity Engine on AOL’s ICQ community, which boasts about 25 million users, and closed a deal with Apple Computer to use its technology with Sherlock, the search component for Apple’s operating system. LookSmart, which recently replaced Inktomi as the default engine for Microsoft’s MSN, will launch Direct Hit in March.

The Popularity Engine aims at expanding the limitations of the two main types of traditional search technology–web-crawling “spider” programs and hand-compiled site directories–while optimizing what they do best. On the one hand, search engines like Excite and InfoSeek send spiders to search the entire web, reading the text on all the pages they find and bringing back those with words that match a user’s search. The problem is that because the typical search is general, like “travel” for instance, and there are millions of pages with that word, the vast number of returned pages makes the odds of their being useful slim. Yahoo! conversely is a hand-compiled directory that has about 100 editors combing the web to separate the wheat from the chaff and categorizing what’s deemed worthy. This makes Yahoo! searches generally more fruitful and that probably explains its popularity. The trouble is people are slow, salaries are expensive and at the exponential rate the web is growing, human editors are bound to get snowed under sooner or later.

Direct Hit’s Popularity Engine aims at solving these problems by having its users function essentially as unpaid editors. As users search with it, the program tracks the sites selected and monitors how much time is spent there in order to determine which sites are most relevant to that search and rank them accordingly. The more frequently a search is performed, the more refined Direct Hit’s results will be since the search will have collected still more data on which sites people found most useful.

So what’s to stop the big guys from developing this kind of program on their own? Nothing, but it’s been around for a year and no one’s managed to do so yet. Says Cassidy: “There are 12 other techniques we use to improve our algorithms that aren’t as obvious, and that’s what makes it hard to duplicate our results.” Danny Sullivan, editor of the Search Engine Watch Newsletter, concurs. “There is no one competing with Direct Hit,” he says.

Perhaps more promising is Direct Hit’s latest product announcement in January–Personalized Search. This program is the first to take into consideration age, gender, geographic location and other demographic information while it performs a search as well as to factor in a site’s popularity with users. A man searching for “flowers” might have floral delivery information returned, while a woman might pull up sites on gardening. Direct Hit says that so far it has three major search sites, which cannot be disclosed, signed on to launch Personalized Search in March and April.

“There is no one competing with Direct Hit.”

“The personalization has a lot more potential,” says the London-based Sullivan. “Here in England someone searching for football is probably looking for soccer, so the idea that a search engine can understand who and where I am is a powerful concept.”

So far so good, but Direct Hit is by no means home free. According to Cassidy, the biggest hurdle the company’s faced so far is all the Internet merger activity that’s got portals constantly changing hands; think Excite and At Home, AOL and Netscape, Infoseek and Disney, and AltaVista and Compaq. “We get close to a deal,” he says, “and then they get bought and that sets us back three months.”

Beyond Direct Hit’s struggle to get portals to pay attention, it is challenged by independent sites like Google, which have both more effective search programs than the big names and small but loyal followings. Google ranks search returns by the number of links a page has to it as well as the importance of the pages linked. A link from Yahoo! for instance will carry much more weight than one from someone’s personal home page. One possible disadvantage for Google is that product listing pages on e-commerce sites, which are growing increasingly important on the Internet, are infrequently linked to from an outside site.

Direct Hit also faces the possibility that a competitor will emerge looking to make the same kind of deals Direct Hit is courting with the portals. Indeed, IBM has a search technology research project called Clever that could pose a significant problem for the young startup. No one knows yet just how or when Big Blue intends to market it. Then there is the advent of “smart browsing” capabilities being built into both Microsoft and Netscape browsers that feed users sites related to any word typed in the URL box, regardless of whether it’s an actual Internet address. Should these new tools prove effective enough, they could put a dent in traffic for search sites of all stripes.

“Direct Hit has established a beachhead as an independent brand-name enhancement to commodity search technology, but they need to keep innovating,” says Gartner’s Valdes, “because this industry is a moving target.”