THE SEARCH ENGINE REPORT October 5, 1998 - Number 23

The Search Engine Report is a monthly newsletter that covers developments with search engines and changes to the Search Engine Watch web site, http://searchenginewatch.com/. This month's issue is sponsored by First Place Software and eGroups.com.

The report has 37,000 subscribers. You may pass this newsletter on to others, as long as it is sent in its entirety.

If you enjoy this newsletter, consider showing your support by becoming a subscriber of the Search Engine Watch web site. It doesn't cost much and provides you with some extra benefits. Details can be found at: http://searchenginewatch.com/about/subscribe.html

Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.

When a prospect does a search for a keyword related to your products or services, do you appear in the top 10 or does your competition? Submitting alone does nothing to insure good visibility. WebPosition is the first software product to monitor and analyze your search positions in the top search engines. WebPosition has been compared to similar "services" on the web and has been overwhelmingly voted the best and most accurate tool for search position management.

With WebPosition you'll know your exact positions for an unlimited number of keywords. You'll know if you drop in rank. You'll know when a search engine FINALLY indexes you. You'll know when you've been dropped from an engine. WebPosition is rated 5 out of 5 stars by ZD Net, and includes a 110 page guide to improving your search positions.

Lots of search engine news this month, all conveniently coming in the same month when I had two trips back to the US and a house move. Busy is an understatement!

I've just arrived in New York for Internet World, so once again I'm finishing off the newsletter on my laptop. I hope to have caught any typos, but if not, I beg forgiveness.

Within the site, you'll find all the search engine rating pages have been updated with August 1998 data. NetRatings and RelevantKnowledge also kindly provided data for some of the smaller services, so there's a good look at both the major players and rising contenders in the search field.

Of interest, Lycos has narrowed its gap with Infoseek or surpassed it, depending on the rating service. And while Yahoo remains well above its nearest competitors, it has been showing a slight decline for consecutive months according to all the services. There are lots of other minor ups and downs, so swing up and get yourself updated.

The MetaCrawler Top Search Terms page has also been updated with August 1998 terms, and shortly, September 1998 terms will be posted.

The Search Engine Alliances Chart has been updated, although the very latest MSN changes mentioned in this newsletter haven't been posted. That will happen shortly.

The Checking Your URL page has been updated, as have some of the Search Engine EKGs. I hope to have the remainder posted within a week, if not sooner. Time ran out before I left for this latest trip. Overall, Northern Light has shown a definite improvement, and everyone else looks about the same.

Those waiting for the Inktomi-powered Microsoft search engine got it in September: MSN Search went live in beta on Sept. 8. The addition is one of the last pieces to fall in place for Microsoft's portal site, MSN.

"This is an evolution of what our search service is going to be," said MSN product manager Nichole Hardy. "Our ultimate vision is to make MSN.com a place for people to get things done on the web."

MSN is a familiar name that is being reused for the site. Forget about MSN's previous incarnations as an online service or subscription-based web site. Both are gone. MSN Internet Access is now the name for Microsoft Internet access service, and you do not need to be a customer to use the MSN site.

MSN is now the Microsoft portal site, and even the URL will soon change to reflect the name, rather than defaulting to the current http://home.microsoft.com. Microsoft had considered called the service Start and in fact the pages were labeled Internet Start until recently. But it was decided that MSN had a Microsoft brand identity that could be tapped.

"If you asked customers what is MSN, they said that's some Microsoft thing. If you asked them what's Start, no one knew," Hardy said.

So what is there at MSN for those searching the web? Unlike its competition, MSN does not immediately exude search and navigation. It's more an umbrella for Microsoft properties. For example, links along the left side of the home page, such as "Auto" and "Travel" lead directly to Microsoft's Carpoint and Expedia sites, respectively.

Of course, there is a search box at the top of the home page, but some how this feels like just one part of the busy page, rather than a central focus.

MSN Search is just one of five other services that are rotated as default choices for this search box. The line-up here has just changed. Now departed as top choices are Yahoo, Excite and AOL NetFind. Left are Infoseek and Lycos, joined by newcomers AltaVista and Snap. Combined, these players are said to have paid US $60 million for the privilege of being listed over the next year.

Those using MSN Search and running comparisons against Inktomi-powered HotBot will notice a number of differences. The results will often not be the same. In some cases, MSN Search seems to have more listings than HotBot. Additionally, MSN Search's listings also sometimes seem to be fresher.

Most of this is due to the fact that Inktomi is now running four separate indexes to serve its various partners.

"As opposed to having everyone run on one gigantic index, we have different ones and partners assigned to them," said Troy Toman, Inktomi's director of search services.

Yahoo continues to use the index created in summer, and that database remains slightly smaller than the main index Inktomi started with. MSN Search is using its own index, as is HotBot. The HotBot index is slightly older and smaller than the MSN Search database, and thus differences are appearing.

The situation with the HotBot index is to be corrected soon, and in fact, Inktomi intends to eventually balance requests from all its partners across the four indexes, rather than the more simplistic means of load balancing by giving each major partner its own index, Toman said.

However, expect that differences may continue even when this is done. As I've written before, Inktomi intends to develop additional ways for partners to distinguish themselves, even when drawing from the same core listings.

One simple but important change users will notice is that MSN Search defaults to 20 results, rather than the standard 10. I think this is better in that it gives users more options at a glance, especially as relatively few people tend to click for additional results. WebCrawler, GoTo and Northern Light also default to beyond the standard top ten list.

MSN also features a directory, which can be easy to overlook. Find it by selecting the "Web Directories" option in a reverse bar at the top of the screen. Then look to the left of the screen, in the section called "Best of the Web." Select a category, and you'll be into the directory.

Microsoft lists about 80,000 web sites in its directory. About 50 people are involved with compiling it, and there are plans to continue growing it over time.

There are also plans to integrate the directory listings into the new MSN Search results, as so many other search services have been doing. Yahoo results may also be mixed in. At the moment, the MSN-Yahoo partnership extends to a prompt saying "try your search on Yahoo," to the left of MSN Search results.

"We know that the directory listing are very popular to those on the web, Hardy said. "That is something that we want to pull together."

MSN Search is also making an impact relating to the Internet Explorer browser. Those using IE3 are directed to a special search page when using the search button, while IE4 users have a search pane appear when using their search button.

In both cases, the default search provider line up has changed as with the MSN home page, except that GoTo is featured instead of Snap. This is also the same page that appears if the "Search Page" option on the MSN home page is selected.

Also expect that Microsoft's version of Smart Browsing, where terms entered into the address box bring back search engine results, will eventually default to MSN search results. However, there are no current plans to program particular keywords, as Netscape has done, Hardy said.

There will be more changes at MSN over the coming month. The curious will want to monitor the beta site, which is running in tandem with the regular MSN site.

Users of HotBot will find a new emphasis on the human. Changes have been made to integrate material generated by human editors into the service.

Hotbot has moved its search box from center stage, with new channels being featured more prominently. The normal favorites are here, such as news, sports and travel. However, rather than an alphabetical listing, HotBot has organized them into common interest areas.

"We felt it was important to bring people a channel focus," said spokesperson Andrew de Vries. "For broader topics, those popular on the web, people need something other than just a search engine."

Content within the channels varies. The News area features headlines from Wired News and is now the new home to the former NewsBot search engine. In Sports, scores are provided through a partnership with MSNBC, while recommended web sites are also prominently listed.

There are similar specialty search and information databases in the other channels, along with links to prominent sites. This directory has been hand-built by Wired Digital editors, and it aims to list only the very best of the web.

The partnership with LookSmart remains, and it serves to provide a directory of sites beyond the top picks listed within the channels. These appear in a small box to the right of the recommended sites, called appropriately, More Sites.

Despite displaying channels prominently on its home page, many users will continue gravitating toward the search box. In response, HotBot integrates channel content into search results, in response to popular or broad search terms.

For example, "stocks" causes the search results page to be topped by an option to search for stock prices. "Travel" brings up links to Lonely Planet travel guides, and "news" leads with two news search options -- though oddly, not the HotBot news search engine.

Excite has been doing this type of search integration since May and calls it "channelized search." HotBot's term is "Smart Results." Regardless of the name, the moves are beneficial for many search engine users who are often after information, not matching web pages.

The human element is also integrated into HotBot's search results through its partnership with Direct Hit. That service creates top ten lists based on what users are clicking on from the "normal" HotBot results.

Direct Hit is easier to use now that HotBot has made the feature more prominent. Just do a search, such as for "travel," and right above the matching web pages will be an image link that says "Top 10 Most Visited Sites for Travel." Click on the link, and you'll see the Direct Hit list of sites.

HotBot has also made a number of changes to its search engine results. It now follows Infoseek's lead in clustering, which means listing only one page per web site in the top results. This prevents any one site from dominating the results, and the greater representation of sites it creates is beneficial to users.

Unlike Infoseek, this feature cannot be turned off. However, clicking on the "see results from this site only" link below each listing will display all matching pages from that site.

HotBot has also rolled out a number of new advanced search options. Stemming is now supported. When switched on, a search for something like "think" will also bring up pages with "thinking" and even "thought."

There's also the ability to do a search for personal pages, such as those commonly hosted on free web sites such as AOL or GeoCities. HotBot does this by classifying sites by domain or looking for common URL features of personal pages, such as the presence of a tilde or the word "member" in the URL.

Both options can be found on the SuperSearch page, reachable from the home page by clicking on the "More Search Options" button just below the normal Search button. The personal page option is in the Page Depth section of the SuperSearch page.

A new search option also exists right on the home page: Search By Language. Now results can be narrowed to pages written in one of nine different languages.

Coming soon will be a porn control, where users can control the degree they want to keep hard core sex sites and other possibly objectionable material out of their results.

One frustrating thing some readers have complained about to me is connectivity problems with the site. Searches can get cutoff in mid-stream or pages may not load properly, both situations I have seen happen many times recently. HotBot said that these may be related to the redesign, and that hopefully they'll be resolved shortly.

There have also been some problems with the database size and freshness, and especially with adding pages, due to Inktomi's expansion. These issues are covered in the crawling article below, and the MSN article above.

Finally, HotBot has quietly added the portal features of free email and home pages, in partnership with Lycos-owned WhoWhere. A television ad campaign will also be underway shortly.

Infoseek made a number of changes to the service in September, most noticeably to its search results page. Up to three categories of results are now presented: Recommended, Reviewed Web Site Topics, and Web Search Results.

Recommended listings only appear in response to general or popular queries, such as "movies," "sports" or "travel." Links here will take you to the most appropriate channels within Infoseek where related material has been aggregated, such as news headlines, sports standings, TV listings and selected web sites. There are also links to related chat areas and other appropriate information.

Web Site Topics displays links to appropriate areas of Infoseek's directory. These are sites that have been reviewed by editors, rated and categorized into various topics. Finally, Web Search Results comes last. These are matches from Infoseek's search engine listings.

Infoseek has been promoting its directory listings above its search engine results for several months, but this latest change makes the directory results much more prominent, and thus more likely that users will be detoured into them.

This will benefit many users, since the directory results will lead them into more specific topics than they may have initially searched for. The Infoseek directory is also of good quality.

On the Infoseek home page, the results are less dramatic. There are new colors, of course. But the main change has been to return the search box to a more prominent position.

One of the chief complaints that many webmasters have about Yahoo is the inability to get listed. Right behind this is the inability to know if they have even been considered for listing within the web's most important search service.

Yahoo receives far more submissions that its editors can review, so just being considered is a challenge. There is no guarantee that an editor will even look at your site.

Consequently, many webmasters feel their submissions are in vain. "Black hole" was a term used by some to describe the submission process, when I did a special Yahoo survey last year.

One solution to the problem is to charge for consideration, something that Yahoo has long thought about. Now it has taken an important step forward with a survey on the idea.

"This has been something that's always been on the table," said Yahoo's Srinija Srinivasan, who oversees the listings process. "People have often told us that I'd love to pay for this service, if I only got a guarantee that you would look at me."

Yahoo has previously rejected this idea, because it's easy for its users and site owners to confuse paying for review with paying to be listed. But as the web matures, the possible advantages of such a service are becoming stronger.

"We think it might be a mutual beneficial all around, that we allow some avenue for people who use the Internet for their business to make sure that their site is reviewed," Srinivasan said. "Revenue from that review can help subsidize the large free submission service."

Yahoo has no concrete plans, an essential part of any service would be that Yahoo would guarantee to review sites within a certain time frame, and site owners would get a definite yes or no about being listed, Srinivasan said.

A formal rejection notice is important, because hardly any sites receive them now. Sending them inevitably opens a dialog where Yahoo has to defend its decision, which in turn consumes time that could be spent building listings. To avoid this, few formal rejections go out. That leaves some site owners in limbo, and they waste time resubmitting.

Of course, the more common reason that a site is not listed is simply because it is never reviewed. With a dedicated staff funded by submission fees, these sites will be guaranteed a look, and it is likely more of them may actually get in.

However, Yahoo is very clear that guaranteed review will not mean guaranteed listing. A site will have to meet Yahoo's existing standards if it is to be listed, regardless of any fee paid.

"The best way to think of it is what we do now is not going anywhere, and this would just be one additional avenue," Srinivasan said.

It is likely that sites that don't pay for consideration may also benefit under such as system. I have absolutely no doubt that many site owners would pay a reasonable fee to be considered. This would take them out of the "free" system, opening up time for editors there to review more free sites. That should give them more opportunity to actually be listed.

Yahoo doesn't yet have the results of the survey, as it just came down on Friday. Expect details as they are released, along with news should plans develop further.

Last issue, I wrote about several software packages that allow metasearching from the desktop. Now Infoseek has released its own metasearch software, a first by any of the major search services.

AltaVista does provide its own Discovery software, but that is designed to work exclusively with the AltaVista search engine. In contrast, Infoseek Express pulls results in from all of the major services in addition to Infoseek.

Express also provides a variety of specialty search services, such as newsgroups, street maps and books. But it is the web searching that will probably attract most people.

The software is completely integrated into your browser. Once installed, a small Express drop down button will appear next to your address box. Selecting "Go To Express" will cause the program to load within your browser, set for web searching. Simply enter your search terms, hit the search button, and Express will pull back results.

The package runs in two modes. In Speed mode, it lists results in order of how fast they are returned by the various services. Alternatively, Relevancy mode takes more time, but it causes the program to list pages that appear in multiple search engines first -- a very nice touch.

In either mode, Infoseek's results are not given a boost over the others. They do seem to appear higher when running in Speed mode, but I suspect this is due to the program retrieving results faster from Infoseek than the other services.

Another nicety is that search terms are highlighted in the search results. Express even has a dedicated highlight mode, where you can enter a URL, some search terms, then have the page displayed with all the search terms highlighted.

Express instructs users to enter search terms separated by commas, so if you were looking for low price computers, you'd enter, "low price, computer." However, searches can be done perfectly fine without using commas, and I would suggest you enter terms exactly as you normally would into a web-based service.

Like other metasearch packages, Express also allows you to download pages from the results for offline viewing, and it will automatically highlight search terms when pages are viewed this way.

The package is free, with Infoseek hoping to make money through ads that it sells. These appear in the upper right hand corner of the Express window, within the browser.

Sensitive to online ad revenue, Infoseek also displays banner ads from its service and those of competing search services in the search results, with a random rotation determining whose ad appears at the top of the page. It's an uncommon degree of fairness for the web these days, but one that makes sense, as Infoseek doesn't want its competitors complaining that the tool plays favorites.

Of course, if you hate ads entirely, there are alternative packages that work similarly to Express. These are described via the link below, but you'll pay a fee to be ad free.

Express has been two years in the making, and it clearly comes across as a project where a lot of time and care have been involved. There are all sorts of options for customizing it that I've not covered, and those who like metasearch software will no doubt enjoy exploring the software's full potential.

Express is available for Window 95/98/NT, and it will work with Netscape or Explorer versions three and higher. However, you must be running a screen resolution of 800 x 600 or better. Infoseek is considering Unix and Mac versions but it is waiting to see how popular the tool is with Windows users first.

For marketers, Express also offers branded versions as part of its distribution effort. In about 10 minutes, you can create a version with your site logo and selected links in a category you create.

Infoseek and Disney put a name to the site they plan to launch later this year: Go. The site will encompass material from Disney, Infoseek and Starwave web properties, such as ESPN.com, ABCNews.com. It's expected to launch in November.

Go is not meant to be a replacement for Infoseek, nor will Infoseek disappear. Portal-like activity will focus on Go, while Infoseek will continue as a search and navigation site.

"We could take Infoseek and reposition it as Go," said Infoseek president Harry Motro. But doing so would weaken a strong brand identified with search by forcing it into a role better served by the new site, he explained.

It seemed everyone had copies of the Starr Report posted online last month, which was great in terms of accessibility. But what really impressed me was the ability to search through portions of President Clinton's video testimony at AltaVista.

For example, I wanted to review Clinton's reaction when the infamous "cigar" question was raised. I went to the video search service, entered "cigar" into the search box, and up came the appropriate video clip. Within seconds, I was viewing exactly the segment I had wanted to see. It was very cool indeed.

The service was made possible through software from a company called Virage, and how it was produced in this case was pretty straightforward. The company captured the video testimony and its closed-captions when it was broadcast on CSPAN, the cable network that covers US politics.

The closed-caption text was converted into HTML files, which in turn were associated with 158 video clips. AltaVista then used its search engine technology to index the HTML files, allowing users to find specific text and then view the associated video clip.

"What you saw was very easy for us, because it was closed captioned video," said Dave Girouard, director of product marketing at Virage.

Closed caption makes things easy because someone has already transcribed the video tape. Things aren't so easy when closed-captioning isn't available, but Virage has ways around this. It can turn to TelePrompTer text or scripts. It has also announced a partnership with IBM for speech recognition, for times when no written record is readily available.

The Holy Grail of video search is the ability to truly search for images irrespective of having associated text. For example, at a conference I was at earlier this year, a spokesperson from Excalibur, another video search company, talked about the challenge of finding that clip where President Clinton hugs Monica Lewinsky in a crowd.

At the moment, search companies can do some basic face recognition, pattern recognition and recognition of when scenes change. But a video search engine to find a scene as described above is many years off.

In the meantime, the Clinton video search at AltaVista shows how useful indexing can be when coupled with written material of some form. And Girouard says to expect to see more of it on the web, given the popular reception the Clinton video search has received. Over half a million people have used the service, he said.

Off the web, Girouard says that video search is extremely popular for Intranets and companies with video libraries that need indexing.

Excite's new communities service is now accessible to anyone, having opened to the general public on Sept. 16. It launched as a beta service, at the same time that Yahoo debuted its own communities service. At both places, users can create mini-sites within the portals that offer special features such as discussion areas. The article below provides more details.

Infoseek is taking over from Excite as the default search provider on Microsoft's WebTV service in October. Infoseek will also handle ad sales for the service. Infoseek is paying a guaranteed US $26 million for the two year agreement, which was announced on Sept. 10. Excite had been WebTV's default choice for the past two years.

A previous look at why these Internet access deals might be significant for the various services.

===================

Northern Light Opens News Search Service

Northern Light has introduced a news search service called Current News. Information in the index is gathered from various news sources, such as the Associated Press and Business Wire. Content is constantly refreshed throughout the day. Options include the ability to sort news by date or relevance, and to narrow searches to within predefined categories and timespans.

Inktomi has agreed to purchase C2B Technologies, which develops comparison shopping software, for a $90 million stock swap. Inktomi plans to use the acquired technology to create a shopping search service that it can offer to its partners. The announcement came on Sept. 1.

No surprises here to readers of this newsletter. The article covers, briefly, trends like GoTo's placement selling and AltaVista's partnership with RealNames. There's also brief mention of meta tag abuse.

Last of a four-part series where the authors tested online store creation services iCat and Yahoo Store. Of interest is that Yahoo Store's biggest support question isn't related to the store software but the inability for its merchants to get listed in the Yahoo directory. At least Yahoo's not playing favorites.

Portal has been so overused, and misused, that those that run portals are trying to get away from the word. An excellent article on current attitudes, with some fun facts at the end. Subscription required.