Did you know that there's a longer, more in-depth version of this newsletter? The twice-monthly "Search Engine Update" newsletter is just one of the many benefits available to Search Engine Watch "site subscribers." Learn more about the advantages to becoming a site subscriber at this page:http://searchenginewatch.com/about/subscribe.html?source=ser0008

Please note that long URLs may break into two lines in some mail readers. Cut and paste, should this occur.

I've posted new statistics from StatMarket and Nielsen//NetRatings regarding the popularity of different search engines. StatMarket shows that the amount of search engine-related traffic sent to web sites by AltaVista has increased significantly over the past six months. The latest figures from Nielsen//NetRatings show that among the most highly used search engines, Yahoo has maintained its usual lead while its major competitors -- except for MSN -- have lost some ground. Among the smaller players, Google's audience continues to grow while search newcomer iWon is firmly established as a competitor against some older search services. Links to both pages can be found via the What's New page, below.

The next Search Engine Strategies conference is less than two weeks away!

Coming to San Francisco on August 14, I'll be presenting and moderating sessions at the conference that feature experts on search engine marketing issues and panelists from the various major search engines themselves. Search engines participating in the panels include About.com, AltaVista, Ask Jeeves, Excite, Google, Inktomi, LookSmart, mySimon, NBC Internet/Snap, Netscape/The Open Directory and Yahoo. There will be a special session on shopping search, which should be of interest to any online retailers. Details about the conference, for attendees or potential sponsors and exhibitors, can be found via the URL below.

After months of beta testing, LookSmart has become the first major search service to charge commercial sites to be reviewed for inclusion in its listings. In addition, the revenue-generating move appears to have helped LookSmart win back its former position as AltaVista's directory provider. It has also strengthened LookSmart's ability to compete as a provider against the Open Directory, which allows any search site to use its human-compiled listings for free.

LookSmart's "Express Submit" service does not guarantee commercial sites a listing or better rankings. This isn't pay for placement, as at GoTo.com, where a site can pay more to appear higher in the search results. Nor is it pay for inclusion, where a site would only be included in listings if it paid a fee. Rather, this is pay for consideration. The payment levied against commercial sites ensures that they will be considered for inclusion into the index in a timely manner, but they should ultimately only be listed if they also meet editorial criteria.

In LookSmart's system, commercial sites can pay US $199 to be listed within the directory, assuming they meet acceptance criteria. The vast majority of sites will meet these, so spending the fee isn't much of a gamble.

"The percentage of rejection is pretty low, but we do hold them to the same standards," said Kristin Morse, LookSmart's director of ecommerce.

The $199 option has been available since January, but a new $49 option has just been added. The only difference is the response time. Those paying $49 may have to wait up to eight weeks for their sites to appear, assuming the sites meet acceptance standards.

The lower price point is a compromise between LookSmart wanting to accommodate sites with little money and the problem of commercial sites continuing to use the free submit option, when it was available to them alongside the pay submit option. By requiring all commercial sites to use the pay option, this should make it easier for non-commercial sites to be listed.

I certainly hope that this proves to be the case. If so, it should benefit both searchers and webmasters, because some non-commercial sites that were lost in the blizzard of commercial submissions may now get the attention they deserve.

LookSmart doesn't guarantee how long it will take non-commercial sites using the free submit option to appear. However, Morse said that any qualified site really should show up within eight weeks. I'd rather hear that this will happen within a week or less, which would show that removing the commercial stream of sites really will benefit the non-profits.

By the way, non-commercial means that your organization is formally registered with the US government to have not-for-profit status. Unfortunately, even if your site is a labor of love that earns you no money, you cannot use free submit unless you are registered.

Do the LookSmart submit changes mean that some deserving commercial sites may go unlisted, because they cannot afford to pay or refuse to pay. Possibly, but not necessarily. LookSmart strongly emphasizes that the submission system is only one way editors find potential listings. Editors are also watching various resources across the web, looking out for new web sites to add independently of the submission service.

"We are keeping intact our usual editorial policy of finding the best of the web," said Scott Stanford, LookSmart's vice president of ecommerce and distribution. "If luck favors you, your site will be found, as soon as our editors get to it.

One serious concern is that LookSmart's standards might drop in order to allow more sites in and thus ultimately make more money. LookSmart has previously said it has no intention of letting standards slip and that listings would continue to be handled by the editorial staff, rather than becoming an advertising option.

LookSmart's search partners will share in the wealth the new submission system will produce. MSN Search, Excite, iWon and AltaVista now or soon will have options allowing people to submit directly to LookSmart and thus appear within the LookSmart-powered portions of their web sites. LookSmart said these partners will receive some of the submission fees, in relation to the submissions they generate.

iWon and AltaVista are recent partner wins by LookSmart, with AltaVista being most notable. LookSmart had provided some of AltaVista's results from January 1998 through October 1999, when the Open Directory took over as the primary directory provider. The attraction of the Open Directory is that it doesn't charge for its listings, while LookSmart was engaging in licensing deals or ad share revenues for those who wished its services. Because of this, it seemed as if LookSmart couldn't continue to attract new partners. After all, the logic went, who would pay for LookSmart's listings when they could get listings from the Open Directory for free?

Now the submission revenue share option flips that argument around 180 degrees. Why go with the Open Directory, when going with LookSmart might make you money? This isn't the only reason future partners might choose LookSmart, but it is an important factor. The News.com article below touches on the revenues AltaVista expects to make. LookSmart takes over from the Open Directory as AltaVista's exclusive directory partner later this month.

Another factor here is LookSmart's recent redesign, made at the end of June. The company recently deemphasized features such as free email, news headlines and horoscopes, designed to help it compete as a portal, and the directory structure has been left in place primarily as a demonstration site for potential partners such as iWon or AltaVista. In many ways, it reflects LookSmart's goal to be a behind-the-scenes directory provider, similar to how Inktomi is a behind-the-scenes crawler-based search engine provider.

"The message for our partners is that we are not in the business to compete with them but rather to support them," said spokesperson Liz Connaghan.

In another challenge to the Open Directory, Wherewithal has gained a former Open Directory editor to help it in its quest to go further by paying editors to list sites. More details from the company's press release.

More details about Wherewithal and a link to an article from the former Open Directory editor that they've hired.

====================

iWon Expands Search Offerings

Over the past three months, iWon has been adding new search resources to its site, which offers users the chance to win money as they search the web or undertake other portal activities, such as checking email or stock quotes. It's a move away from the less-is-more trend that you see at places such as Google, AltaVista's Raging Search and recently revamped Excite search engine. However, iWon feels providing a variety of results serves its users better.

"A lot of people think simple is better and web pages [only” are better," said George Nimeh, iWon's vice president of search and special projects. "But for the time being, we are trying to surface multiple data in a clean and uncrowded way.

Previously, search at iWon brought up information primarily from Inktomi's crawler-based index of web pages. Now information from several search providers, such as LookSmart, Direct Hit and Moreover, is being presented.

Inktomi's crawler-based index remains the power behind the main portion of iWon's listings. Do a search from the home page, and it is information from Inktomi's index that appears in the "Matching Web Sites" section that makes up the bulk of the iWon results page. iWon is also now making use of Inktomi's new 500 million page index, so you should consider it among your choices when seeking unusual or obscure information.

Previously, Inktomi also provided iWon with categorized results. Now, LookSmart fulfills this function. At the top of the results page is a "Related Web Directory Categories" section. The categories listed here come from LookSmart, and clicking on a category link displays an editor-compiled list of web sites related to your search topic.

On the far right hand side of the results page is a "Most Popular" result section. This provides access to listings from the Direct Hit service, which shows what web pages are most popular based on what people click on in search results (see article below, for more about Direct Hit). For any search, suggested topics will appear. Selecting a topic brings up web page listings for that subject. For example, if you search for "science," the Most Popular section will suggest "science projects" as one of several popular results categories. In turn, clicking on that category up Direct Hit's actual listings related to science projects.

At the bottom of the results page is the "Matching News Articles" section. These are matching stories from across the web, as found by the Moreover news search engine (see article below for more about Moreover).

There's also an "iWon Search" box that appears on the right-hand side of the results page that lets you do a specific searches. "Full search" runs your query against all the providers already mentioned. "Top 10 web sites" just runs the search against Direct Hit results. "News articles" lets you search Moreover's news listings. "MP3/audio" is a new option, which allows you to search for multimedia content. Those listings are provided by Inktomi, based on its crawls of the web.

RealNames is also present at the service. On the results page, you'll see an "Internet keywords" link. Clicking on that will take you either directly to a site that has an exact match in RealNames listings or to a list of sites that have RealNames which contain your search term.

iWon has also just announced a new partnership with the Fact City facts search engine will also be added. Information from Fact City will be available at iWon later this year.

If you use the news search option at most search engines, you'll generally get shown matching stories that come from newswires such as the Associated Press and Reuters. That's OK, but the coverage isn't very diverse. Specialized "news search engines" such as Excite's NewsTracker or News Index do a better job. They crawl a wide variety of news sites, such as online newspapers and magazines, in addition to providing wire stories.

Moreover and Yahoo go a step beyond this, because they pick up headlines from subject-oriented sites that even news search engines may miss. Those interested in news should try either Moreover or Yahoo, because of the breadth of coverage they offer. They also provide webmasters with an opportunity to put forward their own newsworthy content. See the article below for more about these services.

In the wake of LookSmart's new pay-for-consideration system, Inktomi has just announced that it will offer a pay for inclusion service later this year. The program will initially launch in conjunction with Network Solutions, but it will likely be expanded to allow Inktomi's search engine partners and perhaps selected submission services to participate, according to Inktomi.

The service will ensure that web site owners can find their home page listed in Inktomi-powered search engines with 48 hours after submitting, and Inktomi will revisit the home page every 48 hours to check for changes. The service will not give the pages any ranking boost.

"This is just about getting into the index. It's still up to the site owners to make sure their content is relevant," said Troy Toman, general manager of Inktomi's search division. "There's no preferential treatment," Toman said.

A free submit option will be retained, but there are no guarantees on how quickly sites will be added or if they will be retained in the index. Inktomi's indexing system may drop pages if they don't appear to be satisfying search queries. The paid option would help webmasters overcome this by keeping their pages listed permanently. It would also offset Inktomi's cost of retaining pages that might seem to have no particular search value.

The main issue here is that any good search engine ought to be listing the home pages of web sites in a timely manner, period. The paid submit system will help site owners know that they can quickly get listed, but searchers might be penalized if Inktomi takes longer to add new sites, in order to boost the value of its paid submit system. Inktomi has not said this would happen.

A more useful system might be one where sites that pay may get spidered to a greater depth. That potentially gives them the opportunity to attract more visitors, because each page is a potential entry point for different topics. This could add value to a paid submit system without penalizing sites that don't pay. Inktomi says that the system could be expanded this way, after it launches.

Overall, it's a difficult balance that has to be struck. There's a real plus to site owners knowing that they'll quickly be added, but a search engine must also be inclusive of all important sites, lest it begin to lack comprehensiveness. However, a paid submit system might reduce the incredibly serious problem of search engine spamming. The free submit systems that most crawler-based search engines operate leave them open to indexing pages with little value and which take away from the good content they'd prefer to add. Some type of paid system might ease the massive pressure on free submit and actually allow non-paying sites better representation. Whether this will happen remains to be seen.

The program is expected to launch in September or October, probably through the Network Solutions site. Pricing has not yet been set, and it's likely the program details could also change from what was announced today.

Ask Jeeves has redesigned its home page and search results page. From the home page, you'll find category links just below the search box. These lead to the Ask Jeeves version of the Open Directory, where sites are ranked in order of popularity. That popularity is as determined by Ask Jeeves-owned Direct Hit, which uses clickthrough measurements to determine what are the most popular pages on the web.

Of course, you can still do keyword searching at Ask Jeeves. After you do a search, the results page has several key sections. At the top are the "normal" Ask Jeeves results, which come from their editors seeking the best sites that match a query. They come under a section that says, "I have found answers to the following questions."

After that, the next section says, "People with similar questions have found these sites relevant." The matches here come from the Direct Hit index of popular pages.

At the bottom of the page, the section that begins "I have also found these sites through other search engines" are meta search results that come from other major search engines.

The folks at Google have been busy, adding a new advanced search page, a preferences page, dictionary links and a Korean-language version.

The new advanced search page allows to perform some searches without knowing any search commands. For example, you can do a phrase search or search for pages just within a particular domain, all by filling out a form. You can also specify that you want pages in a particular language and increase the number of results returned per page, up to 100. Links to Google's specialty search engines can also be found here, such as its Linux or US Government search engines.

The new preferences page lets Google remember your favorite settings. For example, you can make it always display 100 results per page when you come to the site, rather than this being a one-off choice that the advanced search page allows. Language preferences can be set, and you are finally able to set Google's "SafeSearch" porn filter to work before you do a search.

Google now also provides dictionary definitions, served out of Dictionary.com. Do a search, then look at the status message that appears above the search results. What's the status message? That's my made-up word for the line that tells you how many pages were found for a search. For example, the status message for a search on "onomatopoeia" says this:

Google results 1-10 of about 6,680 for onomatopoeia. Search took 0.05 seconds.

In the status message, the search word will usually be an underlined clickable link -- in this case, "onomatopoeia." Selecting it takes you to Dictionary.com, where the word's meaning will be displayed. You can also switch to use the thesaurus option, once there.

I love this new addition, because it's a fast way to get word meanings. The main flaw is that if you don't know how to spell the word, you won't get the definition, much less the correct spelling.

Finally, Google is beta testing a Korean-language version of its service.

AltaVista has once again undertaken another major redesign of its service, the fourth in less than a year. AltaVista says the latest changes are primarily aimed at making it easier for users to navigate within the service and then outwards to what they are looking for. While AltaVista's heart is in the right place, I don't feel this latest attempt makes life any easier. If anything, there's an overload of information. Links, tabs and color boxes seem to be everywhere, competing for attention.

Rather than tell you how to find the new and interesting stuff among the clutter, I'll just highlight the important points and provide direct links at the end of this article.

+ Perform a search from the home page, and you'll discover that AltaVista now highlights your search words with either italics or bold print, where they appear in the web page listings.

+ Use the "Find Results In" box that appears near the top of the results page to send your query to one of several specialty search engines that AltaVista operates. For example, a search for "toshiba portege" could be sent to AltaVista shopping search engine by clicking on the "Products" link or against human-compiled listings by clicking on the "Directories" link.

+ A new Power Search page allows you to finally make use of AltaVista's powerful commands without having to know any special syntax. A nice touch is a pop-up box that displays what domains belong to particular countries, which can be helpful when trying to narrow down searches. Most people will be far better served by this page than the "Advanced Search" page, which remains offered.

+ A new Search Guide area is designed to provide examples of searching in different topics areas, such as the Summer Olympics.

+ A new Entertainment search area is designed to allow you to search and get matching image files, audio and MP3 files, video clips and web pages and radio stations, all on the same page. AltaVista has also made changes that it says should provide more relevant entertainment results. They've done this by generally defaulting the searches to bring back pages from sites they've automatically identified as being related to entertainment, rather than from the web as a whole. This only works when you are within the entertainment area itself.

+ Radio stations are a complete new media type that can be found, but the functionality sounds better than it seems. Internet or terrestrial radio sites listed in response to a query may not actually be playing the song or artist you searched for, even though they are listed. Instead, they may appear simply because they've defined themselves as somehow relevant to that artist or song.

+ A great new feature is the "Search Trends" area, also called AltaVista A-List. Here, AltaVista is highlighting what we are searching for. Lycos has been doing something similar for over a year, and it's always fun reading (yesterday's headline -- Pokemon finally slips from the number one search spot after 30 weeks running). I'm glad to see AltaVista also offering a glimpse into what we search for and how we do it. Hopefully, they'll also add an archive of past postings.

I've written before about the "invisible web," information that search engines cannot or refuse to index because it is locked up within databases. Now a new survey has made an attempt to measure how much information exists outside of the search engines' reach. The company behind the survey is also offering up a solution for those who want tap into this "hidden" material.

The study, conducted by search company BrightPlanet, estimates that the inaccessible part of the web is about 500 times larger than what search engines already provide access to. To put that another way, Google currently claims to have indexed or know about 1 billion web pages, making it the largest crawler-based search engine, based on reported numbers. Using Google as a benchmark, that means BrightPlanet would estimate there are about 500 billion pages of information available on the web, and only 1/500 of that information can be reached via traditional search engines.

Back in April, I wrote an article about an expected growth in vertical search engines or "vortals," as they are sometimes called. This was due to the emergence of new services that I feel make it cheaper and easier for sites to create their own crawler-based search engines that focus on particular topics. Now another group of new services are emerging, those that offer to automatically categorize web pages into Yahoo-like directories. For example, point these services at a set of pages from different sites about cars, and they'll attempt to automatically group the pages into categories that make sense. As a result, we may see more vertical directories, or "vectories," as I'll dub them, also appearing across the web in the near future.

In general, the vectory products aren't magic. You typically have to provide "training documents" for the major categories you wish to populate. You give the software some good examples of pages about cars and trucks, and it will then use them to group other pages into either a cars or trucks category. Human editors may also be involved to create keyword definitions for some categories and to filter out obviously incorrect matches.

Mohomine is aimed at both the vertical portal market and those needing Intranet search solutions. It can create vectories for consumers or organize a company's internal documents into categories. Mohomine also has document discovery capabilities. Show it examples of documents you like, and it can mine the entire web, if you desire, to recover similar matches. Mohomine can classify more than web pages. It can extract textual information from objects such as computer code, allowing such items to be classified alongside HTML documents. The company launched services in March.

Twirlix offers both a crawler-based search engine and an automated directory to its clients. Sites can be ranked by using criteria such as link analysis, the size of a site and freshness of content to determine relevancy. Thumbnail images of each page can also be presented. The company launched its US services in May and has been open in the German market since last November (http://www.twirlix.de/). The Twirlix US and German home pages serve both as a technology demo to potential clients and as a web search site for consumers.

Founded by two students from Carnegie Mellon University, the birthplace of Lycos, Hubat offers automated directory building. The site serves as a demonstration of the company's technology and as a resource for consumers to search the web.

RuleSpace offers the ability to categorize documents for purposes of searching. However, the product really seems positioned as a classification tool for networks. For example, you might use RuleSpace to identify and prevent porn documents from being accessed on corporate networks or to filter out certain types of documents from a directory. The company has been operating since 1996. It lists AOL and Amazon's Alexa as companies using its technology to help categorize information.

Inktomi has offered its "Directory Engine" since July 1999. The product that automatically classifies web sites into categories and makes use of link and caching data to help determine which documents are most popular. The company also recently acquired Ultraseek, which has categorization technology designed for use on Intranets.

Yes, it looks like new top level domain names may finally be coming -- but what they will be, how they will be assigned, and whether they will actually be used remains undecided. So, don't panic. The article below provides a rundown on the plan to accept proposals for new names. ICANN expects to provide application details on August 1, with the deadline for proposals on October 1. Negotiations with the first round of finalists are to be concluded by the end of this year, which means new top level domain names might be available at the beginning of next year.

The success rate of finding information using major search engines has hit its highest point in two years, according to figures from the last NPD Search and Portal Site Study to have been released. A new version of the study was also just completed, so I expect to update this page again sometime in August. Google and iWon have already issued press releases detailing their successes in this latest survey, and links to these are below.

A recent study by Forrester Research found that search engines remain the leading way users in the United Kingdom find web sites. The "UK Internet User Monitor" survey found 81 percent of users said that search engines helped them find the web sites they use, up from 67 percent in 1999. The next most popular source was by following links, a method used by 59 percent of those surveyed. More details can be found at the URL below.

Google and AltaVista aren't the only ones with new advanced search pages. FAST also added one in July. It helps you perform custom queries, enable a porn filter, see more results on one page and more. The new FAST Search Customize page lets you make some settings permanent.

Those using Internet Explorer can now quickly add a new toolbar that gives direct search access to Excite, plus a button to brings up links to Excite's references tools, such as an online dictionary and encyclopedia. The toolbar also gives access to other features of the Excite portal.

A poll for search engine WebTop.com, conducted by research firm MORI, found that only 18 percent of those in Great Britain say they find what they are looking for on the web. Also, 67 percent said they are frustrated when searching for information. The survey was conducted in June, among 600 Internet users.

Auction meta search site Bidder's Edge gets 28 law professors to file a friend-of-the-court brief on its behalf, saying that the temporary ban against spidering eBay threatens search engines and linking in general.

Earlier this year, I wrote about the many problems with using online rating services to determine which are the most popular search engines. From the Standard, here's another comprehensive look at how ratings can be misstated, manipulated and ultimately be untrustworthy.

The U.K. Supreme Court has granted permission in part for Google to appeal against a ruling relating to a dispute over the user information through cookies via use of the Apple Safari browser.
0 Comments