Study Indicates Search Engines Fail To Keep Up With Web

July 19, 1999|By Ryan Davis, The Boston Globe.

As the World Wide Web expands at a staggering rate, search engines are failing to keep up, according to a study by two New Jersey researchers.

The researchers estimate that the Web more than doubled in size between December 1997 and February 1999, but major search engines--services that probe the Web and provide links to sites matching key words--combine to cover less than half of the available cyber information.

None of the 11 major Internet search engines covers even one-sixth of the Web's estimated 800 million publicly accessible pages, according to the study in the journal Nature by Steve Lawrence and C. Lee Giles of NEC Research Institute in Princeton, N.J. That figure is down from about one-third just 14 months earlier.

"The search engines are finding the Web very difficult to manage," Giles said.

The study, conducted in February, found NorthernLight.com to be the most expansive engine and Lycos and Euroseek the least. About 85 percent of Web browsers use search engines to locate information, the researchers wrote. The study also found that it may take months for new pages to be recognized by search engines.

Rajive Mathur, Lycos' director of search services, said his company's database is five times as large as represented in the study. Some of the growth has come since the study was completed, spokesman Brian Payea said.

"February 1999 is a long time ago in the Internet world," he said.

The researchers entered 1,050 queries and analyzed the responses of search engines. They found the return rates of the engines and compared them to Northern Light, which said it had indexed 128 million--or 16 percent--of the estimated 800 million pages.

Giles said his research indicates that "catch-all" search engines are becoming increasingly inadequate for the sophisticated Web user. Those users are looking for more specialized information--such as science studies--than Joe Internet, who is logging on for a cyber test drive.

"We don't all drive the same kind of car," Giles said. The all-inclusive search may go the way of the Model T, he said. The concept will last, he said, but there is a growing market for services that find specialized information, such as the lowest price for a book.

New pages are being added to the Web at a clip of 3 million a day, but the study found that, on average, it took more than six months for a major search engine to find new documents matching a given query.

The overlap between engines remains relatively low, and the major engines combine to cover 42 percent of the Web, the researchers said. Search sites such as MetaCrawler that combine the results of multiple engines provide a great deal more coverage, they wrote.