What is a WWW robot?

A robot is a program that automatically traverses the
Web's hypertext structure by retrieving a document, and
recursively retrieving all documents that are
referenced.

Note that "recursive" here doesn't limit the definition
to any specific traversal algorithm; even if a robot
applies some heuristic to the selection and order of
documents to visit and spaces out requests over a long
space of time, it is still a robot.

Normal Web browsers are not robots, because they are
operated by a human, and don't automatically retrieve
referenced documents (other than inline images).

Web robots are sometimes referred to as Web Wanderers,
Web Crawlers, or Spiders. These names are a bit misleading
as they give the impression the software itself moves
between sites like a virus; this not the case, a robot
simply visits sites by requesting documents from them.