the /robots.txt file is a publicly available file. Anyone can see what sectionsof your server you don't want robots to use.

How To Use It

The "/robots.txt" file is a text file, with one or more records.Usually contains a single record looking like this:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.

Note that you need a separate "Disallow" line for every URL prefix youwant to exclude -- you cannot say "Disallow: /cgi-bin/ /tmp/" on asingle line. Also, you may not have blank lines in a record, as theyare used to delimit multiple records.

Note also that globbing and regular expression arenot supported in either the User-agent or Disallowlines. The '*' in the User-agent field is a special value meaning "anyrobot". Specifically, you cannot have lines like "User-agent: *bot*","Disallow: /tmp/*" or "Disallow: *.gif".

What you want to exclude depends on your server. Everything not explicitly disallowed is considered fairgame to retrieve. Here follow some examples:

To exclude all robots from the entire server

User-agent: *
Disallow: /

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

To exclude all robots from part of the server

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

To exclude a single robot

User-agent: BadBot
Disallow: /

To allow a single robot

User-agent: Google
Disallow:
User-agent: *
Disallow: /

To exclude all files except one

This is currently a bit awkward, as there is no "Allow" field. Theeasy way is to put all files to be disallowed into a separatedirectory, say "stuff", and leave the one file in the level abovethis directory: