Use trackSpiders.cgi to track spiders from major search engines (Google, Yahoo, MSN).
This free web analytics software will examine an Apache web server log
and will display visits from Googlebot, Yahoo Slurp, and MSNBot.
It takes an optional query string in the format s=ygm where each
letter corresponds to the search engine spider for the major search engines.
For example, specifying s=g will only display pages crawled by Googlebot.

Output:
Each hit from a search engine spider will be displayed in a row with these columns:

Time (timestamp of the spider visit)

IP Address (remote ip address of the spider)

Spider (name of the spider)

Status (http error code)

Page Crawled (url of the page visited)

If more than one spider visit is found, a bar chart is displayed comparing
the number of hits from each search engine spider.

Instructions:
Download script
[UNIX]
[Windows]
to your machine.
Edit the line that reads:
LOG_FILE="/var/log/httpd/access_log"
so the CGI script will know where your Apache web server log file resides.
Upload trackSpiders.cgi to a directory on your web server where you can run CGI scripts.
Change permissions to make the file executable (chmod +x or chmod 755).
If any of these instructions do not make sense, ask your webmaster or web hosting company
to handle this for you.
Or, contact us to become a client and we will
install this free web analytics software and other
custom search marketing reports on your site.

Notes:
This free CGI script only works on web sites using the Apache web server on a UNIX platform.
It will parse a web server log that is in Apache combined log format.
The script is open source code.