Search Engine Spider and User Agent Identification Forum

Not to get off the subject but I have found that once you become a little more "high profile" thru your positioning efforts you actually can become a "target" by either jealous or mischeivious competitors.

Your next move is to start using .htaccess to block particularly pesty folks from their covert operations. I had one guy who was somehow repeatedly requesting the same page all day long which added up to 5 megs or more a day. That irritated me. So I added the following line to my .htaccess file:

deny from 100.100.100.

No more wasted bandwidth on account of someone elses stupidity.

I found out about this guy by looking thru the reports generated by Weblog. I personally like this script the best and have it set up on a cron to run everyday at the same time. There are about 1,000,000 scripts that do this out there but just pick one you like and use it everyday. Keep up with whats going on in your site.

By eliminating the riff raff...and if you keep up with it you will....you can reserve your resources for the real people. Then when the next threat pops up you'll be right on top of the situation.