massey wrote:
> Block those bot's IP's. Nothing else will work. If they are so rude as
> to ignore your robots.txt you should just worry about mozdev. There
> must be something about this kind of terrorism in the 'homeland
> security act' :-)
In fact this may be a nice way to get rid of them.
You could put a link in robots.txt to a particular URL which nobody else
would access (linked from an invisible pixel on the main page). Then if
a robot accesses that address, automatically block it's IP address...
David
>> Ben Bucksch wrote:
>>> Pete Collins wrote:
>>>>> Here is an example of a bot ignoring robots.txt just today:
>>>>>>>> You said you have the robots.txt in the bugs/ dir. Does
>> xprint.mozdev.org/robots.txt also disallow bugs/?
>>