We also use the webbot with our web filter service. Customers visit sites that we don't know about and we use the webbot to go and dig the site. In this case we are looking primarily to filter porn for our customers. The site pages that are downloaded are fed into a scan engine that attempts to determine if the site is objectionable or not.

Well dig this, your customers can grow the fuck up and be adults about the 'net as you aren't digging my fucking website as I have too many little piss ants like your crawler all trying to get a piece of my website so you get ... NOTHING! NOT A SINGLE FUCKING PAGE!