Saturday, 16 February 2013

robots.txt - a cautionary tale

I have to confess that I'd been working on robots.txt functionality in Scrutiny and had accidentally left the robots.txt file in place.

That's a cautionary tale, but not the one I wanted to tell. My file was very similar to this (with a couple of urls in the Google disallow list) - the screenshot is from www.robotstxt.org

I've ruled out a few possibilities but I see from Google's own developer help that the name of the Google robot is actually googlebot not Google. I'm assuming that this is the reason for my problem. If you can confirm this or know better, please leave a message in the comments below.