Didn't find any specific information on that tragedy they call a website about what they even support in robots.txt, but it doesn't really matter as they got nothing from me except a place of honor on my blog.

1 comment:

Hi incrediBILLReferance my post on webmasters world, can you put this post on your site.

As for the bot, it is my intention to ensure that it behaves correctly. the bot started out as V1.0 and is now at V1.3 as any problems encounted are fixed and the upgrade is displayed in the version number.

The reason the website is included in the bot name is to allow any webmasters who encounter problems to contact me, so that it can be looked into, i do not include a email address as this is giving any spammers a free meal if the bot visits their site.

So if any webmasters encounter problems with the bot, please visit the site and use the contact form, in the subject list there is a "spider" option, select that for your mail.