I was making amendments to my Google Analytics ‘stuff’ and while doing so, did a few external tests. To my horror I was able to discover that my ‘robots.txt’ was also viewable making this public via a simple Google search !!!

I kid you not...

So it seems not only are Google helping promote my site, they are offering my site to be abused (assume also yours?), and have provided the route to do so. Clearly I am not willing expose my site any further.

My approch may be old school and uses cloaking as an option come solution. Now if somebody or 3rd party masks themselves as a Googlebot they will fail during the scripts robot validation so will be redirected to the main domain via php. This also ties Google from exposing my robots file..

1) As first step you need to add these lines to your .htaccess file, or if you don't have, create one and upload it to the root domain folder.

Only one problem with that in that it will limit it's masking to anyone spoofing specific spiders but still a good way to block any attemps to circumnavigate access you might have allowed for spider bots from google, etc... to access limited content/product info.