Robots.txt and Security

Potential security issues around using Robots.txt to block indexing of content in members sections.

Edited: 2014-12-28 01:01

You should not rely to heavily on robots.txt. If there is something, that you do not want to be discovered by search engines, then either don't host it on your site, or implement a decent server-side security mechanism instead.

Not all robots will adhere to the rules in the robots.txt file. Some may even chose to ignore it entirely. Its therefor best, that you use other security mechanisms, to prevent access to content, that you do not want to be accessible.

Robots.txt and Security

There are quite a few security issues around using Robots.txt, none however critical. Robots.txt is mainly useful if you want to control how the major known search engines will access your site, not as a security mechanism.

In addition, listing secret directories in the robots text file, could inform hackers of otherwise unknown locations on your server. Its therefor important, that you have other security mechanisms in place. Simply providing members of your site with a secret URL, is not enough to prevent access, especially not if you list this URL in your robots.txt file to prevent it from showing up in the search results.