Example robots.txt file

It says any search engine may crawl pages except "/tag" and "/recaptcha". Also it says that we have a sitemap on nameocean.net/sitemap.xml.

How to add robots.txt to your website?

If you are using a simple http server which servers every file on a path like /var/www, you can add robots.txt as a text file. We use nginx at nameocean and serve our robots.txt file from nginx. Here our nginx conf file.