Setting your Robots.txt file for your SuperSite 2

What is a Robots.txt file?

The Robot Exclusion Standard (Robots Exclusion Protocol or robots.txt protocol) is a convention to prevent search engines and web robots from accessing all or part of a website which is otherwise publicly viewable. It can be used in conjunction with Sitemaps, a robot inclusion standard for websites.

A Robots.txt file is an easy way for webmasters to inform search engines and web robots about pages on their sites that need to be excluded from indexing/crawling. The Robots.txt file must be placed in the root of the website heirarchy, for example yourdomainname.com/robots.txt. This text file should contain the instructions in a specific format mentioned below. Search engines and web robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. If this file doesn't exist, they assume that the web owner wishes to provide no specific instructions.

User-agent: *

Disallow: /admin/

The above instructions tell the search engines and web robots to exclude http://Your-SuperSite2-Domain-Name.com/admin from indexing/crawling.

You can also provide instructions regarding the Sitemap of your SuperSite 2: