web development

Robots.txt is a file but not HTML, which is placed on the site to depict the search robots and designate the pages, which the user would not like them to visit. This is definitely a mandate for the Search Engines, but usually, the Search Engines obey what they are asked....