If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

Robots meta tag gives option that crawl or not crawl specific page.It also informs about do-follow and no-follow links on the web page.whereas robots.txt is a list of a directories which mainly used for restriction about crawling of a specific web pages.so both used for different purposes and not better than each other.

Robots.txt file is better compared to meta robots tag. Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Meta tag can be used to disallow a single page. ... There is the very huge difference between meta robot and robots.txt. In robots.txt, we ask crawlers which page you have to crawl and which one you have to exclude but we don't ask crawler to not to index those excluded pages from crawling.

Often the robots.txt file located at the root of the domain name is the ideal tool for communicating with search engines (Robots with Google, Yahoo and Microsofts) in indexing websites. However, in some cases, the /robots.txt file becomes limited and its use is very cumbersome. Heavy META "robots" at the moment are useful even when using it. Quite different from robots.txt file

Robots.txt file is a text file for restricting bots (robots, search engine crawlers ) from a website or certain pages on the website. Using a robots.txt file and with a disallow direction, we can restrict bots or search engine crawling program from websites and or from certain folders and files.

A robots meta tag is used to give direction for robots or the spiders regarding crawling and indexing of particular page. By using a robots meta tag which requests bots to avoid crawling and indexing, that page can be avoided from crawling by search engine bots or robots and avoid indexing in search results. A request name “follow” or “nofollow” is used in robots meta tags provided in the head section of a webpage to consider or not consider the counting of links provided on the page. Not adding a robots meta tag will be considered as the default direction such as index, follow.

Use robots.txt file, use meta robots when the robots.txt file does not work properly because meta robots hurt sites sometimes.
If you use the robots.txt file then you can block different directories or pages with one file, if you use meta robots you have to add it on every page where you want to be and indirectly it makes your code lengthy.