Sitemaps, Meta Data, and robots.txt Forum

I uploaded robots.txt file to the domains I don't want spidered. These domains all have the meta tag <META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">. I'm wondering if the robots.txt will override the meta tag, or whether I will need to change the tag to noindex,nofollow....

Depends on the content of your robots.txt. If you block access to a given page via robots.txt, but the page contains a meta robots tag, the meta is useless, because the spider will never request the page, and will therefore not see the tag.

A well behaved spider will not even look at the pages that are excluded by robots.txt. In other words, for those, it doesn't matter at all what they could find there if they looked. However, there's the off chance that a not quite so well behaved spider might ignore robots.txt and only look at the meta tags, so you don't want to put contradicting information there.

In fact, any kind of policy information (robots.txt and the robots meta tag are nothing else) should always be consistent across all the channels you use. What point is there in saying "no" with your left mouth, and "yes" with the right?

The only situation where this could make any sense would be when excluding only one (or several) specific spiders from robots.txt and allowing all others. But the phrasing of your question makes me think that you either want to allow all of them or none at all.

What about this "reverse" situation: Suppose the robots.txt file contains a single byte (spacebar), indicating that all pages are OK to spider, Will the SE conclude the robots.txt should override a meta robots noindex,nofollow which might appear on an individual page? Or ( as I hope ), will the spider still respect the "noindex" tag on an individual page