The above will ensure that spiders do not index pages that don't contain important data, or that is too volatile. For example, I would hate to see all my users on Google, but few of the messages. Even worst, the message indexes are indexed, but when the users visit, the data is not on that page anymore.