Twitter Opens Up To More Crawling, But Do Search Engines Want Its Search Results In Theirs?

Twitter recently updated its robots.txt file and, though the change opens up millions of pages to being crawled, there’s no guarantee that the main search engines want what Twitter is offering. The Sociable seems to have been the first to notice Twitter’s robots.txt changes , which now specifically allow Google, Bing, Yahoo, Yandex and other bots to crawl through some of Twitter’s search results pages. Twitter: Change Made To Help With Discovery Twitter confirmed the change to us, saying: This change will help people find popular and helpful Twitter pages, such as the #olympics hashtag page (https://twitter.com/search/?q=%23olympics ) Twitter is opening up its content search results — i.e., tweets and hashtags — while still preventing bots from crawling its search results for users, videos and images. Tweet and hashtag search results show via the /search path; searches for...