Google Introduces New Googlebot User-Agent for More Effective Crawling On Smartphone Content

Google announced via Google Webmaster Central Blog that the old user-agent “Googlebot-Mobile” mobile-specific crawler will be retired in the next 3 – 4 weeks. “Googlebot-Mobile” user-agents have been tested since 2011 to improve the mobile search experience for feature phone and smartphone users.

According to Zhijian He, Google’s Smartphone Search Engineer, “We’ve seen cases where a webmaster inadvertently blocked smartphone crawling or indexing when they really meant to block just feature phone crawling or indexing.” Over the years, as Google developed new mobile crawlers, the company struggled to index mobile content for feature phones versus smartphones.

So now Google has taken action to give webmasters more control by revealing “Googlebot,” the new user-agent built specifically to crawl content on smartphones. “Googlebot-Mobile” will continue to crawl content for feature phone content.

Google made sure to specify this important implication of the user-agent update, “The new Googlebot for smartphones crawler will follow robots.txt, robots meta tag, and HTTP header directives for Googlebot instead of Googlebot-Mobile.”

Based on a Google internal analysis, the update should affect less than 0.001% of URLs. As the update rolls out webmasters can test sites, He says, “Using the Fetch as Google feature in Webmaster Tools, and they have full access to the full list of existing crawlers in the Help Center.”

About Garry Grant

Garry Grant is one of the premier experts in the fields of search engine optimization and Internet Marketing. This bold statement is proven by Garry’s remarkable track record of successes. At SEO Inc., Garry is responsible for setting the strategic direction, developing proprietary technologies, overseeing the staff, and setting corporate sales and marketing goals. He has also earned a reputation as an educational and captivating speaker.