Googlebot Crawler comes too often: How can I limit the frequency?

I think, it's really good that Google has taken my website in the index, since I get most of my visitors through the Google search.

I think, it's also good, that Google is trying to keep its index up-to-date and therefore sent its Googlebot Crawler often through the Web.

However, for me, the visits of the crawler are a little too often. I have a web project with quite a lot pages and now, the Google crawler is responsible for 80 percent of my traffic, what is to me a little too much, as my pages change only little over the time.

Computer Expert

Robots-File

If you do not already have a robots.txt in the root directory of your site, attach a file and write in it a value for "Crawl-delay". If your site is named example.com, your file should be available on example.com/robots.txt. Your robots.txt might look like this:

User-agent: *
Crawl-delay: 30

The value indicates in seconds, how much time should there be between two visits of the crawler. The value of 30 means, that Google should only look every 30 seconds. Normally, Google visits your site within one second for several times, so that should be an improvement. However, it is rumored that not many crawlers follow these values.

Google Webmaster Tools

Another way is to use the Google Webmaster Tools. Here, in the settings there is the opportunity to customize the "crawl rate" for each website individually. I think this is the best solution for Google.13/08/2012 at 10:48

Important Note

Please note: The contributions published on askingbox.com are contributions of users and should not substitute professional advice. They are not verified by independents and do not necessarily reflect the opinion of askingbox.com. Learn more.

Participate

Ask your own question or write your own articles on askingbox.com. How to do.