Boredworkers.com

Ever had the feeling that Google wasn’t crawling your site as much as it should. Perhaps it is coming around every other day or once a week. Want to influence the rate in which Googlebot (Google’s web crawler) downloads and caches pages from your site? Now you can… in some sense.

If you’ve been using Google Sitemaps service, you should notice that they have introduced a new feature called Crawl Rate. The feature allows you to see how often Google has been visiting your site, the average number of pages it crawls a day, the amount of bandwidth the robot consumes, and the amount of time spent downloading pages. While all this is mumbo-jumbo that doesn’t really do much except to tell you that Google is coming around once in a while, there is a section at the end of the page that allows you to increase or decrease the rate in which Google trawls your site.

From my understanding, this isn’t the rate in which Googlebot comes by your site per se, but more along the lines of increasing the amount of bandwidth that the crawler consumes. An increased bandwidth cap means an increased number of pages downloaded by the robot. That in turn naturally means that Google sees more of your site. Provided, of course, that you have new content available in the first place. It also means that your older cached pages will be updated more often (again subject to it being changed in the first place).

There are three different crawl rate settings available. By default, the crawl rate is set to Normal which limits the rate in which pages are crawled to conserve your server resources. However if you have loads of resources to burn, you’re free to up the ante or vice versa if your server is slow and tight on bandwidth. Any changes you make will remain for 90 days before reverting to Normal.

• Faster
  A faster crawl will enable us to crawl your site quickly, but may put more load on your server.
• Normal
  Recommended crawl rate.
• Slower
  A slower crawl will reduce Googlebot’s traffic on your server, but we may not be able to crawl your site as often.

Is this really useful? Maybe, maybe not. But it is fun for the curious to be able to change the crawl rate. Whether or not it has any direct effect on the crawler is a different story. I’ll keep an lookout for Googlebot and its bandwidth consumption and let you know if I see any difference.