With the latest updates in Google Webmaster tools, Google has given more control and power to the webmasters to ensure that they can define that how Google Bot will handle their website. Although, the changes were present earlier in the previous version of Webmaster tools, it surely has improved fairly in this version.
After you login to Google Webmaster tools and select the specific Sitemap, on the left hand sidebar you’ll find Settings and that will give you the power to control various aspects of Google Bot. Here are the settings and a little overview of each of them.
1. Geographic Target – This will allow you to specify that which country are you serving the content for. If you want to rank well in United States more than any other country, then you can select United States amongst the list of countries.
2. Preferred Domain – This option was available in the previous version as well. It allows you to specify that which URL structure do you prefer more, is it the standard www.mydomainname.com or just mydomainname.com. It’s good if you select one of the ways, as Googlebot considers www.mydomainname.com and mydomainname.com as two different websites; if you specify one of them then Googlebot will use that particular URL structure and will rank it in its index.
3. Image Search – Image Labeler is a fun service from Google, it’s sort of a game that people can play while improving the way Google indexes the images. As a webmaster, you get the ability to control that if Google can include your blog/website in this service and allow the ‘players’ to label the images included in your blog posts or website.
4. Crawl Rate – This feature was also available in the earlier version, however now it flaunts a slider that allows you to set the speed according to your needs. You can increase and decrease the speed of Googlebot’s crawl rate to ensure that it crawls your blog completely.
I remember correctly that these features were set by me, however after the upgrade of the service some of them were not there. I strongly suggest you to check the options in Google Webmaster Tools and ensure that if the settings are correct for your blog or not.
I wish they would include an option for “Crawl only what is listed in my (XML) sitemap and stop wasting bandwidth crawling stuff I deleted over a year ago.” :)
Good stuff. Those options have been there for a while, but I like the new format and placement within the tools.
@robrrt, to prevent them from crawling other things, just create a robots.txt file. Also, in webmaster tools you can tell them to 'delete' old urls. Webmaster Tools gives you so many option. Most of the time, Google is in control. But there, YOU are in the driver's seat. :-)
could you tell me what should i put a crawl rate according to you
Have the crawl rate on maximum.