Comments on: Control how Google Bot ranks and crawls your blog

By: sumit

sumit — Sat, 10 Jan 2009 10:10:23 +0000

could you tell me what should i put a crawl rate according to you

By: Shirley

Shirley — Fri, 02 Jan 2009 03:15:33 +0000

Good stuff. Those options have been there for a while, but I like the new format and placement within the tools.

@robrrt, to prevent them from crawling other things, just create a robots.txt file. Also, in webmaster tools you can tell them to 'delete' old urls. Webmaster Tools gives you so many option. Most of the time, Google is in control. But there, YOU are in the driver's seat.

By: robrrt

robrrt — Thu, 18 Dec 2008 06:53:25 +0000

I wish they would include an option for “Crawl only what is listed in my (XML) sitemap and stop wasting bandwidth crawling stuff I deleted over a year ago.”