Webmaster forum - Reducing googlebot crawl rate

Author

Reducing googlebot crawl rate

Matt Probert

2005-12-31, 6:16 am

Further to my queries regarding the rate at which Googlebot spiders, I
have received the following official information from Google:

"our robots do not currently recognize the Crawl-delay command [in
robots.txt] but we are able to change the rate by your email request."

Regards

Matt

--
The Probert Encyclopaedia - Beyond Britannica
http://www.probertencyclopaedia.com

Blinky the Shark

2005-12-31, 6:23 pm

Matt Probert wrote:

> "our robots do not currently recognize the Crawl-delay command [in
> robots.txt] but we are able to change the rate by your email request."

An actual response from Google? Isn't that pretty much unheard of?

--
Blinky
Killfiling all posts from Google Groups
Details: http://blinkynet.net/comp/uip5.html

Charles Sweeney

2005-12-31, 6:23 pm

Matt Probert wrote

> Further to my queries regarding the rate at which Googlebot spiders, I
> have received the following official information from Google:
>
> "our robots do not currently recognize the Crawl-delay command [in
> robots.txt] but we are able to change the rate by your email request."

That's interesting, bespoke crawling. Potential administrative nightmare
with the number of sites they list.

--
Charles Sweeney
http://CharlesSweeney.com

Mark Goodge

2005-12-31, 6:24 pm

On 31 Dec 2005 12:24:37 GMT, Charles Sweeney put finger to keyboard
and typed:

>Matt Probert wrote
>
>
>That's interesting, bespoke crawling. Potential administrative nightmare
>with the number of sites they list.

Only if a lot of people email asking them to change it. Having a field
for crawl rate in their site database is pretty straightforward,
especially since their system does crawl different sites at different
rates - my guess would be that normally, this is determined by
whatever algorithms decide on how important a site is. And, given that
such a mechanism exists, a manual override for the automated rate
control isn't that much more complex.

What would be more useful, of course, is for their spider to read the
Crawl-delay line in robots.txt and update the database from that. If
enough people email them to ask for a manual rate change, that might
just encourage them to implement it :-)

Mark
--
http://www.OrangeHedgehog.com - RSS feeds and Google Adsense tools
"My light will shine so brightly it will blind you"

Charles Sweeney

2005-12-31, 6:24 pm

Mark Goodge wrote

> On 31 Dec 2005 12:24:37 GMT, Charles Sweeney put finger to keyboard
> and typed:

[color=darkred]
> Only if a lot of people email asking them to change it.

Absolutely.

--
Charles Sweeney
http://CharlesSweeney.com

Mark Goodge

2005-12-31, 6:24 pm

On 31 Dec 2005 14:51:29 GMT, Charles Sweeney put finger to keyboard
and typed:

>Mark Goodge wrote
>
>
>
>
>Absolutely.

I suspect, though, that the number of requests to have crawling slowed
down will be small enough to be manageable. Most people - especially
those who have already swallowed the SEO snake oil - would prefer
crawling to be speeded up. But the answer to that request is always
going to be simply "no".

Mark
--
http://www.MineOfUseless.info - everything you never needed to know!
"And so we're told this is the golden age"

Brian Wakem

2005-12-31, 6:24 pm

Mark Goodge wrote:

> On 31 Dec 2005 14:51:29 GMT, Charles Sweeney put finger to keyboard
> and typed:
>
>
> I suspect, though, that the number of requests to have crawling slowed
> down will be small enough to be manageable. Most people - especially
> those who have already swallowed the SEO snake oil - would prefer
> crawling to be speeded up. But the answer to that request is always
> going to be simply "no".
>
> Mark

I would think most people have no idea at what rate their site is crawled,
or what problems it could cause.

--
Brian Wakem
Email: http://homepage.ntlworld.com/b.wakem/myemail.png