Further to my queries regarding the rate at which Googlebot spiders, I
have received the following official information from Google:

"our robots do not currently recognize the Crawl-delay command [in
robots.txt] but we are able to change the rate by your email request."

Regards

Matt


--
The Probert Encyclopaedia - Beyond Britannica
http://www.probertencyclopaedia.com

Matt Probert wrote:


> "our robots do not currently recognize the Crawl-delay command [in
> robots.txt] but we are able to change the rate by your email request."

An actual response from Google?  Isn't that pretty much unheard of?


--
Blinky
Killfiling all posts from  Google Groups
Details: http://blinkynet.net/comp/uip5.html

Matt Probert wrote

> Further to my queries regarding the rate at which Googlebot spiders, I
> have received the following official information from Google:
>
> "our robots do not currently recognize the Crawl-delay command [in
> robots.txt] but we are able to change the rate by your email request."

That's interesting, bespoke crawling.  Potential administrative nightmare
with the number of sites they list.

--
Charles Sweeney
http://CharlesSweeney.com

On 31 Dec 2005 12:24:37 GMT, Charles Sweeney put finger to keyboard
and typed:

>Matt Probert wrote
> 
>
>That's interesting, bespoke crawling.  Potential administrative nightmare
>with the number of sites they list.

Only if a lot of people email asking them to change it. Having a field
for crawl rate in their site database is pretty straightforward,
especially since their system does crawl different sites at different
rates - my guess would be that normally, this is determined by
whatever algorithms decide on how important a site is. And, given that
such a mechanism exists, a manual override for the automated rate
control isn't that much more complex.

What would be more useful, of course, is for their spider to read the
Crawl-delay line in robots.txt and update the database from that. If
enough people email them to ask for a manual rate change, that might
just encourage them to implement it :-)

Mark
--
http://www.OrangeHedgehog.com - RSS feeds and Google Adsense tools
"My light will shine so brightly it will blind you"

Mark Goodge wrote

> On 31 Dec 2005 12:24:37 GMT, Charles Sweeney put finger to keyboard
> and typed:
 

> Only if a lot of people email asking them to change it.

Absolutely.

--
Charles Sweeney
http://CharlesSweeney.com

On 31 Dec 2005 14:51:29 GMT, Charles Sweeney put finger to keyboard
and typed:

>Mark Goodge wrote
> 
> 
> 
>
>Absolutely.

I suspect, though, that the number of requests to have crawling slowed
down will be small enough to be manageable. Most people - especially
those who have already swallowed the SEO snake oil - would prefer
crawling to be speeded up. But the answer to that request is always
going to be simply "no".

Mark
--
http://www.MineOfUseless.info - everything you never needed to know!
"And so we're told this is the golden age"

Mark Goodge wrote:

> On 31 Dec 2005 14:51:29 GMT, Charles Sweeney put finger to keyboard
> and typed:
> 
>
> I suspect, though, that the number of requests to have crawling slowed
> down will be small enough to be manageable. Most people - especially
> those who have already swallowed the SEO snake oil - would prefer
> crawling to be speeded up. But the answer to that request is always
> going to be simply "no".
>
> Mark


I would think most people have no idea at what rate their site is crawled,
or what problems it could cause.



--
Brian Wakem
Email: http://homepage.ntlworld.com/b.wakem/myemail.png