<br><br><div class="gmail_quote">On Fri, Aug 20, 2010 at 9:34 AM, David Goodwin <span dir="ltr"><<a href="mailto:david@codepoets.co.uk">david@codepoets.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><br>
</div>My understanding is that Google is quite good at indexing a site... and<br>
does a little, and then comes back, does a little more etc - rather than<br>
trying to DOS your site.<br></blockquote><div><br></div><div>In past experiences I've seen it grind a slow-ish site to a halt. Obviously it was nowhere near DoS levels, but it was spread over 15 mins with around 100 reqs/sec for the majority of that time.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
And even if google were misbehaving, I think you can (if you need to)<br>
specify a spider rate through the webmaster's site @ google.</blockquote><div><br></div><div>As far as I know the crawl rate is only as far as how frequently it updates your site, i.e how much your site content changes. Rather than the number of requests per second it sends.</div>
<div><br></div><div>I could be wrong though.</div><div><br></div><div>Regards,</div><div><br></div><div>Paul. </div></div>