[Phpwm] Problems with live Silverstripe website (httpd spawing proceses, high CPU)

[Paul Matthews]

On Fri, Aug 20, 2010 at 9:34 AM, David Goodwin <david at codepoets.co.uk>wrote:

>
> My understanding is that Google is quite good at indexing a site... and
> does a little, and then comes back, does a little more etc - rather than
> trying to DOS your site.
>

In past experiences I've seen it grind a slow-ish site to a halt. Obviously
it was nowhere near DoS levels, but it was spread over 15 mins with around
100 reqs/sec for the majority of that time.

And even if google were misbehaving, I think you can (if you need to)
> specify a spider rate through the webmaster's site @ google.


As far as I know the crawl rate is only as far as how frequently it updates
your site, i.e how much your site content changes. Rather than the number of
requests per second it sends.

I could be wrong though.

Regards,

Paul.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.lug.org.uk/pipermail/phpwm/attachments/20100820/744cadf1/attachment.htm>