[Phpwm] Problems with live Silverstripe website (httpd spawing proceses, high CPU)

[nerijus at discountdomainsuk.com]

You can set crawl rate for most of major search engines using robots.txt:

Crawl-delay defines the amount of seconds to wait between successive
requests to the same server.

Regards,
Nerijus


> On Fri, Aug 20, 2010 at 9:34 AM, David Goodwin
> <david at codepoets.co.uk>wrote:
>
>>
>> My understanding is that Google is quite good at indexing a site... and
>> does a little, and then comes back, does a little more etc - rather than
>> trying to DOS your site.
>>
>
> In past experiences I've seen it grind a slow-ish site to a halt.
> Obviously
> it was nowhere near DoS levels, but it was spread over 15 mins with around
> 100 reqs/sec for the majority of that time.
>
> And even if google were misbehaving, I think you can (if you need to)
>> specify a spider rate through the webmaster's site @ google.
>
>
> As far as I know the crawl rate is only as far as how frequently it
> updates
> your site, i.e how much your site content changes. Rather than the number
> of
> requests per second it sends.
>
> I could be wrong though.
>
> Regards,
>
> Paul.
> _______________________________________________
> Phpwm mailing list
> Website : http://www.phpwm.org
> Twitter : http://www.twitter.com/phpwm
> Facebook: http://www.facebook.com/group.php?gid=2361609907
>
> Post to list: Phpwm at mailman.lug.org.uk
> Archive etc : https://mailman.lug.org.uk/mailman/listinfo/phpwm