[Phpwm] Problems with live Silverstripe website (httpd spawing proceses, high CPU)

[phil at infolinkelectronics.co.uk]

Google claim, and I have yet to hear this challenged, that they completely
obey a robots.txt file where found.

If you have a larger site you can email them and they will increase the
spidering frequency / page speed, resulting in faster indexing and listing.

Regards,

Phil Beynon
Sales & Operations Director

** Infolink Electronic Systems Ltd.
** S/W Birmingham's Longest Established ISV & VAR
** http://www.infolinkelectronics.co.uk
** Professional Web 2.0 Design & Advanced Hosting Platforms
** http://www.coeus-computica.co.uk
** Affordable Knowledgebase Solutions
** Contact: Sales at infolinkelectronics.co.uk
** Tel / Fax 0121 441 3558

> You can set crawl rate for most of major search engines using robots.txt:
>
> Crawl-delay defines the amount of seconds to wait between successive
> requests to the same server.
>
> Regards,
> Nerijus
>
>
> > On Fri, Aug 20, 2010 at 9:34 AM, David Goodwin
> > <david at codepoets.co.uk>wrote:
> >
> >>
> >> My understanding is that Google is quite good at indexing a site... and
> >> does a little, and then comes back, does a little more etc -
> rather than
> >> trying to DOS your site.
> >>
> >
> > In past experiences I've seen it grind a slow-ish site to a halt.
> > Obviously
> > it was nowhere near DoS levels, but it was spread over 15 mins
> with around
> > 100 reqs/sec for the majority of that time.
> >
> > And even if google were misbehaving, I think you can (if you need to)
> >> specify a spider rate through the webmaster's site @ google.
> >
> >
> > As far as I know the crawl rate is only as far as how frequently it
> > updates
> > your site, i.e how much your site content changes. Rather than
> the number
> > of
> > requests per second it sends.
> >
> > I could be wrong though.
> >
> > Regards,
> >
> > Paul.
> > _______________________________________________
> > Phpwm mailing list
> > Website : http://www.phpwm.org
> > Twitter : http://www.twitter.com/phpwm
> > Facebook: http://www.facebook.com/group.php?gid=2361609907
> >
> > Post to list: Phpwm at mailman.lug.org.uk
> > Archive etc : https://mailman.lug.org.uk/mailman/listinfo/phpwm
>
>
>
> _______________________________________________
> Phpwm mailing list
> Website : http://www.phpwm.org
> Twitter : http://www.twitter.com/phpwm
> Facebook: http://www.facebook.com/group.php?gid=2361609907
>
> Post to list: Phpwm at mailman.lug.org.uk
> Archive etc : https://mailman.lug.org.uk/mailman/listinfo/phpwm
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 9.0.851 / Virus Database: 271.1.1/3080 - Release Date:
> 08/19/10 19:35:00
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Phil Beynon.vcf
Type: text/x-vcard
Size: 636 bytes
Desc: not available
URL: <http://mailman.lug.org.uk/pipermail/phpwm/attachments/20100820/6cc94cd7/attachment.vcf>