[Sussex] Apache throttling/fair use

[Thomas Adam]

On Sun, Jan 23, 2005 at 09:55:11PM +0000, Steve Dobson wrote:
> 1). Limit the number of pages downloaded per time period,
>
> 2). Stop web crawlers (like wget) from just grabbing the site's
> content.


http://www.leekillough.com/robots.html

Is quite good.  And shows you how to negate an erroneous or faked
userAgent string...

-- Thomas Adam

-- 
I know nothing, and understand even less.