[sclug] Web log analysis tool?

[ed]

On Tue, 23 Jan 2007 00:29:13 +0000
David Given <dg at cowlark.com> wrote:

> ed wrote:
> [...]
> > oh, sounds like an interesting project.
> > 
> > how do you define popularity?  
> 
> Bandwidth. What I want is an early-warning system that tells me if
> lots of downloads happen when I'm not expecting it.
> 
> Actually, thinking about it, this is probably a fairly simple
> scripting project; I can write a little script that parses the
> logfile at rotate time, figures out what files were read (and how

by that time your connection is already hosed. do you want an ISP grade
solution or something just to notify you that there has been some
activity during the last day/hour?

most 'load balancers' are realtime. they inspect packets, ssl proxy (so
i am lead to believe).

> much was read --- it's all helpfully in the logfile), divides them up
> per directory, sorts and collates the result, and mails it to me. An
> easy trick for awk or Lua.
> 
> I'll consider, and get back to you...

i suggest that you either poll the interface (which is one thing you
wish to know about) or additionally check the logs now and then.

one mp3 file would certainly effect me. i think i'd have no internet
for a while, so by this time i would already have looked at my IF with
tcpdump and spotted all the port 80 noise.

this might be more the direction that you could go in, count the port
80 traffic over a minute... if its greater than a certain value then
send yourself an email.

something zabbix might also be useful for you, this can be configured
to send warnings when interfaces go over certain values, or cpu creeps
past a tolerance, it's quite versatile.

-- 
Regards, Ed                      :: http://www.linuxwarez.co.uk
proud c++ person
A meteor didn't kill the dinosaurs, Chuck Norris did.