[Gllug] Apache log files

[william pink]

On Tue, Apr 7, 2009 at 11:57 AM, - Tethys <tethys at gmail.com> wrote:

> On Tue, Apr 7, 2009 at 11:11 AM, william pink <will.pink at gmail.com> wrote:
>
> > I have the rather horrible task of splitting up lots (40Gb's worth) of
> > Apache log files by date, the last time I did this I found the line
> number I
> > then tailed the file and outputted it into a new file which was a long
> > arduous task. I imagine this can be done in a few minutes with some
> > Regex/Sed/AwkBash trickery but I wouldn't know where to start can anyone
> > give me any pointers to get started?
>
>         #!/bin/bash
>
>        indatefmt="+%d/%b/%Y"
>        outdatefmt="+%Y-%m-%d"
>
>        start_date="mar 25"
>        end_date=$(date "$indatefmt")
>
>        count=0
>        while true
>        do
>                indate=$(date "$indatefmt" -d "$start_date + $count days")
>                outdate=$(date "$outdatefmt" -d "$start_date + $count days")
>
>                fgrep "$indate" big_logfile > "small_logfile.$outdate"
>
>                [ "$indate" = "$end_date" ] && break
>                ((count++))
>        done
>
> It's a bit inefficient, as it scans the log file multiple times,
> but for comparatively small log files like you have, that shouldn't
> be too arduous. It'll also pick up any entries that happen to have
> the date format you're looking for in the URL, for example. To work
> around either of those, using a scripting language like python or
> perl to read and examine each line in turn is probably the right
> solution. But the quick and dirty approach above will probably be
> fine for you.
>
> Then fix your setup so it logs to per-date files to start with...
>
> Tet
>
> --
> The greatest shortcoming of the human race is our inability to
> understand the exponential function -- Albert Bartlett
> --
> Gllug mailing list  -  Gllug at gllug.org.uk
> http://lists.gllug.org.uk/mailman/listinfo/gllug
>

Hi Tet,

Thats just what I needed, I promise to practice my bash scripting while this
script runs through these logs.


Many Thanks,
Will
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.lug.org.uk/pipermail/gllug/attachments/20090407/6df4b574/attachment.html>
-------------- next part --------------
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug