|
Juri Haberland wrote:
>
> Hi list,
>
> this is somewhat offtopic but I thought that you guys will have an answer
> to my question as many of you may be faced with the same problem:
> I'm looking for a script that merges and sorts the access_log files of my
> three real servers running apache. The logs can be up to 500MB if combined.
> I found some perl scripts on the web but they are very memory intensive and
> are also very slow.
> Does one of you have a script that does the job more or less god?
>
> TIA,
> Juri
>
> --
> Juri Haberland <juri@xxxxxxxxxxxxxx>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users
cat one log to the end of the other then run
sort -t - -k 3 ${WHEREVER}/access.log > new.log
then you can run webalizer on it.
Thats what I use, doesnt take more than about
30 seconds. If you can copy the logs from your
real servers to another box and run sort there, it
seems to be better
Heck, heres the whole(sanitized) script
-------------------CUT--------------------------
#!/bin/bash
##
## Set constants
##
DATE=`date "+%d-%b-%Y"`
YESTERDAY=`date --date="1 day ago" "+%d-%b-%Y"`
ROOT="/usr/stats"
SSH="/usr/local/bin/ssh"
## First(1) Remove the tar files left yesterday
find ${ROOT} -name "*.tar.bz2" |xargs -r rm -v
##
## First get the access logs
## Make sure some_account has read-only access to the logs
su - some_account -c "$SSH some_account@xxxxxxxxxxxx \"cat
/usr/local/apache/logs/access.log\" > ${ROOT}/logs/$DATE.log"
su - some_account -c "$SSH some_account@xxxxxxxxxxxx \"cat
/usr/local/apache/logs/access.log\" >> ${ROOT}/logs/$DATE.log"
##
## Second sort the contents in date order
##
sort -t - -k 3 ${ROOT}/logs/$DATE.log > ${ROOT}/logs/access.log
##
## Third run webalizer on the sorted files
## Just set webalizer to dump the files in ${ROOT}/logs
/usr/local/bin/webalizer -c /usr/stats/conf/webalizer.conf
##
## Forth remove all the crud
## You still got the originals on the real servers
find ${ROOT} -name "*.log"|xargs -r rm -v
##
## Fifth tar up all the files for transport to somewhere else
cd ${ROOT}/logs && tar cfI ${DATE}.tar.bz2 *.png *.tab *.html && chown
some_account.some_account ${DATE}.tar.bz2
----------------END CUT----------------------
stuart.vcf
Description: Card for Stuart Fox
|
