|
Hello Kees!
Kees Hoekzema schrieb:
> Today we changed the port numbers for 213.239.154.35:80 to port 81 so our
> visitors went to the new site. However, this also increased the load on the
> loadbalancer dramaticly. At one point I had to stop each and every
> additional service on the loadbalancer so it was only doing iptables and
> ipvs, and still it was using up to 100% system-cpu time.
>
Since ipvs and iptables both run in kernel space you should see no cpu
time of these processes
e.g. with the top and vmstat command.
There may be other processes on your loadbalancers that consume the
system-cpu-time.
Have you checked for cloaked hackers aka rootkits?
We have sites with much over 100mbits traffic and around 100.000
connections an
hardware comparable to your setup exept we have more RAM. So it seems
unlikly that ipvs is the problem.
I noticed the InActConn on the .35 service was quite high, when the site was
doing 60 mbit I noticed over 120.000 inactive connections. Can this be a
problem? I tried to use 'ipvsadm --set 30 30 30' to lower the timeouts, but
ipvsadm --list -cn still shows a lot of connections with a timeout > 30
seconds. And basicly all in the TIME_WAIT state.
InActConn = conn in TIME_WAIT state :-)
Maybe you run out of your system memory if you have may long living
connections. to serve
your 120.000 connections over 30 secs you will need
120.000 con/sec times 128 Bytes times 30 sec = 440 MB RAM.
So you are near this limit with your 512 MB boxes.
Best regards,
Volker
--
====================================================
inqbus it-consulting +49 ( 341 ) 5643800
Dr. Volker Jaenisch http://www.inqbus.de
Herloßsohnstr. 12 0 4 1 5 5 Leipzig
N O T - F Ä L L E +49 ( 170 ) 3113748
====================================================
|
