LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

[lvs-users] LVS-NAT wrr crashing on 64-bits

To: "'LinuxVirtualServer.org users mailing list.'" <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: [lvs-users] LVS-NAT wrr crashing on 64-bits
From: "Kees Hoekzema" <kees@xxxxxxxxxxxx>
Date: Wed, 11 Jul 2007 14:27:53 +0200
Well, apparently it didn't have to do anything with the NAT issue Cristi was
having; so let's split those two problems as it would seem I have an other
problem than him ;).

-kees

 

> -----Original Message-----
> From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
> [mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf 
> Of Kees Hoekzema
> Sent: Wednesday, July 11, 2007 14:20
> To: 'LinuxVirtualServer.org users mailing list.'
> Subject: Re: [lvs-users] LVS-NAT issue
> 
>  
> 
> > -----Original Message-----
> > The problem is as folows: the setup works randomly, fron 15 mins to 
> > 1-2 hours, flawlessly, i might add, serving content from 
> both backend 
> > machines. However, it randomly stops doing that. When that 
> happens, i 
> > cannot ping the VIP from the outside, only from within the 
> LAN (i have 
> > a backup LB, not configured yet, i plan to use ultramonkey 
> later on). 
> > I checked logs, tcpdumped but with no clue as of what is 
> causing this.
> > Some input would be really appreciated.
> 
> Now I know this is an old message, and this issue has been 
> 'resolved' by not using LVS-NAT anymore, but recently I had a 
> similar problem.
> 
> Let me explain my setup first; I have two loadbalancers, 
> which use wrr to direct trafic to 5 realservers. A small 
> script on the loadbalancers checks the realservers 
> periodically and requests some numbers from them. Based on 
> those numbers the weight of the server is adjusted using 
> 'ipvsadm --edit-server'.
> 
> The setup i described above worked flawlessly for years (well 
> - after an iptables problem, and after a small patch to the 
> wrr code) until my trafic could spike so high the 
> loadbalancers were not able to handle it properly.
> So we decided to upgrade the loadbalancers with new hardware.
> 
> The new hardware runs on a quadcore 64-bits Xeon, while the 
> old had a 32 bits Celeron, so quite an upgrade, and more 
> notable, the new server was able to process 950 mbit with 
> only 20% cpu time, while the old one was eating up more than 
> 90% cputime at around 60 mbit.
> 
> So we went from a 32 bits OS to a 64 bits OS. We tested the 
> hardware and it seemed stable, next we put them into 
> production and after several hours they would crash and would 
> not respond to anything, much like Cristi experienced before. 
> So we pulled them out and put in the old loadbalancers again 
> and we started testing a bit more.
> 
> After running and writing several program's i got the 
> loadbalancers to crash finally again but this time in our 
> testing environment. To achieve a crash i had to generate 
> enough traffic from different ip's and ports through the ipvs 
> services while running 'ipvsadm --edit-server' on the loadbalancer.
> Running the traffic through iptables wouldn't crash the 
> server, nor would one client ip from different ports bashing 
> the services work.
> 
> So i started debugging a lot more and i am still working on 
> it, the problem being is that the server will freeze totally, 
> so i can't look up anything.
> but it seems that changing the weights on the server will 
> make your system crash if you run it on a 64 bits OS. our 
> 'old' 32 bits environment still happily changes the values of 
> the servers every couple of seconds without crashing. So 
> there is somewhere in the code of the ipvsadm program, or in 
> the kernel code a problem - so i'll keep debugging.
> 
> What i want to know is if there is anyone out there with:
> 1) a 64 bits installation
> 2) using wrr
> 3) is changing the weights on the server while the server is 
> getting heavy traffic from multiple ip:ports And is 
> experiencing the same problems as i do; a freezing server 
> which needs a cold reset
> 
> For the moment, ill just keep looking at traces to see if i 
> can spot anything particular, and i hope anyone got a 
> suggestion as to where to look / what debugger to use.
> 
> -kees
> 
> 
> _______________________________________________
> LinuxVirtualServer.org mailing list - 
> lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send requests to 
> lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> 



<Prev in Thread] Current Thread [Next in Thread>