LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] Connections are "clumped" together on real_server failur

To: LinuxVirtualServer.org users mailing list. <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] Connections are "clumped" together on real_server failure
From: Jay Faulkner <jay.faulkner@xxxxxxxxxxxxx>
Date: Tue, 30 Jun 2009 07:24:31 -0500
> -----Original Message-----
> From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx [mailto:lvs-users-
> bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Malcolm Turnbull
> Sent: Tuesday, June 30, 2009 5:38 AM
> To: LinuxVirtualServer.org users mailing list.
> Subject: Re: [lvs-users] Connections are "clumped" together on
> real_server failure
> 
> Jay,
> 
> Try switching from WLC to WRR? (I'm guessing that you are using WLC).
> 


I'm a little nervous doing that, if it doesn't solve the issue, then we're in 
bigger trouble once it happens (because new connections will still be going to 
the overloaded server).

Are you certain this fixes the issue?

Jason Faulkner
Linux Systems Engineer
Rackspace Email & Appsq



> 
> 
> 2009/6/25 Jay Faulkner <jay.faulkner@xxxxxxxxxxxxx>:
> > Hey guys,
> >
> > We've been noticing some really strange behavior with keepalived/LVS
> when a real_server fails. The behavior we see is that when a r_s fails,
> all the persistent connections on that server fall over to a single
> server.
> >
> > For example:
> >
> > Server 1 - 25000 active conn. - 1500 inactive conn.
> > Server 2 - 35000 active conn. - 1000 inactive conn.
> > Server 3 - 20000 active conn. - 500 inactive conn.
> >
> > Server 2 fails. Instead of seeing all the traffic split between the
> remaining real_servers, we see it all clump over to the same server
> >
> > Server 1 - 25000 active conn. - 1500 inactive conn.
> > Server 2 - failed
> > Server 3 - 55000 active conn. - 1500 inactive conn.
> >
> > Now, server 3 falls over and dies due to having too many connections
> on it and becoming overloaded.
> >
> > We've seen this behavior consistently in all of our keepalived
> installations (20 servers or so? Median of about 20 virtual_servers
> each), and I was curious if this is configurable behavior, or even
> expected.
> >
> > Thanks!
> >
> > Jason Faulkner
> > Linux Engineer, Rackspace Email & Apps
> > jason.faulkner@xxxxxxxxxxxxx<mailto:jason.faulkner@xxxxxxxxxxxxx>
> > o: (540) 443-2101 (ex. 505-2101)
> >
> > _______________________________________________
> > Please read the documentation before posting - it's available at:
> > http://www.linuxvirtualserver.org/
> >
> > LinuxVirtualServer.org mailing list - lvs-
> users@xxxxxxxxxxxxxxxxxxxxxx
> > Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> >
> 
> 
> 
> --
> Regards,
> 
> Malcolm Turnbull.
> 
> Loadbalancer.org Ltd.
> Phone: +44 (0)870 443 8779
> http://www.loadbalancer.org/
> 
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
> 
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>