Subject: [lvs-users] LVS and stale connections

From: Fabio Coatti <[email-masked]>
Date: Tue, 31 Mar 2009 14:08:09 +0200

Hi all, 
we are using ipvs since years on several production servers and it works just 
fine, but now we are facing a weird problem. 
I'm unable to find a solution browsing around, so I'm asking for some hints 
here.
Basically, the scenario is the following:
5 servers (RIP), balanced with ipvs on linux 2.6.23, DR is used.
All works just fine, but if for some reasons the real servers freezes and need 
reboot, ipvs keeps as "alive" all the connections so the balancing is not 
correctly done. 
say, after reboot a couple (out of five) of servers shows with ipvsadm -L more 
than 1000 connections (and all of those are fake, as the Rservers have been 
rebooted) and ipvs doesn't pass any new connection until the count lowers 
enough and only 3 servers are actually working.
To fix this, I've tried to remove from ipvs real servers, virtual IP, I've also 
zeroed the tables but no success: when I reconfigure it all the RIP shows that 
all the connections are still there. The situation improves only after the 
connection timeout expires (usually 8 hours). If I lower this timeout, only 
new connection expires in a shorter time, so this is not a good solution. We 
need to keep high timeouts for several reasons (say, databases).
Is there any way to clear this counter (syscalls, proc, whatever) so I can 
recover a correct behaviour of the balancer without booting the machine? I can 
figure out that it's a situation that ipvs can't handle automtically, but a way 
to clear tables should be useful.

If course I'm available for further details.

Many thanks in advance for any answer.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - [email-masked]
Send requests to [email-masked]
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

---

From: [email-masked]
Date: Tue, 31 Mar 2009 10:03:25 -0400 (EDT)

-----Original Message-----
From: "Fabio Coatti" <[email-masked]>
Sent: Tuesday, March 31, 2009 8:08am
To: [email-masked]
Subject: [lvs-users] LVS and stale connections

>enough and only 3 servers are actually working.

We've seen this problem too; the only way I've found to clear that counter is to pull a server from rotation and leave it pulled for 15-30 minutes.

--
Jason Faulkner 
Linux Systems Engineer
Mailtrust, a division of Rackspace

---

From: Joseph Mack NA3T <[email-masked]>
Date: Tue, 31 Mar 2009 07:04:41 -0700 (PDT)

On Tue, 31 Mar 2009, Fabio Coatti wrote:

> Basically, the scenario is the following: 5 servers (RIP), 
> balanced with ipvs on linux 2.6.23, DR is used. All works 
> just fine, but if for some reasons the real servers 
> freezes and need reboot, ipvs keeps as "alive" all the 
> connections so the balancing is not correctly done.

ipvs has no way of knowing that a machine has been rebooted, 
so this is the expected behaviour. Health checking is all 
external to ipvs. Since in LVS-DR the director doesn't see 
the returning packets, it has to guess when connections 
expire, and you've set a long timeout.

Either

o use the -SH scheduler to direct clients to the same 
database server and return the timeouts to the default 
period.

o use one of the ioctls that Horms wrote to handle crashed 
realservers when ipvs is using persistence. These clear out 
the connection table

Joe

-- 
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

---

From: Malcolm Turnbull <[email-masked]>
Date: Tue, 31 Mar 2009 15:15:55 +0100

Ignore me I didn't see the first post saying no health checker was being
used.

2009/3/31 Malcolm Turnbull <[email-masked]>

> I've never heard of this being a problem? 1000 doesn't sound like a lot of
> stale connections?
> Is their some more info on this?
>
> I have customers doing 50K conns/second with no problems.
>
>
>
>
> 2009/3/31 <[email-masked]>
>
>> We've seen this problem too; the only way I've found to clear that counter
>> is to pull a server from rotation and leave it pulled for 15-30 minutes.
>>
>> --
>> Jason Faulkner
>> Linux Systems Engineer
> Regards,
> Malcolm Turnbull.
> Loadbalancer.org Ltd.
> Phone: +44 (0)870 443 8779
> http://www.loadbalancer.org/
>

-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

---

From: Malcolm Turnbull <[email-masked]>
Date: Tue, 31 Mar 2009 15:13:56 +0100

-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.