LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: Traffic to a "dead" server

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: Re: Traffic to a "dead" server
From: Casey Zacek <cz@xxxxxxxxxxxx>
Date: Fri, 20 Oct 2006 07:51:46 -0500
Joseph Mack NA3T wrote (at Thu, Oct 19, 2006 at 10:29:13AM -0700):
> On Thu, 19 Oct 2006, Casey Zacek wrote:
> 
> >Correct me if I'm wrong, please, but I believe what should have
> >happened was that all traffic should be directed to the only remaining
> >active RS once the failed RS gets deleted.
> 
> only new traffic. Read the HOWTO.

Ok, I read the HOWTO.  I must have missed the part you're referencing.
Maybe I'm being misunderstood.

On the director, I do this:

iptables -A PREROUTING -d 10.2.34.100 -p tcp -m tcp -j MARK --set-mark 0xb
ipvsadm -A -f 11 -s wlc -p 1200
ipvsadm -a -f 11 -r 10.2.34.5:0 -g -w 50
ipvsadm -a -f 11 -r 10.2.34.11:0 -g -w 50

On both RSes, I run this:

tcpdump -n -i eth0 host 10.2.34.254 port 80

On my workstation (10.2.34.254), I do this:

telnet 10.2.34.100 80

And I see my connection on 10.2.34.11's tcpdump output.  So, I've
established persistence to 10.2.34.11.  Now, on the director, I do
this:

ipvsadm -d -f 11 -r 10.2.34.11:0

And again, I telnet.  Now, I show up on 10.2.34.5, as I would expect,
but apparently I'm in the minority on this list. (?)

I believe the above simulates a failure in keepalived since I do not
use the inhibit_on_failure option.  The output of 'ipvsadm -lnf 11' at
various stages backs my belief.  All is fine with the world at this
point.

The problem I encountered was that at some point there was a
breakdown, and connections were showing up on the deleted Real Server
instead of the still-live Real Server.  I am not certain if any
traffic was making it to the still-live RS at all.  I know that any
attempt I made would terminate at the deleted RS.

As you all can imagine, this is a touchy issue with this customer now,
as my load-balancer solution failed him, so I cannot play around with
his solution in order to duplicate it.  I've never seen this behavior
in the 3-4(more?) years that I've been employing LVS.  I've seen
plenty of keepalived healtchecker freeze-ups, but never an apparent
bug in the IPVS kernel code.

-- 
Casey Zacek
Senior Engineer
NeoSpire, Inc.

<Prev in Thread] Current Thread [Next in Thread>