[lvs-users] ECMP Active-Active Setup - Failover issues

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: [lvs-users] ECMP Active-Active Setup - Failover issues
From: Poh Chiat <poh.chiat@xxxxxxxxx>
Date: Fri, 1 Jun 2018 11:38:56 +0900

I'm trying this setup where an upstream router ECMPs to multiple
active-active LVS nodes with the exact same configuration.

--- | router | -- ECMP --> | LVS nodes | -- IPinIP --> | HAProxy nodes |

The LVS nodes are configured as such:
1. BIRD to advertise VIP to upstream router
2. ipvs setup with VIP and sh scheduler
3. ipvs uses TUN forwarding method
4. loopback interface has VIP

LVS load balances to some real servers with a tunnel interface setup.
The real servers will direct return to the client.

In one of my tests, when I stop BIRD and hence drop the route to node
A, the upstream router starts forwarding packets to node B. So far so

What I noticed then is that node B will send TCP RST packets back to
the client. Instead, what I hoped to see was that
1. node B receives the packet
2. sh scheduler means that the same real server picked by node A will
also be picked by node B
3. the same real server receives the packet and the connection stays up

When I enable daemon state sync between node A and B the connection
stays up. However, state sync happens periodically and so it is still
possible for some connections to be dropped during this interval.

>From the above observation, my guess is that ipvs evaluated the packet
(perhaps in ip_vs_in()), but somehow managed it differently because
the connection does not exist in the table.

What I hope to understand is
1. is ipvs sending the RST? or is that somewhere else in the netfilter system?
2. since sh scheduler (and the recent maglev scheduler patch) are
consistent hashes, they don't actually have a state to sync (except
for purely optimization purposes). Is it reasonable to hope that we do
not need state sync to get this working at all?
3. should I be setting up this in a different way?

Forgive me if I have gaps in my knowledge of the Linux networking
stack and ipvs's hook with netfilter (and the schedulers and
connection tracking too).

Thank you.

Please read the documentation before posting - it's available at: mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to

<Prev in Thread] Current Thread [Next in Thread>