LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

[lvs-users] SYN spiraling between master and slave IPVS balancers

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: [lvs-users] SYN spiraling between master and slave IPVS balancers
From: Dmitry Akindinov <dimak@xxxxxxxxxxx>
Date: Tue, 05 Feb 2013 20:45:12 +0400
Hello,

We have met a quite troublesome situation which causes an internal SYN 
storm.

The simplified version of the configuration consists of 2 servers - A 
and B, both running Linux kernel 3.7.4-20.

Both have the IPVS software enabled, A is acting as the active load 
balancer, B as a backup.
Both servers act as real servers also.

At some point, there is an incoming TCP connection from IPpair 
(address:port) I.
The load balancer A decides to process it locally. Connection is 
established, and the balancer status is distributed to server B via 
syncing broadcast.

The client closes connection, and again the status is updated on B via 
the broadcast - the connection is now in the "TCP_WAIT" state.

Pretty soon (within 10 seconds) the client opens the new TCP connection 
using the same IP pair I.
It is not a good TCP practice, but nevertheless, some clients work this way.

This time the load balancer A decides that the connection is to be 
handled on the server B (persistence is switched off).
The SYN packet is relayed to the server B, which finds an existing 
routing record for that pair I.
And that record (in the CLOSE state) - points to the server A, and the 
SYN packet is relayed there.

The server A processes it again, directs it to the server B again, and 
the loop spirals, since the server B does not have the new connection 
table element I synced.

We can send packet dumps illustrating the problem.

If our analysis is correct, what are the available workarounds?
a) we can always use "persistent" option with time larger than CLOSE 
(TIME_WAIT?) state time.
b) on the server B we can remove the iptables records marking incoming 
packets with a flag used with the IPVS uses.
We can insert those iptable rule(s) only when the server B becomes the 
main load balancer. But will it stop IPVS from running all incoming 
packets via its (synced) connections table?

-- 
Best regards,
Dmitry Akindinov

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>