|
Hello,
We have met a quite troublesome situation which causes an internal SYN
storm.
The simplified version of the configuration consists of 2 servers - A
and B, both running Linux kernel 3.7.4-20.
Both have the IPVS software enabled, A is acting as the active load
balancer, B as a backup.
Both servers act as real servers also.
At some point, there is an incoming TCP connection from IPpair
(address:port) I.
The load balancer A decides to process it locally. Connection is
established, and the balancer status is distributed to server B via
syncing broadcast.
The client closes connection, and again the status is updated on B via
the broadcast - the connection is now in the "TCP_WAIT" state.
Pretty soon (within 10 seconds) the client opens the new TCP connection
using the same IP pair I.
It is not a good TCP practice, but nevertheless, some clients work this way.
This time the load balancer A decides that the connection is to be
handled on the server B (persistence is switched off).
The SYN packet is relayed to the server B, which finds an existing
routing record for that pair I.
And that record (in the CLOSE state) - points to the server A, and the
SYN packet is relayed there.
The server A processes it again, directs it to the server B again, and
the loop spirals, since the server B does not have the new connection
table element I synced.
We can send packet dumps illustrating the problem.
If our analysis is correct, what are the available workarounds?
a) we can always use "persistent" option with time larger than CLOSE
(TIME_WAIT?) state time.
b) on the server B we can remove the iptables records marking incoming
packets with a flag used with the IPVS uses.
We can insert those iptable rule(s) only when the server B becomes the
main load balancer. But will it stop IPVS from running all incoming
packets via its (synced) connections table?
--
Best regards,
Dmitry Akindinov
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
