LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: Busted Cluster

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: Busted Cluster
From: Johan van den Berg <vdberj@xxxxxxxxxxx>
Date: Mon, 14 Mar 2005 11:17:33 +0200
nigel@xxxxxxxxxxx wrote:

     Although, periodically we notice that an incoming request to the cluster 
hangs forever. Has anyone else experienced this? Is it LVS related? But this is 
a relatively small problem.
We have the exact same issue with NAT. We've pinpointed it to packets that are being rerouted back from the real server to the client incorrectly. When a packet arrives at the director, it stores a connection entry in ipvs, rewrites the packet and sends it on towards a real server. When the real server responds, the director is supposed to do a lookup in the ipvs connection table, and rewrite and forward the packet to the client to whom the connection exists. This works most of the time...

Sometimes though, even though the connection entry is actually in the connection table on the director, the lookup fails, and the packet falls through ipvs, and ends up in the normal iptables filter, meaning that, depending on if you director forwards outbound packets from the real servers using NAT, or if you are dropping packets from the real servers not assosciated with ipvs, your packet might end up at the client with the wrong source ip or not end up there at all (hence the hangs).

To get an idea if this happens, simply do a tcpdump on your director, looking for the tcp reset flag (see the iptables man page on how to do this). If you find that a lot of clients are sending you resets, then most likely the client received a response from a real server with the wrong ip, and therefore rejects the connection, or, your director is sending resets to your real servers for connections that fell through ipvs.

See the following list archives for some more info:

http://www.in-addr.de/pipermail/lvs-users/2005-March/013450.html
http://www.in-addr.de/pipermail/lvs-users/2005-February/013315.html


Kind regards
Johan van den Berg

---------------------------------------------------------------------------
This message (and attachments) is subject to restrictions and a disclaimer. Please refer to http://www.unisa.ac.za/disclaimer for full details.
---------------------------------------------------------------------------
<<<<gwavasig>>>>
<<<< gwavasig >>>>




<Prev in Thread] Current Thread [Next in Thread>