LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] LVS-DR Cluster Some Real Servers Stuck in SYN_RECV

To: Bruce Rudolph <brudolph@xxxxxxxxxxx>
Subject: Re: [lvs-users] LVS-DR Cluster Some Real Servers Stuck in SYN_RECV
Cc: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
From: Julian Anastasov <ja@xxxxxx>
Date: Sat, 1 Mar 2014 21:01:46 +0200 (EET)
        Hello,

On Sat, 1 Mar 2014, Bruce Rudolph wrote:

> My current findings.
> 
> The overall LVS cluster is working at a degraded performance because 
> four of the five real servers are failing. The failure is strange. When 
> a client sends a request to the VIP (Virtual IP address) the LVS 
> Director (load balancer) distributes it to one of the real servers based 
> on the scheduling algorithm (LC).
> 
> Legend for the examples
> 
>     VIP = Virtual IP Address for the LVS cluster
>     DIR = the LVS Director or Load Balancer
>     RS = Real Server - the web service we have running listening on port 80
> 
> 
> The servers that are failing are doing so because of the following sequence:
> ERROR SEQUENCE
> 
>     Client sends SYN to VIP
>     DIR forwards SYN to an available RS
>     RS receives the SYN and responds to Client with SYN-ACK

        If there is reponse, check on real server that
it is correct:

1. It should contain VIP in saddr in IP header. This is expected
because director should send the request to real server
with VIP in daddr. Also, the client should see the same
server port (vport) in the response.

2. 'tcpdump -lennn src host VIP' on real server can show
to which destination MAC is sent the response

3. If it is going via director you can notice it with
tcpdump also on director. I guess, DR setups do not use
director for responses, otherwise they would use NAT mode
to avoid the source spoofing checks. I guess all your
real servers use same default gateway.

>     Client does not receive the SYN-ACK so it never sends an ACK. It
>     continues to send a SYN trying to establish a connection until the
>     timeout. THIS IS THE FAILURE POINT.

Regards

--
Julian Anastasov <ja@xxxxxx>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>