|
Hi all
Just like to start by stating that the problem I explain here is totally
intermittent. Everything works fine 98% of the time.
I have a LVS Director listening to about 20 IP's, and forwarding the
requests for HTTP/HTTPS/SSH/FTP/POP3 etc to 7 different real servers,
Linux and Microsoft alike.
Not quite sure if I am correct, but I really think it is load related,
because it mostly only happens when my websites take quite a knock.
Here is a normal tcp connection, using tcpdump -i any host [client ip]
port 80 on the director :
16:55:34.382922 pc-2178249.unisa.ac.za.48700 > www2.unisa.ac.za.http: S
239044467:239044467(0) win 5840 <mss 1460,sackOK,timestamp 15489001
0,nop,wscale 7> (DF) # Syn packet from client to VIP on director
16:55:34.382946 pc-2178249.unisa.ac.za.48700 >
umweb2.cluster.unisa.ac.za.http: S 239044467:239044467(0) win 5840 <mss
1460,sackOK,timestamp 15489001 0,nop,wscale 7> (DF) # Ipvs rewrites
destination to RIP (same packet but now on internal network)
16:55:34.383027 umweb2.cluster.unisa.ac.za.http >
pc-2178249.unisa.ac.za.48700: S 424550733:424550733(0) ack 239044468 win
65535 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0,nop,nop,sackOK> (DF)
# RIP responds with a SYN/ACK
16:55:34.383041 www2.unisa.ac.za.http > pc-2178249.unisa.ac.za.48700: S
424550733:424550733(0) ack 239044468 win 65535 <mss 1460,nop,wscale
0,nop,nop,timestamp 0 0,nop,nop,sackOK> (DF) # And Ipvs rewrites the
source to reflect the VIP
16:55:34.383251 pc-2178249.unisa.ac.za.48700 > www2.unisa.ac.za.http: .
ack 1 win 46 <nop,nop,timestamp 15489001 0> (DF) # client ACK's the SYN,
and connection is established.
16:55:34.383265 pc-2178249.unisa.ac.za.48700 >
umweb2.cluster.unisa.ac.za.http: . ack 1 win 46 <nop,nop,timestamp
15489001 0> (DF) # Ipvs once again correctly rewrites the VIP to RIP
16:55:34.383497 pc-2178249.unisa.ac.za.48700 > www2.unisa.ac.za.http: P
1:104(103) ack 1 win 46 <nop,nop,timestamp 15489002 0> (DF) # client
sends data (HTTP GET request)
16:55:34.383511 pc-2178249.unisa.ac.za.48700 >
umweb2.cluster.unisa.ac.za.http: P 1:104(103) ack 1 win 46
<nop,nop,timestamp 15489002 0> (DF) # VIP to RIP
etc.....
And here is what happens every now and again when things go wrong:
16:55:35.406321 pc-2178249.unisa.ac.za.48704 > www2.unisa.ac.za.http: S
244216559:244216559(0) win 5840 <mss 1460,sackOK,timestamp 15490025
0,nop,wscale 7> (DF) # Client tries to connect to www2.unisa.ac.za by
sending a SYN packet
16:55:35.406340 pc-2178249.unisa.ac.za.48704 >
umweb2.cluster.unisa.ac.za.http: S 244216559:244216559(0) win 5840 <mss
1460,sackOK,timestamp 15490025 0,nop,wscale 7> (DF) # IPVS rewrites the
packet destination to the real server, umweb2.cluster.unisa.ac.za
16:55:35.406424 umweb2.cluster.unisa.ac.za.http >
pc-2178249.unisa.ac.za.48704: S 424874716:424874716(0) ack 244216560 win
65535 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0,nop,nop,sackOK> (DF)
# Real server responds correctly with a SYN accompanied by an ACK for
the original SYN
16:55:35.406521 ulweb4.unisa.ac.za.http > pc-2178249.unisa.ac.za.48704:
S 424874716:424874716(0) ack 244216560 win 65535 <mss 1460,nop,wscale
0,nop,nop,timestamp 0 0,nop,nop,sackOK> (DF) # But IPVS rewrites the
packet incorrectly, and now the packet seems to come from a different
host ???? The VIP it uses here is a valid VIP on the director, but there
is no reason why it should use this VIP, and not www2.unisa.ac.za, which
was the originally requested VIP in the first place...
16:55:35.406731 pc-2178249.unisa.ac.za.48704 > ulweb4.unisa.ac.za.http:
R 244216560:244216560(0) win 0 (DF) # Client gets a SYN/ACK from an
unknown host, not related to the original request, and therefore sends
the RESET back to the sending host
16:55:35.406760 pc-2178249.unisa.ac.za.48704 >
umweb2.cluster.unisa.ac.za.http: R 244216560:244216560(0) win 0 (DF) #
I'm guessing that iptables nat rewrites the RESET to route back to the
original sender.
This is causing hanging and timeouts from the clients (the whole
world). Really an urgent issue.
Does anyone have any advice?
Kind regards
Johan van den Berg
---------------------------------------------------------------------------
This message (and attachments) is subject to restrictions and a disclaimer.
Please refer to http://www.unisa.ac.za/disclaimer for full details.
---------------------------------------------------------------------------
<<<<gwavasig>>>>
<<<< gwavasig >>>>
|
