|
Hi,
I use an IPVS balanced database cluster (the setup is described in my previous
thread "Mysterious packet drops in a IPVS-DR setup").
From time to time, my director sends a tcp reset packet into an established
connection, thus terminating the connection on the initiators side. The
balanced realserver doesn't know anything about the reset packet and tries to
resend its last unacknowledged tcp packet until it times out.
Below are the dumps from the database server, the database load balancer and
the webserver, respectivly.
Handshake and initial communication is normal until 12:17:57.493527 where the
database server sends a large packet with 2896 bytes.
It is fragmented and arrives at the webserver at timecode 12:17:57.727493 /
12:17:57.727494.
An acknowledge is sent at 12:17:57.729146 which arrives at the loadbalancer at
12:18:05.873445. But instead of sending the ack packet to the database
server, the loadbalancer issues a TCP reset at 12:18:05.873464!!!
The reset immediatly closes the connection on the webserver, but it's still
open on the database server which now tries to resend the packet for a while.
This happens not on every connection with 2896 byte large packets, but if it
happens, is is always the case that the director sends the tcp reset as a
reply to the ack packet which should acknowledge the arrival of this 2896
byte packet.
Any ideas?
Greetings & merry xmas,
Jan
TCPDUMPS:
DATABASE SERVER
12:17:49.546766 IP webserver.57289 > dbv1.mysql: S 3446126927:3446126927(0)
win 5840 <mss 1460,sackOK,timestamp 667148057[|tcp]>
12:17:49.546790 IP dbv1.mysql > webserver.57289: S 2926799505:2926799505(0)
ack 3446126928 win 5792 <mss 1460,sackOK,timestamp 50890978[|tcp]>
12:17:49.547016 IP webserver.57289 > dbv1.mysql: . ack 1 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.547248 IP dbv1.mysql > webserver.57289: P 1:61(60) ack 1 win 1448
<nop,nop,timestamp 50890978 667148057>
12:17:49.547516 IP webserver.57289 > dbv1.mysql: . ack 61 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.548016 IP webserver.57289 > dbv1.mysql: P 1:73(72) ack 61 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.548044 IP dbv1.mysql > webserver.57289: . ack 73 win 1448
<nop,nop,timestamp 50890978 667148057>
12:17:49.548120 IP dbv1.mysql > webserver.57289: P 61:66(5) ack 73 win 1448
<nop,nop,timestamp 50890978 667148057>
12:17:49.548515 IP webserver.57289 > dbv1.mysql: P 73:86(13) ack 66 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.548589 IP dbv1.mysql > webserver.57289: P 66:77(11) ack 86 win 1448
<nop,nop,timestamp 50890978 667148057>
12:17:49.549014 IP webserver.57289 > dbv1.mysql: P 86:148(62) ack 77 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.586863 IP dbv1.mysql > webserver.57289: . ack 148 win 1448
<nop,nop,timestamp 50890982 667148057>
12:17:57.493527 IP dbv1.mysql > webserver.57289: . 77:2973(2896) ack 148 win
1448 <nop,nop,timestamp 50891772 667148057>
12:17:57.697127 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50891793 667148057>
12:17:58.117119 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50891835 667148057>
12:17:58.957131 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50891919 667148057>
12:18:00.637176 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50892087 667148057>
12:18:03.997280 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50892423 667148057>
12:18:10.717515 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50893095 667148057>
12:18:24.157871 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50894439 667148057>
12:18:51.038660 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50897127 667148057>
LOADBALANCER
12:17:57.924726 IP webserver.57289 > dbv1.mysql: S 3446126927:3446126927(0)
win 5840 <mss 1460,sackOK,timestamp 667148057[|tcp]>
12:17:57.924737 IP webserver.57289 > dbv1.mysql: S 3446126927:3446126927(0)
win 5840 <mss 1460,sackOK,timestamp 667148057[|tcp]>
12:17:57.924976 IP webserver.57289 > dbv1.mysql: . ack 2926799506 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:57.924984 IP webserver.57289 > dbv1.mysql: . ack 1 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:57.925476 IP webserver.57289 > dbv1.mysql: . ack 61 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:57.925484 IP webserver.57289 > dbv1.mysql: . ack 61 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:57.925975 IP webserver.57289 > dbv1.mysql: P 0:72(72) ack 61 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:57.925984 IP webserver.57289 > dbv1.mysql: P 0:72(72) ack 61 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:57.926475 IP webserver.57289 > dbv1.mysql: P 72:85(13) ack 66 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:57.926483 IP webserver.57289 > dbv1.mysql: P 72:85(13) ack 66 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:57.926975 IP webserver.57289 > dbv1.mysql: P 85:147(62) ack 77 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:57.926982 IP webserver.57289 > dbv1.mysql: P 85:147(62) ack 77 win 1460
<nop,nop,timestamp 667148057 50890978>
12:18:05.873445 IP webserver.57289 > dbv1.mysql: . ack 2973 win 2908
<nop,nop,timestamp 667150044 50891772>
12:18:05.873464 IP dbv1.mysql > webserver.57289: R 2926802478:2926802478(0)
win 0
12:18:06.075446 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:06.495436 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:07.335293 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:09.015381 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:12.375557 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:19.095660 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:32.535863 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:59.416525 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:19:53.177589 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:21:40.699975 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:23:40.692661 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:25:40.695093 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:27:40.687656 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:29:40.690213 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:31:40.692644 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
WEBSERVER
12:17:49.779457 IP webserver.57289 > dbv1.mysql: S 3446126927:3446126927(0)
win 5840 <mss 1460,sackOK,timestamp 667148057[|tcp]>
12:17:49.779723 IP dbv1.mysql > webserver.57289: S 2926799505:2926799505(0)
ack 3446126928 win 5792 <mss 1460,sackOK,timestamp 50890978[|tcp]>
12:17:49.779744 IP webserver.57289 > dbv1.mysql: . ack 1 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.780222 IP dbv1.mysql > webserver.57289: P 1:61(60) ack 1 win 1448
<nop,nop,timestamp 50890978 667148057>
12:17:49.780247 IP webserver.57289 > dbv1.mysql: . ack 61 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.780690 IP webserver.57289 > dbv1.mysql: P 1:73(72) ack 61 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.780971 IP dbv1.mysql > webserver.57289: . ack 73 win 1448
<nop,nop,timestamp 50890978 667148057>
12:17:49.781096 IP dbv1.mysql > webserver.57289: P 61:66(5) ack 73 win 1448
<nop,nop,timestamp 50890978 667148057>
12:17:49.781136 IP webserver.57289 > dbv1.mysql: P 73:86(13) ack 66 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.781595 IP dbv1.mysql > webserver.57289: P 66:77(11) ack 86 win 1448
<nop,nop,timestamp 50890978 667148057>
12:17:49.781712 IP webserver.57289 > dbv1.mysql: P 86:148(62) ack 77 win 1460
<nop,nop,timestamp 667148057 50890978>
12:17:49.819829 IP dbv1.mysql > webserver.57289: . ack 148 win 1448
<nop,nop,timestamp 50890982 667148057>
12:17:57.727493 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50891772 667148057>
12:17:57.727494 IP dbv1.mysql > webserver.57289: . 1525:2973(1448) ack 148 win
1448 <nop,nop,timestamp 50891772 667148057>
12:17:57.729146 IP webserver.57289 > dbv1.mysql: . ack 2973 win 2908
<nop,nop,timestamp 667150044 50891772>
12:17:57.729366 IP dbv1.mysql > webserver.57289: R 2926802478:2926802478(0)
win 0
12:17:57.931150 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50891793 667148057>
12:17:57.931166 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:17:58.351209 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50891835 667148057>
12:17:58.351227 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:17:59.191204 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50891919 667148057>
12:17:59.191233 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:00.871440 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50892087 667148057>
12:18:00.871458 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:04.232041 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50892423 667148057>
12:18:04.232058 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:10.952993 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50893095 667148057>
12:18:10.953025 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:24.394894 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50894439 667148057>
12:18:24.394914 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
12:18:51.278826 IP dbv1.mysql > webserver.57289: . 77:1525(1448) ack 148 win
1448 <nop,nop,timestamp 50897127 667148057>
12:18:51.278858 IP webserver.57289 > dbv1.mysql: R 3446127075:3446127075(0)
win 0
|
