|
Hi,
On one of our clusters we have problems with ipvs at the moment. Our
cluster is built with 2 front-end failover ipvs-nodes (managed with
ldirectord), with 3 Apache back-end nodes, handling both http as well as
https. So all the traffic on a virtual ip on port 80 or 443 of the
front-end servers is redirected to the backend webservers.
Two weeks ago, we were running a 2.6.8-2-686-smp Debian stable kernel,
containing ipvs 1.2.0. We experienced weekly (6 to 8 days) server
crashes, which caused the machines to hang completely without any
log-information whatsoever. These crashes seemed to be related to IPVS,
since all our servers have the exact same configuration, except for the
additional ipvs-modules on the front-end servers. Additionally, the same
Dell SC1425 servers are used for all servers.
For this reason we upgraded our kernel to 2.6.16-2-686-smp (containing
ipvs 1.2.1) on Debian stable, which we installed from backports
(http://www.backports.org). There aren't any crashes on these machines
anymore. However, there are two strange things we noticed since this
upgrade. First of all, the number of active connections has increased
dramatically, from 1,200 with a 2.6.8-2-686-smp kernel, to well over
30,000 with the new kernel. We are handling the same amount of traffic.
# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP XXX.net wlc persistent 120
-> apache1:https Route 10 2 0
-> apache2:https Route 10 25 0
-> apache3:https Route 10 14 0
TCP XXX.net wlc persistent 120
-> apache1:www Route 10 10928 13
-> apache2:www Route 10 11433 6
-> apache3:www Route 10 11764 10
We are using the following IPVS modules:
ip_vs
ip_vs_rr
ip_vs_wlc
Secondly, Internet Explorer users are experiencing problems exactly
since the upgrade to the new ipvs version. With Internet Explorer, an
enormous amount of tcp-connections is opened when visiting a website.
Users are experiencing high loads on their local machines, and crashing
Internet Explorers. With any version of FireFox this is working fine
by the way. Nevertheless, this started exactly since our IPVS upgrade.
I've added two tcpdumps at the bottom of this e-mail. The first one is
output of FireFox 1.0. The second is a tcpdump taken from the same
machine, using IE6. I'm sorry for the length of the attachments, but it
seems rather relevant.
Has anyone experienced the same behaviour, and is there any way to fix
this problem?
Regards,
Sebastiaan Tesink
TCPDUMP FireFox:
12:12:38.703125 IP sebas.1222 > XXX.net.80: F 1647804967:1647804967(0)
ack 3141814770 win 65322
12:12:39.734375 IP sebas.1262 > XXX.net.80: S 3507090758:3507090758(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:39.750000 IP XXX.net.80 > sebas.1262: S 3222090698:3222090698(0)
ack 3507090759 win 5840 <mss 1460,nop,nop,sackOK>
12:12:39.750000 IP sebas.1262 > XXX.net.80: . ack 1 win 65535
12:12:39.750000 IP sebas.1262 > XXX.net.80: P 1:632(631) ack 1 win 65535
12:12:39.765625 IP XXX.net.80 > sebas.1262: . ack 632 win 6941
12:12:39.968750 IP XXX.net.80 > sebas.1262: . 1:1461(1460) ack 632 win 6941
12:12:39.968750 IP XXX.net.80 > sebas.1262: . 1461:2921(1460) ack 632
win 6941
12:12:39.968750 IP sebas.1262 > XXX.net.80: . ack 2921 win 65535
12:12:40.000000 IP XXX.net.80 > sebas.1262: . 2921:4381(1460) ack 632
win 6941
12:12:40.000000 IP XXX.net.80 > sebas.1262: . 4381:5841(1460) ack 632
win 6941
12:12:40.000000 IP sebas.1262 > XXX.net.80: . ack 5841 win 65535
12:12:40.000000 IP XXX.net.80 > sebas.1262: . 5841:7301(1460) ack 632
win 6941
12:12:40.015625 IP XXX.net.80 > sebas.1262: . 7301:8761(1460) ack 632
win 6941
12:12:40.015625 IP sebas.1262 > XXX.net.80: . ack 8761 win 65535
12:12:40.015625 IP XXX.net.80 > sebas.1262: . 8761:10221(1460) ack 632
win 6941
12:12:40.015625 IP XXX.net.80 > sebas.1262: . 10221:11681(1460) ack 632
win 6941
12:12:40.015625 IP sebas.1262 > XXX.net.80: . ack 11681 win 65535
12:12:40.031250 IP sebas.1263 > XXX.net.80: S 3728419719:3728419719(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:40.031250 IP XXX.net.80 > sebas.1262: P 11681:13141(1460) ack 632
win 6941
12:12:40.046875 IP XXX.net.80 > sebas.1262: P 13141:14201(1060) ack 632
win 6941
12:12:40.046875 IP sebas.1262 > XXX.net.80: . ack 14201 win 65535
12:12:40.046875 IP XXX.net.80 > sebas.1263: S 3222009191:3222009191(0)
ack 3728419720 win 5840 <mss 1460,nop,nop,sackOK>
12:12:40.046875 IP sebas.1263 > XXX.net.80: . ack 1 win 65535
12:12:40.140625 IP sebas.1263 > XXX.net.80: P 1:621(620) ack 1 win 65535
12:12:40.156250 IP XXX.net.80 > sebas.1263: . ack 621 win 6820
12:12:40.187500 IP XXX.net.80 > sebas.1263: P 1:1010(1009) ack 621 win 6820
TCPDUMP Internet Explorer 6:
12:12:13.531250 IP sebas.1229 > XXX.net.80: S 470549542:470549542(0) win
65535 <mss 1460,nop,nop,sackOK>
12:12:13.546875 IP XXX.net.80 > sebas.1229: S 3201505166:3201505166(0)
ack 470549543 win 5840 <mss 1460,nop,nop,sackOK>
12:12:13.546875 IP sebas.1229 > XXX.net.80: . ack 1 win 65535
12:12:13.546875 IP sebas.1229 > XXX.net.80: P 1:364(363) ack 1 win 65535
12:12:13.562500 IP XXX.net.80 > sebas.1229: . ack 364 win 6432
12:12:13.781250 IP XXX.net.80 > sebas.1229: . 1:1461(1460) ack 364 win 6432
12:12:13.781250 IP XXX.net.80 > sebas.1229: . 1461:2921(1460) ack 364
win 6432
12:12:13.781250 IP sebas.1229 > XXX.net.80: . ack 2921 win 65535
12:12:13.796875 IP XXX.net.80 > sebas.1229: . 2921:4381(1460) ack 364
win 6432
12:12:13.796875 IP XXX.net.80 > sebas.1229: . 4381:5841(1460) ack 364
win 6432
12:12:13.796875 IP sebas.1229 > XXX.net.80: . ack 5841 win 65535
12:12:13.796875 IP XXX.net.80 > sebas.1229: . 5841:7301(1460) ack 364
win 6432
12:12:13.812500 IP XXX.net.80 > sebas.1229: . 7301:8761(1460) ack 364
win 6432
12:12:13.812500 IP sebas.1229 > XXX.net.80: . ack 8761 win 65535
12:12:13.828125 IP XXX.net.80 > sebas.1229: . 8761:10221(1460) ack 364
win 6432
12:12:13.828125 IP XXX.net.80 > sebas.1229: . 10221:11681(1460) ack 364
win 6432
12:12:13.828125 IP sebas.1229 > XXX.net.80: . ack 11681 win 64075
12:12:13.828125 IP sebas.1229 > XXX.net.80: . ack 11681 win 65535
12:12:13.843750 IP sebas.1230 > XXX.net.80: S 7002404:7002404(0) win
65535 <mss 1460,nop,nop,sackOK>
12:12:13.843750 IP XXX.net.80 > sebas.1229: P 11681:13141(1460) ack 364
win 6432
12:12:13.843750 IP XXX.net.80 > sebas.1229: FP 13141:14590(1449) ack 364
win 6432
12:12:13.843750 IP sebas.1229 > XXX.net.80: . ack 14591 win 65535
12:12:13.859375 IP XXX.net.80 > sebas.1230: S 3197109873:3197109873(0)
ack 7002405 win 5840 <mss 1460,nop,nop,sackOK>
12:12:13.859375 IP sebas.1230 > XXX.net.80: . ack 1 win 65535
12:12:13.859375 IP sebas.1230 > XXX.net.80: P 1:397(396) ack 1 win 65535
12:12:13.875000 IP XXX.net.80 > sebas.1230: . ack 397 win 6432
12:12:13.875000 IP XXX.net.80 > sebas.1230: . 1:1461(1460) ack 397 win 6432
12:12:13.875000 IP XXX.net.80 > sebas.1230: P 1461:1753(292) ack 397 win
6432
12:12:13.875000 IP sebas.1230 > XXX.net.80: . ack 1753 win 65535
12:12:13.875000 IP XXX.net.80 > sebas.1230: F 1753:1753(0) ack 397 win 6432
12:12:13.875000 IP sebas.1230 > XXX.net.80: . ack 1754 win 65535
12:12:13.921875 IP sebas.1230 > XXX.net.80: F 397:397(0) ack 1754 win 65535
12:12:13.937500 IP XXX.net > sebas: ICMP XXX.net tcp port 80
unreachable, length 48
12:12:13.937500 IP sebas.1229 > XXX.net.80: F 364:364(0) ack 14591 win 65535
12:12:13.984375 IP sebas.1231 > XXX.net.80: S 1193754119:1193754119(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:14.015625 IP XXX.net.80 > sebas.1231: S 3207290837:3207290837(0)
ack 1193754120 win 5840 <mss 1460,nop,nop,sackOK>
12:12:14.015625 IP sebas.1231 > XXX.net.80: . ack 1 win 65535
12:12:14.015625 IP sebas.1231 > XXX.net.80: P 1:378(377) ack 1 win 65535
12:12:14.031250 IP XXX.net.80 > sebas.1231: . ack 378 win 6432
12:12:14.031250 IP XXX.net.80 > sebas.1231: . 1:1461(1460) ack 378 win 6432
12:12:14.031250 IP XXX.net.80 > sebas.1231: . 1461:2921(1460) ack 378
win 6432
12:12:14.031250 IP sebas.1231 > XXX.net.80: . ack 2921 win 65535
12:12:14.062500 IP XXX.net.80 > sebas.1231: . 2921:4381(1460) ack 378
win 6432
12:12:14.062500 IP XXX.net.80 > sebas.1231: FP 4381:5428(1047) ack 378
win 6432
12:12:14.062500 IP sebas.1231 > XXX.net.80: . ack 5429 win 65535
12:12:14.062500 IP sebas.1231 > XXX.net.80: F 378:378(0) ack 5429 win 65535
12:12:14.078125 IP XXX.net > sebas: ICMP XXX.net tcp port 80
unreachable, length 48
12:12:14.140625 IP sebas.1232 > XXX.net.80: S 3056504619:3056504619(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:14.171875 IP XXX.net.80 > sebas.1232: S 3205156333:3205156333(0)
ack 3056504620 win 5840 <mss 1460,nop,nop,sackOK>
12:12:14.171875 IP sebas.1232 > XXX.net.80: . ack 1 win 65535
12:12:14.171875 IP sebas.1232 > XXX.net.80: P 1:376(375) ack 1 win 65535
12:12:14.187500 IP XXX.net.80 > sebas.1232: . ack 376 win 6432
12:12:14.187500 IP XXX.net.80 > sebas.1232: . 1:1461(1460) ack 376 win 6432
12:12:14.203125 IP XXX.net.80 > sebas.1232: . 1461:2921(1460) ack 376
win 6432
12:12:14.203125 IP sebas.1232 > XXX.net.80: . ack 2921 win 65535
12:12:14.203125 IP XXX.net.80 > sebas.1232: FP 2921:3825(904) ack 376
win 6432
12:12:14.203125 IP sebas.1232 > XXX.net.80: . ack 3826 win 64631
12:12:14.203125 IP sebas.1232 > XXX.net.80: F 376:376(0) ack 3826 win 64631
12:12:14.218750 IP XXX.net > sebas: ICMP XXX.net tcp port 80
unreachable, length 48
12:12:14.656250 IP sebas.1234 > XXX.net.80: S 2725398837:2725398837(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:14.656250 IP sebas.1235 > XXX.net.80: S 2593430886:2593430886(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:14.671875 IP XXX.net.80 > sebas.1234: S 3201201943:3201201943(0)
ack 2725398838 win 5840 <mss 1460,nop,nop,sackOK>
12:12:14.671875 IP sebas.1234 > XXX.net.80: . ack 1 win 65535
12:12:14.671875 IP sebas.1234 > XXX.net.80: P 1:544(543) ack 1 win 65535
12:12:14.671875 IP XXX.net.80 > sebas.1235: S 3206352611:3206352611(0)
ack 2593430887 win 5840 <mss 1460,nop,nop,sackOK>
12:12:14.671875 IP sebas.1235 > XXX.net.80: . ack 1 win 65535
12:12:14.671875 IP sebas.1235 > XXX.net.80: P 1:540(539) ack 1 win 65535
12:12:14.687500 IP sebas.1236 > XXX.net.80: S 3033031558:3033031558(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:14.687500 IP XXX.net.80 > sebas.1234: . ack 544 win 6516
12:12:14.703125 IP XXX.net.80 > sebas.1235: . ack 540 win 6468
12:12:14.703125 IP XXX.net.80 > sebas.1236: S 3211575606:3211575606(0)
ack 3033031559 win 5840 <mss 1460,nop,nop,sackOK>
12:12:14.703125 IP sebas.1236 > XXX.net.80: . ack 1 win 65535
12:12:14.703125 IP sebas.1236 > XXX.net.80: P 1:555(554) ack 1 win 65535
12:12:14.734375 IP XXX.net.80 > sebas.1236: . ack 555 win 6648
12:12:14.859375 IP sebas.1233 > 64.233.183.147.80: . ack 5875 win 65206
12:12:15.359375 IP XXX.net.80 > sebas.1234: P 1:353(352) ack 544 win 6516
12:12:15.375000 IP XXX.net.80 > sebas.1234: . 353:1813(1460) ack 544 win
6516
12:12:15.375000 IP sebas.1234 > XXX.net.80: . ack 1813 win 65535
12:12:15.390625 IP XXX.net.80 > sebas.1234: . 1813:3273(1460) ack 544
win 6516
12:12:15.390625 IP XXX.net.80 > sebas.1234: FP 3273:4603(1330) ack 544
win 6516
12:12:15.390625 IP sebas.1234 > XXX.net.80: . ack 4604 win 65535
12:12:15.390625 IP sebas.1234 > XXX.net.80: F 544:544(0) ack 4604 win 65535
12:12:15.390625 IP XXX.net.80 > sebas.1235: P 1:353(352) ack 540 win 6468
12:12:15.390625 IP sebas.1237 > XXX.net.80: S 1005722073:1005722073(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:15.390625 IP XXX.net.80 > sebas.1235: . 353:1813(1460) ack 540 win
6468
12:12:15.390625 IP sebas.1235 > XXX.net.80: . ack 1813 win 65535
12:12:15.406250 IP XXX.net > sebas: ICMP XXX.net tcp port 80
unreachable, length 48
12:12:15.406250 IP XXX.net.80 > sebas.1237: S 3206601363:3206601363(0)
ack 1005722074 win 5840 <mss 1460,nop,nop,sackOK>
12:12:15.406250 IP sebas.1237 > XXX.net.80: . ack 1 win 65535
12:12:15.406250 IP sebas.1237 > XXX.net.80: P 1:553(552) ack 1 win 65535
12:12:15.406250 IP XXX.net.80 > sebas.1235: . 1813:3273(1460) ack 540
win 6468
12:12:15.421875 IP XXX.net.80 > sebas.1235: . 3273:4733(1460) ack 540
win 6468
12:12:15.421875 IP sebas.1235 > XXX.net.80: . ack 4733 win 65535
12:12:15.421875 IP XXX.net.80 > sebas.1235: . 4733:6193(1460) ack 540
win 6468
12:12:15.437500 IP XXX.net.80 > sebas.1237: . ack 553 win 6624
12:12:15.437500 IP XXX.net.80 > sebas.1235: . 6193:7653(1460) ack 540
win 6468
12:12:15.437500 IP sebas.1235 > XXX.net.80: . ack 7653 win 65535
12:12:15.437500 IP XXX.net.80 > sebas.1235: FP 7653:8354(701) ack 540
win 6468
12:12:15.437500 IP sebas.1235 > XXX.net.80: . ack 8355 win 64834
12:12:15.437500 IP sebas.1235 > XXX.net.80: F 540:540(0) ack 8355 win 64834
12:12:15.437500 IP sebas.1238 > XXX.net.80: S 3576365933:3576365933(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:15.453125 IP XXX.net > sebas: ICMP XXX.net tcp port 80
unreachable, length 48
12:12:15.453125 IP XXX.net.80 > sebas.1236: P 1:973(972) ack 555 win 6648
12:12:15.453125 IP XXX.net.80 > sebas.1236: F 973:973(0) ack 555 win 6648
12:12:15.453125 IP sebas.1236 > XXX.net.80: . ack 974 win 64563
12:12:15.453125 IP XXX.net.80 > sebas.1238: S 3198594519:3198594519(0)
ack 3576365934 win 5840 <mss 1460,nop,nop,sackOK>
12:12:15.453125 IP sebas.1238 > XXX.net.80: . ack 1 win 65535
12:12:15.453125 IP sebas.1236 > XXX.net.80: F 555:555(0) ack 974 win 64563
12:12:15.468750 IP sebas.1238 > XXX.net.80: P 1:546(545) ack 1 win 65535
12:12:15.484375 IP XXX.net > sebas: ICMP XXX.net tcp port 80
unreachable, length 48
12:12:15.484375 IP XXX.net.80 > sebas.1238: . ack 546 win 6540
12:12:15.812500 IP sebas.1240 > XXX.net.80: S 1405020268:1405020268(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:15.828125 IP XXX.net.80 > sebas.1240: S 3207699759:3207699759(0)
ack 1405020269 win 5840 <mss 1460,nop,nop,sackOK>
12:12:15.828125 IP sebas.1240 > XXX.net.80: . ack 1 win 65535
12:12:15.828125 IP sebas.1240 > XXX.net.80: P 1:500(499) ack 1 win 65535
12:12:15.843750 IP XXX.net.80 > sebas.1240: . ack 500 win 6432
12:12:16.250000 IP XXX.net.80 > sebas.1237: P 1:352(351) ack 553 win 6624
12:12:16.265625 IP XXX.net.80 > sebas.1237: . 352:1812(1460) ack 553 win
6624
12:12:16.265625 IP sebas.1237 > XXX.net.80: . ack 1812 win 65535
12:12:16.281250 IP sebas.1229 > XXX.net.80: F 364:364(0) ack 14591 win 65535
12:12:16.281250 IP XXX.net.80 > sebas.1237: P 1812:2699(887) ack 553 win
6624
12:12:16.281250 IP sebas.1237 > XXX.net.80: F 553:553(0) ack 2699 win 64648
12:12:16.281250 IP sebas.1242 > XXX.net.80: S 3500530823:3500530823(0)
win 65535 <mss 1460,nop,nop,sackOK>
12:12:16.281250 IP XXX.net.80 > sebas.1238: P 1:352(351) ack 546 win 6540
12:12:16.281250 IP XXX.net.80 > sebas.1238: . 352:1812(1460) ack 546 win
6540
12:12:16.281250 IP sebas.1238 > XXX.net.80: . ack 1812 win 65535
12:12:16.296875 IP XXX.net > sebas: ICMP XXX.net tcp port 80
unreachable, length 48
12:12:16.296875 IP XXX.net.80 > sebas.1242: S 3204250979:3204250979(0)
ack 3500530824 win 5840 <mss 1460,nop,nop,sackOK>
12:12:16.296875 IP sebas.1242 > XXX.net.80: . ack 1 win 65535
12:12:16.296875 IP sebas.1242 > XXX.net.80: P 1:550(549) ack 1 win 65535
12:12:16.312500 IP XXX.net.80 > sebas.1238: . 1812:3272(1460) ack 546
win 6540
12:12:16.312500 IP XXX.net.80 > sebas.1238: P 3272:3583(311) ack 546 win
6540
12:12:16.312500 IP sebas.1238 > XXX.net.80: . ack 3583 win 65535
12:12:16.312500 IP sebas.1238 > XXX.net.80: F 546:546(0) ack 3583 win 65535
12:12:16.312500 IP sebas.1243 > XXX.net.80: S 449513386:449513386(0) win
65535 <mss 1460,nop,nop,sackOK>
12:12:16.328125 IP XXX.net.80 > sebas.1238: F 3583:3583(0) ack 546 win 6540
12:12:16.328125 IP sebas.1238 > XXX.net.80: . ack 3584 win 65535
12:12:16.328125 IP XXX.net.80 > sebas.1242: . ack 550 win 6588
12:12:16.328125 IP XXX.net > sebas: ICMP XXX.net tcp port 80
unreachable, length 48
12:12:16.328125 IP XXX.net.80 > sebas.1243: S 3205346986:3205346986(0)
ack 449513387 win 5840 <mss 1460,nop,nop,sackOK>
12:12:16.328125 IP sebas.1243 > XXX.net.80: . ack 1 win 65535
12:12:16.328125 IP sebas.1243 > XXX.net.80: P 1:550(549) ack 1 win 65535
12:12:16.359375 IP XXX.net.80 > sebas.1243: . ack 550 win 6588
12:12:16.390625 IP sebas.1231 > XXX.net.80: F 378:378(0) ack 5429 win 65535
12:12:16.390625 IP sebas.1230 > XXX.net.80: F 397:397(0) ack 1754 win 65535
12:12:16.500000 IP XXX.net.80 > sebas.1240: P 1:605(604) ack 500 win 6432
12:12:16.500000 IP XXX.net.80 > sebas.1240: F 605:605(0) ack 500 win 6432
12:12:16.500000 IP sebas.1240 > XXX.net.80: . ack 606 win 64931
12:12:16.500000 IP sebas.1240 > XXX.net.80: F 500:500(0) ack 606 win 64931
--
E-mail: sebas@xxxxxxxxxxxxxxxxxx
Website: http://www.virtualconcepts.nl
Phone: +31 (0)6 1111 4539
|
