|
On 26.11.11 21:50, Joseph Mack NA3T wrote:
> On Sat, 26 Nov 2011, Alois Treindl wrote:
>
>> The httpd real server introduces 200 millisecond delays
>> between packets when sending.
> ^^^^^^^
> I assume you mean "replying"
yes. replying to the client's http request, trasnmitting data to the client
>
> Whenever I get this it's something like
>
> o the routing is wrong (are you getting a shower of icmp
> packets?)
how would I see them with tcpdump?
> o the httpd is phoning home (asking for a service on the
> internet eg DNS) that has to time out
turned this off, I think. httpd is logging without using DNS, nothing is
running on real servers except the rare requests from my LVS test access.
As I said, I can remove the LVS router and can connect a cable directly
and then I get high performance.
I try to look into TCP_NODELAY because the 200 msec point in that direction.
Your help is really appreciated.
I have tried to compile ipvsadm 1.26 but it has too many problems
compiling on my RHEL6. I have to stick with the CentOs 6 ipvsadm 1.25.9
I begin to despair because I have a fixed setup date for the new cluster
to keep early next week, and I cannot test it when the LVS balancer is
not working.
---------
Here is a fraction of tcpdump, as seen on the client, started off
on the client by wget http://LVS/large_file_name
as80.stro.ch is the client, lv2 the LVS
19:51:25.610063 IP as80.astro.ch.39594 > lv2.astro.com.http: Flags [S],
seq 3584270897, win 5840, options [mss 1460,sackOK,TS val 432865933 ecr
0,nop,wscale 7], length 0
19:52:45.207361 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [S],
seq 545625682, win 5840, options [mss 1460,sackOK,TS val 432945531 ecr
0,nop,wscale 7], length 0
19:52:45.207746 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [S.],
seq 951261745, ack 545625683, win 5792, options [mss 1460,sackOK,TS val
12093210 ecr 432945531,nop,wscale 7], length 0
19:52:45.207763 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [.],
ack 1, win 46, options [nop,nop,TS val 432945531 ecr 12093210], length 0
19:52:45.207882 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [P.],
seq 1:122, ack 1, win 46, options [nop,nop,TS val 432945532 ecr
12093210], length 121
19:52:45.208202 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.],
ack 122, win 46, options [nop,nop,TS val 12093211 ecr 432945532], length 0
19:52:45.209374 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.],
seq 1:1449, ack 122, win 46, options [nop,nop,TS val 12093212 ecr
432945532], length 1448
19:52:45.209384 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.],
seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 12093212 ecr
432945532], length 1448
19:52:45.209389 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.],
seq 2897:4345, ack 122, win 46, options [nop,nop,TS val 12093212 ecr
432945532], length 1448
DELAY
19:52:45.410088 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.],
seq 1:1449, ack 122, win 46, options [nop,nop,TS val 12093413 ecr
432945532], length 1448
19:52:45.410103 IP as80.astro.ch.43606 > lv2.astro.com.http: Flags [.],
ack 1449, win 69, options [nop,nop,TS val 432945734 ecr 12093413], length 0
19:52:45.410482 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.],
seq 4345:5793, ack 122, win 46, options [nop,nop,TS val 12093413 ecr
432945734], length 1448
19:52:45.410492 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.],
seq 5793:7241, ack 122, win 46, options [nop,nop,TS val 12093413 ecr
432945734], length 1448
DELAY
19:52:45.611086 IP lv2.astro.com.http > as80.astro.ch.43606: Flags [.],
seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 12093614 ecr
432945734], length
1448
----------------
here is the traffic as seen on the LVS, internal interface to the client w1
(it is not the same file transfer, this is why the timestamps are different)
18:53:44.819241 IP as80.astro.ch.36962 > w1.http: Flags [S], seq
862769131, win 5840, options [mss 1460,sackOK,TS val 429405138 ecr
0,nop,wscale 7], length 0
18:53:44.819412 IP w1.http > as80.astro.ch.36962: Flags [S.], seq
1480014612, ack 862769132, win 5792, options [mss 1460,sackOK,TS val
8552820 ecr 429405138,nop,wscale 7], length 0
18:53:44.819505 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 1, win
46, options [nop,nop,TS val 429405138 ecr 8552820], length 0
18:53:44.819588 IP as80.astro.ch.36962 > w1.http: Flags [P.], seq 1:122,
ack 1, win 46, options [nop,nop,TS val 429405138 ecr 8552820], length 121
18:53:44.819746 IP w1.http > as80.astro.ch.36962: Flags [.], ack 122,
win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 0
18:53:44.820270 IP w1.http > as80.astro.ch.36962: Flags [.], seq 1:4345,
ack 122, win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 4344
DELAY
18:53:45.021411 IP w1.http > as80.astro.ch.36962: Flags [.], seq 1:1449,
ack 122, win 46, options [nop,nop,TS val 8553022 ecr 429405138], length 1448
18:53:45.021577 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 1449,
win 69, options [nop,nop,TS val 429405341 ecr 8553022], length 0
18:53:45.021757 IP w1.http > as80.astro.ch.36962: Flags [.], seq
4345:7241, ack 122, win 46, options [nop,nop,TS val 8553022 ecr
429405341], length 2896
DELAY
18:53:45.222410 IP w1.http > as80.astro.ch.36962: Flags [.], seq
1449:2897, ack 122, win 46, options [nop,nop,TS val 8553223 ecr
429405341], length 1448
18:53:45.222593 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 2897,
win 91, options [nop,nop,TS val 429405542 ecr 8553223], length 0
18:53:45.222752 IP w1.http > as80.astro.ch.36962: Flags [.], seq
7241:10137, ack 122, win 46, options [nop,nop,TS val 8553223 ecr
429405542], length 2896
DELAY
18:53:45.423386 IP w1.http > as80.astro.ch.36962: Flags [.], seq
2897:4345, ack 122, win 46, options [nop,nop,TS val 8553424 ecr
429405542], length 1448
18:53:45.423567 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 4345,
win 114, options [nop,nop,TS val 429405742 ecr 8553424], length 0
18:53:45.423728 IP w1.http > as80.astro.ch.36962: Flags [.], seq
10137:13033, ack 122, win 46, options [nop,nop,TS val 8553424 ecr
429405742], length 2896
DELAY
18:53:45.624412 IP w1.http > as80.astro.ch.36962: Flags [.], seq
4345:5793, ack 122, win 46, options [nop,nop,TS val 8553625 ecr
429405742], length 1448
18:53:45.624572 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 5793,
win 137, options [nop,nop,TS val 429405943 ecr 8553625], length 0
18:53:45.624753 IP w1.http > as80.astro.ch.36962: Flags [P.], seq
13033:15929, ack 122, win 46, options [nop,nop,TS val 8553625 ecr
429405943], length 2896
DELAY
18:53:45.825385 IP w1.http > as80.astro.ch.36962: Flags [.], seq
5793:7241, ack 122, win 46, options [nop,nop,TS val 8553826 ecr
429405943], length 1448
18:53:45.825547 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 7241,
win 159, options [nop,nop,TS val 429406144 ecr 8553826], length 0
18:53:45.825728 IP w1.http > as80.astro.ch.36962: Flags [.], seq
7241:10137, ack 122, win 46, options [nop,nop,TS val 8553826 ecr
429406144], length 2896
DELAY
18:53:46.026413 IP w1.http > as80.astro.ch.36962: Flags [.], seq
7241:8689, ack 122, win 46, options [nop,nop,TS val 8554027 ecr
429406144], length 1448
18:53:46.026572 IP as80.astro.ch.36962 > w1.http: Flags [.], ack 8689,
win 182, options [nop,nop,TS val 429406345 ecr 8554027], length 0
------------
same transfer, observed on LVS external interface, time stamps agree
with sample 2
18:53:44.819225 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [S],
seq 862769131, win 5840, options [mss 1460,sackOK,TS val 429405138 ecr
0,nop,wscale 7], length 0
18:53:44.819424 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [S.],
seq 1480014612, ack 862769132, win 5792, options [mss 1460,sackOK,TS val
8552820 ecr 429405138,nop,wscale 7], length 0
18:53:44.819499 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.],
ack 1, win 46, options [nop,nop,TS val 429405138 ecr 8552820], length 0
18:53:44.819584 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [P.],
seq 1:122, ack 1, win 46, options [nop,nop,TS val 429405138 ecr
8552820], length 121
18:53:44.819758 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
ack 122, win 46, options [nop,nop,TS val 8552820 ecr 429405138], length 0
18:53:44.820284 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 1:4345, ack 122, win 46, options [nop,nop,TS val 8552820 ecr
429405138], length 4344
DELAY
18:53:45.021424 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 1:1449, ack 122, win 46, options [nop,nop,TS val 8553022 ecr
429405138], length 1448
18:53:45.021567 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.],
ack 1449, win 69, options [nop,nop,TS val 429405341 ecr 8553022], length 0
18:53:45.021771 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 4345:7241, ack 122, win 46, options [nop,nop,TS val 8553022 ecr
429405341], length 2896
DELAY
18:53:45.222424 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 1449:2897, ack 122, win 46, options [nop,nop,TS val 8553223 ecr
429405341], length 1448
18:53:45.222583 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.],
ack 2897, win 91, options [nop,nop,TS val 429405542 ecr 8553223], length 0
18:53:45.222766 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 7241:10137, ack 122, win 46, options [nop,nop,TS val 8553223 ecr
429405542], length 2896
DELAY
18:53:45.423399 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 2897:4345, ack 122, win 46, options [nop,nop,TS val 8553424 ecr
429405542], length 1448
18:53:45.423557 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.],
ack 4345, win 114, options [nop,nop,TS val 429405742 ecr 8553424], length 0
18:53:45.423742 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 10137:13033, ack 122, win 46, options [nop,nop,TS val 8553424 ecr
429405742], length 2896
DELAY
18:53:45.624425 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 4345:5793, ack 122, win 46, options [nop,nop,TS val 8553625 ecr
429405742], length 1448
18:53:45.624562 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.],
ack 5793, win 137, options [nop,nop,TS val 429405943 ecr 8553625], length 0
18:53:45.624915 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [P.],
seq 13033:15929, ack 122, win 46, options [nop,nop,TS val 8553625 ecr
429405943], length 2896
DELAY
18:53:45.825398 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 5793:7241, ack 122, win 46, options [nop,nop,TS val 8553826 ecr
429405943], length 1448
18:53:45.825537 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.],
ack 7241, win 159, options [nop,nop,TS val 429406144 ecr 8553826], length 0
18:53:45.825742 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 7241:10137, ack 122, win 46, options [nop,nop,TS val 8553826 ecr
429406144], length 2896
18:53:46.026426 IP lv2.astro.com.http > as80.astro.ch.36962: Flags [.],
seq 7241:8689, ack 122, win 46, options [nop,nop,TS val 8554027 ecr
429406144], length 1448
18:53:46.026562 IP as80.astro.ch.36962 > lv2.astro.com.http: Flags [.],
ack 8689, win 182, options [nop,nop,TS val 429406345 ecr 8554027], length 0
----------
I also tried to look at the tcp socks involved in the transfer, with ss
But I do not really understand what I see.
seen on client:
ss -t -i -o -e dst lv2.astro.com
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.53.104.2:51478 192.53.104.100:http
uid:101 ino:22082073 sk:ffff88082b018100
ts sack cubic wscale:7,7 rto:201 rtt:1/0.75 ato:50 cwnd:4 send 46.3Mbps
rcv_rtt:174.75 rcv_space:60816
seen on w1 real server:
ss -t -i -o -e dst 192.53.104.2
ESTAB 0 36200 10.1.1.1:http 192.53.104.2:51506
timer:(on,132ms,0) uid:104 ino:37970 sk:ffff88081164f6c0
ts sack cubic wscale:7,7 rto:201 rtt:1.875/0.75 ato:40 cwnd:10
ssthresh:2 send 61.8Mbps rcv_space:5792
--------------
when I do a direct http transfer not through the LVS, (which is fast,
everything is conencted with 1Gb links), then I see these tcp socks
on client:
ss -t -i -o -e dst 192.53.104.2
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.53.104.2:36300
192.53.104.99:http uid:101 ino:22146399 sk:ffff880451ef4180
ts sack cubic wscale:7,7 rto:201 rtt:1/0.75 ato:40 cwnd:4 send 46.3Mbps
rcv_rtt:1.875 rcv_space:6199088
on server:
ESTAB 0 2920616 ::ffff:192.53.104.99:http
::ffff:192.53.104.2:36305 timer:(on,206ms,0) uid:104 ino:47402
sk:ffff8801261071c0
ts sack cubic wscale:7,7 rto:206 rtt:6.5/0.75 ato:40 cwnd:507
ssthresh:504 send 903.6Mbps rcv_space:5792
They seem to have the same values.
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
