LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: Problems with IPVS

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: Problems with IPVS
From: Roberto Nibali <ratz@xxxxxxxxxxxx>
Date: Fri, 10 Nov 2006 09:38:27 +0100
Sorry, haven't been able to look at it yet. I've been moving houses and I was involved in a car accident yesterday and today I've come down with a cold. The laptop with the files is in the car cemetery and police hasn't allowed me to go there yet.

 How are you? Successully recovered laptop?

Pretty busy and preparing to leave for Senegal in 2 weeks.

 Can take a look once more into this problem? ;)

I did at some point, but was unable to find something directly indicating a solvable issue. I would need to sit in front of the system and check the routing cache, socket status, connection tracking or LVS state tracking and gather proper tcpdump samples that are comparable.

You've provided following information:

                phone     packet handling  status
tcpdump.cap :     SE          -j DNAT        OK
tcpdump2.cap:     SE          LVS-NAT        NOK
tcpdump3.cap:    Nokia        LVS-NAT        OK

The traces can be summarized as follows (onlny tcpdump2 and tcpdump3 are interesting in the direct comparison:

tcpdump2
--------
15 packets seen, 15 TCP packets traced
elapsed wallclock time: 0:00:00.004058, 3696 pkts/sec analyzed
trace file elapsed time: 0:00:22.569938
TCP connection info:
2 TCP connections traced:
TCP connection 1:
        host a:        192.168.150.23:80
        host b:        10.5.10.58:16425
        complete conn: no       (SYNs: 0)  (FINs: 0)
        first packet:  Mon Oct 16 15:42:10.491745 2006
        last packet:   Mon Oct 16 15:42:10.491745 2006
        elapsed time:  0:00:00.000000
        total packets: 1
        filename:      tcpdump2.cap
   a->b:                              b->a:
     total packets:             1           total packets:             0
     ack pkts sent:             1           ack pkts sent:             0
     pure acks sent:            0           pure acks sent:            0
     sack pkts sent:            0           sack pkts sent:            0
     dsack pkts sent:           0           dsack pkts sent:           0
     max sack blks/ack:         0           max sack blks/ack:         0
     unique bytes sent:      1448           unique bytes sent:         0
     actual data pkts:          1           actual data pkts:          0
     actual data bytes:      1448           actual data bytes:         0
     rexmt data pkts:           0           rexmt data pkts:           0
     rexmt data bytes:          0           rexmt data bytes:          0
     zwnd probe pkts:           0           zwnd probe pkts:           0
     zwnd probe bytes:          0           zwnd probe bytes:          0
     outoforder pkts:           0           outoforder pkts:           0
     pushed data pkts:          0           pushed data pkts:          0
     SYN/FIN pkts sent:       0/0           SYN/FIN pkts sent:       0/0
     req 1323 ws/ts:          N/Y           req 1323 ws/ts:          N/N
urgent data pkts: 0 pkts urgent data pkts: 0 pkts urgent data bytes: 0 bytes urgent data bytes: 0 bytes mss requested: 0 bytes mss requested: 0 bytes max segm size: 1448 bytes max segm size: 0 bytes min segm size: 1448 bytes min segm size: 0 bytes avg segm size: 1446 bytes avg segm size: 0 bytes max win adv: 6432 bytes max win adv: 0 bytes min win adv: 6432 bytes min win adv: 0 bytes zero win adv: 0 times zero win adv: 0 times avg win adv: 6432 bytes avg win adv: 0 bytes initial window: 1448 bytes initial window: 0 bytes initial window: 1 pkts initial window: 0 pkts
     ttl stream length:        NA           ttl stream length:        NA
     missed data:              NA           missed data:              NA
truncated data: 0 bytes truncated data: 0 bytes truncated packets: 0 pkts truncated packets: 0 pkts data xmit time: 0.000 secs data xmit time: 0.000 secs
     idletime max:        22569.9 ms        idletime max:             NA ms
     throughput:               NA           throughput:               NA

     RTT samples:               0           RTT samples:               0
     RTT min:                 0.0 ms        RTT min:                 0.0 ms
     RTT max:                 0.0 ms        RTT max:                 0.0 ms
     RTT avg:                 0.0 ms        RTT avg:                 0.0 ms
     RTT stdev:               0.0 ms        RTT stdev:               0.0 ms

     RTT from 3WHS:           0.0 ms        RTT from 3WHS:           0.0 ms

     RTT full_sz smpls:         0           RTT full_sz smpls:         0
     RTT full_sz min:         0.0 ms        RTT full_sz min:         0.0 ms
     RTT full_sz max:         0.0 ms        RTT full_sz max:         0.0 ms
     RTT full_sz avg:         0.0 ms        RTT full_sz avg:         0.0 ms
     RTT full_sz stdev:       0.0 ms        RTT full_sz stdev:       0.0 ms

     post-loss acks:            0           post-loss acks:            0
     segs cum acked:            0           segs cum acked:            0
     duplicate acks:            0           duplicate acks:            0
     triple dupacks:            0           triple dupacks:            0
     max # retrans:             0           max # retrans:             0
     min retr time:           0.0 ms        min retr time:           0.0 ms
     max retr time:           0.0 ms        max retr time:           0.0 ms
     avg retr time:           0.0 ms        avg retr time:           0.0 ms
     sdv retr time:           0.0 ms        sdv retr time:           0.0 ms
================================
TCP connection 2:
        host c:        10.5.10.58:16426
        host d:        192.168.150.27:80
        complete conn: no       (SYNs: 2)  (FINs: 1)
        first packet:  Mon Oct 16 15:42:19.588928 2006
        last packet:   Mon Oct 16 15:42:33.061684 2006
        elapsed time:  0:00:13.472756
        total packets: 14
        filename:      tcpdump2.cap
   c->d:                              d->c:
     total packets:             5           total packets:             9
     ack pkts sent:             4           ack pkts sent:             9
     pure acks sent:            3           pure acks sent:            1
     sack pkts sent:            2           sack pkts sent:            0
     dsack pkts sent:           0           dsack pkts sent:           0
     max sack blks/ack:         1           max sack blks/ack:         0
     unique bytes sent:       621           unique bytes sent:      2457
     actual data pkts:          1           actual data pkts:          6
     actual data bytes:       621           actual data bytes:      8249
     rexmt data pkts:           0           rexmt data pkts:           4
     rexmt data bytes:          0           rexmt data bytes:       5792
     zwnd probe pkts:           0           zwnd probe pkts:           0
     zwnd probe bytes:          0           zwnd probe bytes:          0
     outoforder pkts:           0           outoforder pkts:           0
     pushed data pkts:          1           pushed data pkts:          1
     SYN/FIN pkts sent:       1/0           SYN/FIN pkts sent:       1/1
     req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y
     adv wind scale:            0           adv wind scale:            2
     req sack:                  Y           req sack:                  Y
     sacks sent:                2           sacks sent:                0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts urgent data bytes: 0 bytes urgent data bytes: 0 bytes mss requested: 1460 bytes mss requested: 1460 bytes max segm size: 621 bytes max segm size: 1448 bytes min segm size: 621 bytes min segm size: 1009 bytes avg segm size: 620 bytes avg segm size: 1374 bytes max win adv: 16384 bytes max win adv: 7036 bytes min win adv: 16384 bytes min win adv: 7036 bytes zero win adv: 0 times zero win adv: 0 times avg win adv: 16384 bytes avg win adv: 7036 bytes initial window: 621 bytes initial window: 2457 bytes initial window: 1 pkts initial window: 2 pkts ttl stream length: NA ttl stream length: 2457 bytes missed data: NA missed data: 0 bytes truncated data: 0 bytes truncated data: 0 bytes truncated packets: 0 pkts truncated packets: 0 pkts data xmit time: 0.000 secs data xmit time: 12.644 secs
     idletime max:        11867.4 ms        idletime max:         6743.2 ms
throughput: 46 Bps throughput: 182 Bps

     RTT samples:               2           RTT samples:               1
     RTT min:                 0.1 ms        RTT min:               280.2 ms
     RTT max:                 0.1 ms        RTT max:               280.2 ms
     RTT avg:                 0.1 ms        RTT avg:               280.1 ms
     RTT stdev:               0.0 ms        RTT stdev:               0.0 ms

     RTT from 3WHS:           0.1 ms        RTT from 3WHS:         280.2 ms

     RTT full_sz smpls:         1           RTT full_sz smpls:         1
     RTT full_sz min:         0.1 ms        RTT full_sz min:       280.2 ms
     RTT full_sz max:         0.1 ms        RTT full_sz max:       280.2 ms
     RTT full_sz avg:         0.1 ms        RTT full_sz avg:       280.1 ms
     RTT full_sz stdev:       0.0 ms        RTT full_sz stdev:       0.0 ms

     post-loss acks:            0           post-loss acks:            0
     segs cum acked:            0           segs cum acked:            0
     duplicate acks:            1           duplicate acks:            2
     triple dupacks:            0           triple dupacks:            0
     max # retrans:             0           max # retrans:             4
     min retr time:           0.0 ms        min retr time:         842.6 ms
     max retr time:           0.0 ms        max retr time:        6743.2 ms
     avg retr time:           0.0 ms        avg retr time:        3160.9 ms
     sdv retr time:           0.0 ms        sdv retr time:        2609.4 ms


tcpdump3
--------
21 packets seen, 21 TCP packets traced
elapsed wallclock time: 0:00:00.004798, 4376 pkts/sec analyzed
trace file elapsed time: 0:00:05.679351
TCP connection info:
2 TCP connections traced:
TCP connection 1:
        host a:        10.5.129.39:38609
        host b:        192.168.150.27:80
        complete conn: yes
        first packet:  Mon Oct 16 15:59:13.409987 2006
        last packet:   Mon Oct 16 15:59:16.449495 2006
        elapsed time:  0:00:03.039507
        total packets: 10
        filename:      tcpdump3.cap
   a->b:                              b->a:
     total packets:             5           total packets:             5
     ack pkts sent:             4           ack pkts sent:             5
     pure acks sent:            2           pure acks sent:            2
     sack pkts sent:            0           sack pkts sent:            0
     dsack pkts sent:           0           dsack pkts sent:           0
     max sack blks/ack:         0           max sack blks/ack:         0
     unique bytes sent:       590           unique bytes sent:       926
     actual data pkts:          1           actual data pkts:          1
     actual data bytes:       590           actual data bytes:       926
     rexmt data pkts:           0           rexmt data pkts:           0
     rexmt data bytes:          0           rexmt data bytes:          0
     zwnd probe pkts:           0           zwnd probe pkts:           0
     zwnd probe bytes:          0           zwnd probe bytes:          0
     outoforder pkts:           0           outoforder pkts:           0
     pushed data pkts:          1           pushed data pkts:          1
     SYN/FIN pkts sent:       1/1           SYN/FIN pkts sent:       1/1
     req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y
     adv wind scale:            0           adv wind scale:            2
     req sack:                  Y           req sack:                  Y
     sacks sent:                0           sacks sent:                0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts urgent data bytes: 0 bytes urgent data bytes: 0 bytes mss requested: 1360 bytes mss requested: 1460 bytes max segm size: 590 bytes max segm size: 926 bytes min segm size: 590 bytes min segm size: 926 bytes avg segm size: 589 bytes avg segm size: 925 bytes max win adv: 48960 bytes max win adv: 6972 bytes min win adv: 48034 bytes min win adv: 6972 bytes zero win adv: 0 times zero win adv: 0 times avg win adv: 48497 bytes avg win adv: 6972 bytes initial window: 590 bytes initial window: 926 bytes initial window: 1 pkts initial window: 1 pkts ttl stream length: 590 bytes ttl stream length: 926 bytes missed data: 0 bytes missed data: 0 bytes truncated data: 0 bytes truncated data: 0 bytes truncated packets: 0 pkts truncated packets: 0 pkts data xmit time: 0.000 secs data xmit time: 0.000 secs
     idletime max:         1279.0 ms        idletime max:         1698.0 ms
throughput: 194 Bps throughput: 305 Bps

     RTT samples:               3           RTT samples:               2
     RTT min:                 0.1 ms        RTT min:               819.0 ms
     RTT max:                 0.2 ms        RTT max:               977.5 ms
     RTT avg:                 0.1 ms        RTT avg:               898.2 ms
     RTT stdev:               0.0 ms        RTT stdev:               0.0 ms

     RTT from 3WHS:           0.1 ms        RTT from 3WHS:         819.0 ms

     RTT full_sz smpls:         1           RTT full_sz smpls:         1
     RTT full_sz min:         0.2 ms        RTT full_sz min:       977.5 ms
     RTT full_sz max:         0.2 ms        RTT full_sz max:       977.5 ms
     RTT full_sz avg:         0.2 ms        RTT full_sz avg:       977.4 ms
     RTT full_sz stdev:       0.0 ms        RTT full_sz stdev:       0.0 ms

     post-loss acks:            0           post-loss acks:            0
     segs cum acked:            0           segs cum acked:            1
     duplicate acks:            1           duplicate acks:            1
     triple dupacks:            0           triple dupacks:            0
     max # retrans:             0           max # retrans:             0
     min retr time:           0.0 ms        min retr time:           0.0 ms
     max retr time:           0.0 ms        max retr time:           0.0 ms
     avg retr time:           0.0 ms        avg retr time:           0.0 ms
     sdv retr time:           0.0 ms        sdv retr time:           0.0 ms
================================
TCP connection 2:
        host c:        10.5.129.39:50071
        host d:        192.168.150.27:80
        complete conn: yes
        first packet:  Mon Oct 16 15:59:16.469074 2006
        last packet:   Mon Oct 16 15:59:19.089338 2006
        elapsed time:  0:00:02.620264
        total packets: 11
        filename:      tcpdump3.cap
   c->d:                              d->c:
     total packets:             5           total packets:             6
     ack pkts sent:             4           ack pkts sent:             6
     pure acks sent:            2           pure acks sent:            2
     sack pkts sent:            0           sack pkts sent:            0
     dsack pkts sent:           0           dsack pkts sent:           0
     max sack blks/ack:         0           max sack blks/ack:         0
     unique bytes sent:       670           unique bytes sent:      2269
     actual data pkts:          1           actual data pkts:          2
     actual data bytes:       670           actual data bytes:      2269
     rexmt data pkts:           0           rexmt data pkts:           0
     rexmt data bytes:          0           rexmt data bytes:          0
     zwnd probe pkts:           0           zwnd probe pkts:           0
     zwnd probe bytes:          0           zwnd probe bytes:          0
     outoforder pkts:           0           outoforder pkts:           0
     pushed data pkts:          1           pushed data pkts:          1
     SYN/FIN pkts sent:       1/1           SYN/FIN pkts sent:       1/1
     req 1323 ws/ts:          Y/Y           req 1323 ws/ts:          Y/Y
     adv wind scale:            0           adv wind scale:            2
     req sack:                  Y           req sack:                  Y
     sacks sent:                0           sacks sent:                0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts urgent data bytes: 0 bytes urgent data bytes: 0 bytes mss requested: 1360 bytes mss requested: 1460 bytes max segm size: 670 bytes max segm size: 1348 bytes min segm size: 670 bytes min segm size: 921 bytes avg segm size: 669 bytes avg segm size: 1133 bytes max win adv: 48960 bytes max win adv: 7132 bytes min win adv: 47607 bytes min win adv: 7132 bytes zero win adv: 0 times zero win adv: 0 times avg win adv: 48513 bytes avg win adv: 7132 bytes initial window: 670 bytes initial window: 2269 bytes initial window: 1 pkts initial window: 2 pkts ttl stream length: 670 bytes ttl stream length: 2269 bytes missed data: 0 bytes missed data: 0 bytes truncated data: 0 bytes truncated data: 0 bytes truncated packets: 0 pkts truncated packets: 0 pkts data xmit time: 0.000 secs data xmit time: 0.000 secs
     idletime max:         1118.5 ms        idletime max:         1694.3 ms
throughput: 256 Bps throughput: 866 Bps

     RTT samples:               3           RTT samples:               2
     RTT min:                 0.1 ms        RTT min:               720.9 ms
     RTT max:                 0.1 ms        RTT max:               974.0 ms
     RTT avg:                 0.1 ms        RTT avg:               847.4 ms
     RTT stdev:               0.0 ms        RTT stdev:               0.0 ms

     RTT from 3WHS:           0.1 ms        RTT from 3WHS:         720.9 ms

     RTT full_sz smpls:         1           RTT full_sz smpls:         1
     RTT full_sz min:         0.1 ms        RTT full_sz min:       974.0 ms
     RTT full_sz max:         0.1 ms        RTT full_sz max:       974.0 ms
     RTT full_sz avg:         0.1 ms        RTT full_sz avg:       973.9 ms
     RTT full_sz stdev:       0.0 ms        RTT full_sz stdev:       0.0 ms

     post-loss acks:            0           post-loss acks:            0
     segs cum acked:            0           segs cum acked:            2
     duplicate acks:            1           duplicate acks:            0
     triple dupacks:            0           triple dupacks:            0
     max # retrans:             0           max # retrans:             0
     min retr time:           0.0 ms        min retr time:           0.0 ms
     max retr time:           0.0 ms        max retr time:           0.0 ms
     avg retr time:           0.0 ms        avg retr time:           0.0 ms
     sdv retr time:           0.0 ms        sdv retr time:           0.0 ms

There's not so much a difference. I've tried to generate gnuplot graphs from the sample to see a visual difference but time ran out and with it my interest :). The segment size and time sequence graphs are the only ones which are interesting. My findings are all at:

http://www.drugphish.ch/~ratz/lvs-plots/

I've removed the tcpdump capture files since they contained some "business-critical" information (at least if it was my business).

 P.S. BTW thanks for mpt-status for x86_64. ;) I thought why your name
reminds me somebody. :)

Gosh, true, now that you mention it :). Thanks for the HP bl35 testing. The new 1.2.0 release should work like a charm on your hardware.

Best regards,
Roberto Nibali, ratz
--
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc

<Prev in Thread] Current Thread [Next in Thread>