[lvs-users] LVS-TUN trouble with return packets

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: [lvs-users] LVS-TUN trouble with return packets
From: Patrick Zaloum <pzaloum@xxxxxxxxx>
Date: Thu, 21 Oct 2010 18:14:11 -0400
I have set up an IPVS environment using keepalived. My IPVS machines
are in a DMZ, and my real servers are behind the firewall. I have
apache running on the real servers and I am providing a VIP with
HTTP/HTTPS service pointing to the RIP's.

I have created the tunl0 device with the VIP, and no-arp, on the real servers.

I can ping the VIP from a client, and health check on the IPVS shows
both realservers as healthy.

If I attempt to connect to the service from a client, I get a timeout.
I took a tcpdump in various places as I troubleshooted. My client is
receiving the return packet from the real server (as per the design)
but does not seem to accept it. I noticed in the dump that the
sequence numbers were not what I would expect: I send a SYN to the
VIP, it gets sent to a RIP over the IPIP tunnel, realserver responds
an ACK to the client. In the SYN if the sequence number is 1000 the
real server should ACK 1001... what is happening is that the
realserver is ACKing the tunnel packet, not the encapsulated packet. I
suspect this is where my problem is but I haven't found anything that
resembled this issue on Google.

Can anyone suggest a fix?

I will paste some relevant tcpdump output. Notice my CLIENT SYN packet
is 4244383796, TUNNEL SYN packet is 1869554645. What the client
receives from the RIP is ACKing with 1869554646 and not 4244383797 as
I would have expected. If you look at the packet sent in the tunnel
(CIP to RIP Tunnel) the SYN number is the same as the IPIP packet, NOT
the same one my client IP sent initially.

18:01:29.521993 IP CIP.42852 > VIP.https: S 4244383796:4244383796(0)
win 5840 <mss 1460,sackOK,timestamp 99292997 0,nop,wscale 6>

18:01:29.522040 IP IPVS > RIP: IP {CIP.42852 > VIP.https: S
1869554645:1869554645(0) win 5840 <mss 1380,sackOK,timestamp 99292997
0,nop,wscale 6>} (ipip-proto-4)

CIP to RIP (Tunnel)
18:01:29.522040 IP CIP.42852 > VIP.https: S 1869554645:1869554645(0)
win 5840 <mss 1380,sackOK,timestamp 99292997 0,nop,wscale 6>

18:01:29.522175 IP VIP.https > CIP.42852: S 2673651702:2673651702(0)
ack 1869554646 win 5792 <mss 1460,sackOK,timestamp 552990048
99292997,nop,wscale 7>

Am I missing something here? Is this behaviour by design?

Thanks in advance!

Please read the documentation before posting - it's available at: mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to

<Prev in Thread] Current Thread [Next in Thread>