|
On Mon, Aug 15, 2005 at 02:46:53PM -0400, Jari Takkala wrote:
> Hello,
>
> We have a fairly large installation of both single and failover-pair load
> balancers in an LVS-NAT setup. On one of these fail over pairs, we have been
> experiencing problems where the SYN+ACK reply from a real server does not get
> source NAT'd to the VIP IP.
>
> We are load balancing both HTTP and FTP connections, and it seems like the
> problem only manifests itself when we add the FTP service into ipvsadm. The
> odd thing is that it affects all VIP's.
>
> Here is a tcpdump on the external interface of the load balancer:
>
> 03:20:42.944203 0:12:d9:92:d2:b2 0:2:b3:a1:90:d4 ip 74: 216.220.XX.XXX.9345 >
> 10.99.23.64.http: S [tcp sum ok] 250882373:250882373(0) win 5840 <mss
> 1380,sackOK,timestamp 2855118908 0,nop,wscale 0> (DF) (ttl 63, id 41526, len
> 60)
> 03:20:42.944533 0:2:b3:a1:90:d4 0:12:d9:92:d2:b2 ip 78: 10.99.22.53.http >
> 216.220.XX.XXX.9345: S [tcp sum ok] 3305203912:3305203912(0) ack 250882374
> win 16560 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0,nop,nop,sackOK> (DF)
> (ttl 127, id 55367, len 64)
> 03:20:42.944671 0:12:d9:92:d2:b2 0:2:b3:a1:90:d4 ip 78: 216.220.XX.XXX.9345 >
> 10.99.22.53.http: R [tcp sum ok] 1:1(0) ack 1 win 16560 <mss 1460,nop,wscale
> 0,nop,nop,timestamp 0 0,nop,nop,sackOK> (DF) (ttl 127, id 55367, len 64)
>
> 216.220.XX.XXX is the client IP (CIP)
> 10.99.23.64 is the virtual IP on the outside interface of the load balancer
> (VIP)
> 10.99.22.53 is the IP of the real server (RIP)
>
> As can be seen, the incoming SYN from 216.220.XX.XXX goes to the VIP IP of
> 10.99.23.64, it gets translated to one of the real servers (10.99.22.53), the
> real server returns a SYN+ACK back to the CIP, however the reply is not
> translated.
>
> Here is the tcpdump from the internal interface:
>
> 03:20:42.944316 0:2:b3:a1:90:d5 0:2:55:9c:55:c3 ip 74: 216.220.XX.XXX.9345 >
> 10.99.22.53.http: S [tcp sum ok] 250882373:250882373(0) win 5840 <mss
> 1380,sackOK,timestamp 2855118908 0,nop,wscale 0> (DF) (ttl 63, id 41526, len
> 60)
> 03:20:42.944483 0:2:55:9c:55:c3 0:2:b3:a1:90:d5 ip 78: 10.99.22.53.http >
> 216.220.XX.XXX.9345: S [tcp sum ok] 3305203912:3305203912(0) ack 250882374
> win 16560 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0,nop,nop,sackOK> (DF)
> (ttl 128, id 55367, len 64)
> 03:20:42.944703 0:2:b3:a1:90:d5 0:2:55:9c:55:c3 ip 78: 216.220.XX.XXX.9345 >
> 10.99.22.53.http: R [tcp sum ok] 1:1(0) ack 1 win 16560 <mss 1460,nop,wscale
> 0,nop,nop,timestamp 0 0,nop,nop,sackOK> (DF) (ttl 126, id 55367, len 64)
>
> Packet flow on this interface is normal. MAC addresses in both tcpdump's
> appear normal.
>
> Here is the debug output:
>
> Aug 13 03:20:43 kernel: IPVS: lookup/in TCP
> 216.220.XX.XXX:9345->10.99.23.64:80 hit
> Aug 13 03:20:43 kernel: IPVS: Incoming TCP 216.220.XX.XXX:9345->10.99.23.64:80
> Aug 13 03:20:43 kernel: Enter: ip_vs_nat_xmit, ip_vs_conn.c line 680
> Aug 13 03:20:43 kernel: IPVS: NAT to 10.99.22.53:80
> Aug 13 03:20:43 kernel: Leave: ip_vs_nat_xmit, ip_vs_conn.c line 820
> Aug 13 03:20:43 kernel: Enter: ip_vs_out, ip_vs_core.c line 646
> Aug 13 03:20:43 kernel: IPVS: lookup/out TCP
> 10.99.22.53:80->216.220.XX.XXX:9345 not hit
> Aug 13 03:20:43 kernel: IPVS: packet for TCP 216.220.XX.XXX:9345 continue
> traversal as normal.
> Aug 13 03:20:43 kernel: Enter: ip_vs_out, ip_vs_core.c line 646
> Aug 13 03:20:43 kernel: IPVS: lookup/out TCP
> 216.220.XX.XXX:9345->10.99.22.53:80 not hit
> Aug 13 03:20:43 kernel: IPVS: packet for TCP 10.99.22.53:80 continue
> traversal as normal.
>
> The VIP IP is an interface alias on eth0. The VIP's are shared between the
> load balancers using heartbeat.
>
> eth0:4 Link encap:Ethernet HWaddr 00:02:B3:A1:90:D4
> inet addr:10.99.23.64 Bcast:10.99.23.255 Mask:255.255.255.0
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
> Interrupt:11 Base address:0xece0 Memory:febff000-febff038
>
> We have IP forwarding enabled in /proc/sys/net/ipv4/ip_forward so that we can
> send traffic directly to the real servers if we need to. Software versions in
> use are:
>
> # uname -a
> Linux lb-a 2.4.26 #3 Fri Jul 2 14:12:31 EDT 2004 i686 i686 i386 GNU/Linux
> # ipvsadm -v
> ipvsadm v1.21 2002/07/09 (compiled with popt and IPVS v1.0.11)
> # ipvsadm -l -n
> IP Virtual Server version 1.0.11 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 10.99.23.64:80 wlc persistent 300
> -> 10.99.22.52:80 Masq 1 11 4
> -> 10.99.22.58:80 Masq 1 10 16
> -> 10.99.22.53:80 Masq 1 10 2
> -> 10.99.22.215:80 Masq 1 11 2
> TCP 10.99.23.57:21 wlc
> -> 10.99.22.208:21 Masq 1 0 0
> -> 10.99.22.207:21 Masq 1 0 0
> TCP 10.99.23.51:80 wlc persistent 300
> -> 10.99.22.199:80 Masq 1 12 3
> -> 10.99.22.197:80 Masq 1 12 3
>
> Does anyone have any idea why this problem would be occurring?
Strange indeed. Do you have the ip_vs_ftp module loaded,
it looks as if this might be missing.
> Looking at the debug output, it looks like IPVS is not matching the
> outgoing packet against its hash table entry. And would anyone be able
> to tell me what the relationship is between the ip_vs module, and the
> ip_tables, iptable_nat, and ip_conntrack modules? It seems like all of
> those modules get loaded at boot time, however we are not doing any
> firewalling on this load balancer ('iptables -L -n' and 'iptables -L
> -n -t nat' show no rules loaded).
ip_vs is not related to the other modules. They are probably
being loaded as a result of running iptables, or perhaps they
are just always loaded at boot.
--
Horms
|
