|
Hello fellow LVS/IPVS users..
I have been running IPVS for a few years with great success but seem to
have recently run into a very perplexing problem in TUN mode with what
seems to be stale/leaked routing lookups?
On a IPVS director with multiple interfaces (and policy base routing)
it seems IPVS TUN code will emit IPIP packets from the correct
interface (according to PBR) but use/select the wrong source IP in the
TUN IP header. This is reproducible on our systems, but is _VERY_
infrequent (eg. 354 out of 1.5mil packets had an incorrect/invalid
source IP on our system in the last 24 hours)
Details of the failing IPVS director as follows:
<snip>
uname -r
3.18.14 (mainline vanilla)
head -1 /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=32768)
ip route show
default via 172.23.200.1 dev vlan200
172.23.10.0/24 dev vlan10 proto kernel scope link src 172.23.10.11
172.23.20.0/24 dev vlan20 proto kernel scope link src 172.23.20.11
172.23.200.0/24 dev vlan200 proto kernel scope link src 172.23.200.11
192.168.254.0/27 dev vlan500 proto kernel scope link src
192.168.254.15
ip rule show
0: from all lookup local
32764: from all to 192.168.254.32/27 lookup 3
32765: from 192.168.254.0/27 lookup 3
32766: from all lookup main
32767: from all lookup default
ip route show table 3
default via 192.168.254.1 dev vlan500 src 192.168.254.15
ipvsadm -Ln
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 254 wlc
-> 192.168.254.48:0 Tunnel 1 0 1528
FWM 255 wlc
-> 172.23.10.100:0 Tunnel 1 0 35
</snip>
In the following examples I have a remote client (8.8.8.8) attempting
to hit a LVS-TUN service (4.4.4.4:443), the packets are marked in
iptables mangle and are matched by IPVS using FWMARK (254 or 255).
IPVS clusters with realservers on locally connected networks (FWM 255),
the emitted TUN packets seems to work as expected 100% of the time. Eg.
IPVS encapsulates the original packet (src 8.8.8.8 dst 4.4.4.4) in a IP
header (src 172.23.10.11 dst 172.23.10.100) and emits the packet via
vlan10 interface.
<snip>
tcpdump -n -i vlan10
IP 172.23.10.11 > 172.23.10.100: IP 8.8.8.8.62757 > 4.4.4.4.443: Flags
[S], seq 2658459997, win 65535, options [mss 1460,nop,wscale
5,nop,nop,TS val 832819348 ecr 0,sackOK,eol], length 0 (ipip-proto-4)
</snip>
Alternatively if the inbound traffic is handled by a IPVS cluster with
remote L3 realservers, things work most of the time, then the
unexpected happens)
IPVS clusters with realservers on a remote L3 network (FWM 254), IPVS
encapsulates the original packet (src 8.8.8.8 dst 4.4.4.4) in a IP
header (src 192.168.254.15 dst 192.168.254.48) and emits the packet via
the vlan500 interface passing it to the L2 (mac address) of
192.168.254.1.
This worked swimmingly well until I noticed that very intermittently
the following happens:
IPVS encapsulates the original packet (src 8.8.8.8 dst 4.4.4.4) in a IP
header (src 172.23.10.11 dst 192.168.254.48) and emits the packet via
the vlan500 interface passing it to the mac address of 192.168.254.1.
(NOTE: the use of 172.23.10.11 rather than the expected 192.168.254.48
source IP in the TUN header)
Example of broken packet via tcpdump
<snip>
tcpdump -n -i vlan500
IP 172.23.10.11 > 192.168.254.48: IP 8.8.8.8.54063 > 4.4.4.4.443: Flags
[S], seq 1418130791, win 65535, options [mss 1386,nop,wscale
5,nop,nop,TS val 1213718660 ecr 0,sackOK,eol], length 0 (ipip-proto-4)
</snip>
So in the last 24 hours out of 1.5M packets emitted by IPVS on vlan500
(FWM 254) I had 349 packets which get emitted with the wrong source IP
address in the Tunnel IP header. The periods where the wrong source IP
is used by IPVS seem to last for ~2-5min at a time, and affects all
traffic in the LVS cluster with remote L3 realservers.
Another interesting meta-point is, on a LVS director with lots of
interfaces and active IPVS clusters, I only see IPVS select invalid
source IPs which meet the following criteria:
1. The IP exists on on of the directors local interfaces
2. The IP is currently used as a source IP for TUN traffic in other
IPVS clusters that are currently active (passing TUN traffic to local
realservers).
I am guessing this is related to the "saddr" address being stale/not
reinitialized/leaked in the do_output_route4 function of ip_vs_xmit.c?
Im not sure what would cause this type of issue in IPVS, or if its
really related to recent routing changes in the kernel?
Anyone have thoughts/suggestions?
Thanks
-Mike
--
Michael Vallaly <lvs@xxxxxxxxxxxxx>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
