LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] ipvsadm is not forwarding connections

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: Re: [lvs-users] ipvsadm is not forwarding connections
From: Anders Henke <anders.henke@xxxxxxxx>
Date: Mon, 23 Jun 2014 11:57:18 +0200
On 18.06.2014, Stephen Carville wrote:
> I set up a CentOS 6.5 box to test ipvsadm. So far I have been unable to
> get it to forward connections. When I try to connect, it doesn't write
> anything in /var/log/messages to tell me what is happening. Netstat
> doesn't see anything listening on the interface IP (I read elsewhere
> that is normal) and tshark sees the incoming SYN but there is either a
> timeout or a RST.
> 
> Rules right now:
> 
> $ ipvsadm -L
> 
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
> TCP  10.212.160.40:4172 lc persistent 360
>   -> 10.212.170.162:4172          Route   1      0          0
>   -> 10.212.170.163:4172          Route   1      0          0
> 
> IP forwarding is turned on:
> 
> $ sysctl net.ipv4.ip_forward
> net.ipv4.ip_forward = 1

Short answer: switch to kernel 3.6 or newer, turn off rp_filter for the 
interface receiving the reply packet, and replace rp_filter functionality by 
more accurate and flexible iptables rules in the FORWARD chain.

Long:

When you're using direct routing/gatewaying or tunneling, your replies will 
have the VIP address as the source address of the packet. They're routed via 
your balancer, who also knows the same IP address to be a local address.

Depending on /proc/sys/net/ipv4/conf/*/rp_filter or kernel version, you need to 
either change a little bit of configuration or apply special kernel patches 
(forward_shared).

If you're running a linux kernel <3.6, your kernel will always drop those 
packets as being spoofed. If you're turning on net/ipv4/conf/*/log_martians, 
you'll also see kernel log messages on this.

>From the stone age of IPVS, there is a kernel patch introducing a sysctl 
>switch (forward_shared) to turn off this behaviour for selected interfaces. If 
>misconfigured (e.g. "turn it on for all interfaces"), this patch will 
>introduce an option for third parties to spoof packets onto your network 
>(which is something you don't want to do).

http://www.ssi.bg/~ja/ does offer recent patches for about any major linux 
kernel source from 2.4.19 to 3.15, just in case you're interested.



Linux Kernel 3.6 did somehow obsolete the forward_shared-patch by changing the 
behaviour of rp_filter for the interface receiving the spoofed packet:
-if rp_filter is turned on and ip_forward=1, the kernel will most likely drop 
your reply packets, as they're most likely received by the 'wrong' interface: 
an interface without a matching entry in the routing table for this IP address. 
Given the point many linux distros do enable rp_filter per default, you most 
likely hit that wall.
-if rp_filter is turned off (=0) and ip_forward=1, the kernel will gladly 
forward your "spoofed"  reply packets.

The exact use of rp_filter on a dedicated load balancer is fairly limited, if 
at all. It's much better to replace its functionality by dedicated iptables 
rules and restricting the FORWARD chain.

For your configuration, this results in the following configuration changes:

$ iptables -P FORWARD DROP
$ iptables -A FORWARD -i eth1 -o eth0 --source 10.212.160.40 -j ACCEPT
$ sysctl -w proc/sys/net/ipv4/conf/eth1/rp_filter=0

If you're keen on extra-tight security, you may also copy the permissive line 
and add "-m --mac-source $realserver-mac" for every realserver. This ensures 
only your real servers may send "spoofed" traffic via your balancer.



Best,

Anders
-- 
1&1 Internet AG              Expert Systems Architect (IT Operations)
Brauerstrasse 50             v://49.721.91374.0
D-76135 Karlsruhe            f://49.721.91374.225

Amtsgericht Montabaur HRB 6484
Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann, 
Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek, 
Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>