LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] LVS-SNAT only works locally (ubuntu 11.04 amd64)

To: Tom van Leeuwen <tom.van.leeuwen@xxxxxxxxxxxxx>
Subject: Re: [lvs-users] LVS-SNAT only works locally (ubuntu 11.04 amd64)
Cc: "lvs-users@xxxxxxxxxxxxxxxxxxxxxx" <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>, Ivan Havlicek <ivan@xxxxxxxxxxx>, sven _ <svensven@xxxxxxxxx>
From: Julian Anastasov <ja@xxxxxx>
Date: Mon, 30 May 2011 11:55:05 +0300 (EEST)
        Hello,

On Mon, 30 May 2011, Tom van Leeuwen wrote:

> I've installed the new kernel package and am now running:
> loadbalancer-ng ~ # uname -a
> Linux loadbalancer-ng 2.6.38-8-server #42 SMP Mon May 30 08:18:35 CEST 2011
> x86_64 x86_64 x86_64 GNU/Linux
> 
> I can confirm that your patch works!

        Thanks for the testing! I'm CC-ing some people who had
problems in the past with SNAT for IPVS. And I posted the patch
yesterday for inclusion in next kernel.

> I've done several tests:
>       TESTS
>       1) routed vip without alias:
>       ipvsadm -A -t 3.3.3.3:80 -s rr
>       ipvsadm -a -t 3.3.3.3:80 -r 172.16.29.5:80 -m
>       loadbalancer-ng seems to forward the traffic to the default gw
>       without triggering ipvs related stuff
>       DOES NOT WORK, BUT HAS ALWAYS BEEN A REQUIREMENT IF I'M NOT
>       MISTAKEN

        Yes, we require local delivery (local IP address or
local route) because that is how the traffic reaches LOCAL_IN.

>       2) alias vip
>       ip addr add 3.3.3.3 dev lo
>       ipvsadm -A -t 3.3.3.3:80 -s rr
>       ipvsadm -a -t 3.3.3.3:80 -r 172.16.29.5:80 -m
>       WORKS
> 
>       3) alias vip with snat
>       ip addr add 3.3.3.3 dev lo
>       iptables -t nat -A POSTROUTING -m ipvs --vaddr 3.3.3.3 --vport
>       80 -j SNAT --to-source 172.16.29.10
>       ipvsadm -A -t 3.3.3.3:80 -s rr
>       ipvsadm -a -t 3.3.3.3:80 -r 172.16.29.5:80 -m
>       WORKS
> 
>       4) alias vip with snat using some random ip
>       ip addr add 3.3.3.3 dev lo
>       ipvsadm -A -t 3.3.3.3:80 -s rr
>       ipvsadm -a -t 3.3.3.3:80 -r 172.16.29.5:80 -m
>       iptables -t nat -A POSTROUTING -m ipvs --vaddr 3.3.3.3 --vport
>       80 -j SNAT --to-source 1.1.1.1
>       WORKS
> 
>       5) realserver with loadbalancer-ng as default gw. and outgoing
>       MASQUERADE on the loadbalancer-ng
>       So: test 4) setup +
>       iptables -t nat -A POSTROUTING -o eth1 -s 172.16.29.5 -j
>       MASQUERADE
>       WORKS
> 
>       6) test 5) but with SNAT --to-source 3.3.3.3 (VIP)
>       iptables -t nat -A POSTROUTING -o eth1 -s 172.16.29.5 -j SNAT
>       --to-source 3.3.3.3
>       WORKS
> 
> Thanks again!
> 
> With kind regards,
> Tom van Leeuwen
> 
> 
> On 05/28/2011 12:47 AM, Julian Anastasov wrote:
> 
>       Hello,
> 
> On Fri, 27 May 2011, Tom van Leeuwen wrote:
> 
> Hi list, this is my first post, so please be gentle.
> I'm trying to get a lvs (lab)setup to work with snat.
> I've been reading several posts and the mailinglist, and they told that 
> the snat should work with kernel 2.6.36 and higher.
> 
>       May be the problem starts with 2.6.37, I now
> see that the things worked by luck before 2.6.37.
> I'm appending a patch against 2.6.39 for testing.
> It should apply for 2.6.38 too. I hope xt_ipvs has
> no problem with matching the traffic when conntrack=1.
> 
>      6   360 SNAT       all  --  *      *       0.0.0.0/0            
> 0.0.0.0/0           vaddr 172.16.31.10 vport 80 to:172.16.29.10
> 
> loadbalancer-ng ~ # sysctl net.ipv4.vs.conntrack
> net.ipv4.vs.conntrack = 1
> 
> =========================================================
> 
>       Fix the IPVS priority in LOCAL_IN hook,
> so that SNAT target in POSTROUTING is supported for IPVS
> traffic.
> 
>       Before 2.6.37 we used priority 100 in LOCAL_IN to
> process remote requests. We used the same priority as
> iptables SNAT and if IPVS handlers are installed before
> SNAT handlers we supported SNAT in POSTROUTING for the IPVS
> traffic. If SNAT is installed before IPVS, the netfilter
> handlers are before IPVS and netfilter checks the NAT
> table twice for the IPVS requests: once in LOCAL_IN where
> IPS_SRC_NAT_DONE is set and second time in POSTROUTING
> where the SNAT rules are ignored because IPS_SRC_NAT_DONE
> was already set in LOCAL_IN.
> 
>       But in 2.6.37 we changed the IPVS priority for
> LOCAL_IN to be unique (101) forgetting the fact that for
> IPVS traffic we should not walk both LOCAL_IN and
> POSTROUTING nat tables.
> 
>       So, change the priority for processing remote
> IPVS requests from 101 to 99, i.e. before NAT_SRC (100)
> because we prefer to support SNAT in POSTROUTING
> instead of LOCAL_IN. It also moves the priority for
> IPVS replies from 99 to 98.
> 
> Signed-off-by: Julian Anastasov <ja@xxxxxx>
> ---
> 
> diff -urp v2.6.39/linux/net/netfilter/ipvs/ip_vs_core.c linux/net/netfilter/
> ipvs/ip_vs_core.c
> --- v2.6.39/linux/net/netfilter/ipvs/ip_vs_core.c     2011-05-20 
> 10:38:08.0000000
> 00 +0300
> +++ linux/net/netfilter/ipvs/ip_vs_core.c     2011-05-28 00:42:53.845348561 
> +0300
> @@ -1792,7 +1792,7 @@ static struct nf_hook_ops ip_vs_ops[] __
>               .owner          = THIS_MODULE,
>               .pf             = PF_INET,
>               .hooknum        = NF_INET_LOCAL_IN,
> -             .priority       = 99,
> +             .priority       = 98,
>       },
>       /* After packet filtering, forward packet through VS/DR, VS/TUN,
>        * or VS/NAT(change destination), so that filtering rules can be
> @@ -1802,7 +1802,7 @@ static struct nf_hook_ops ip_vs_ops[] __
>               .owner          = THIS_MODULE,
>               .pf             = PF_INET,
>               .hooknum        = NF_INET_LOCAL_IN,
> -             .priority       = 101,
> +             .priority       = 99,
>       },
>       /* Before ip_vs_in, change source only for VS/NAT */
>       {
> @@ -1844,7 +1844,7 @@ static struct nf_hook_ops ip_vs_ops[] __
>               .owner          = THIS_MODULE,
>               .pf             = PF_INET6,
>               .hooknum        = NF_INET_LOCAL_IN,
> -             .priority       = 99,
> +             .priority       = 98,
>       },
>       /* After packet filtering, forward packet through VS/DR, VS/TUN,
>        * or VS/NAT(change destination), so that filtering rules can be
> @@ -1854,7 +1854,7 @@ static struct nf_hook_ops ip_vs_ops[] __
>               .owner          = THIS_MODULE,
>               .pf             = PF_INET6,
>               .hooknum        = NF_INET_LOCAL_IN,
> -             .priority       = 101,
> +             .priority       = 99,
>       },
>       /* Before ip_vs_in, change source only for VS/NAT */
>       {

Regards

--
Julian Anastasov <ja@xxxxxx>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>