LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] Port mapping with LVS-DR using fwmark

To: Jacoby Hickerson <hickersonjl@xxxxxxxxx>
Subject: Re: [lvs-users] Port mapping with LVS-DR using fwmark
Cc: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
From: Julian Anastasov <ja@xxxxxx>
Date: Fri, 17 Jan 2014 23:39:55 +0200 (EET)
        Hello,

On Thu, 16 Jan 2014, Jacoby Hickerson wrote:

> I've searched Google and this mailing list but haven't quite seen the same
> configuration and/or setup as mine.
> 
> The ldirectord documentation states that port mapping on the same server
> where the director resides is not possible other than masq, however it says
> "non-fwmark".  My setup is using fwmark, however, when trying to port map
> from port 80 to another port, the client connection hangs.  Here are the
> exact details of my setup:
> 
> The VIP is on the same box as the director and RIP 172.17.0.16.  This setup

        VIPs are always on director. You mean RIP 172.17.0.16
and VIP 172.17.0.24 are on same box? Then there are 2 cases:
local and non-local client?

        Only the masq forwarding method can change daddr and
dport in packet and it should work even for local client and
for local real server. Except if some recent regression happens.

> works fine when no port mapping is being done, but I need to move the port
> to something higher than 1024.
> 
> virtual=172.17.0.24:80
> real=172.17.0.16:50000 gate 100
> real=172.17.0.17:50000 gate 100
> service=http
> scheduler=rr
> protocol=tcp
>   checktype=connect
>   fwmark=100
> 
> iptables:
> iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp
> --dport 80 -j MARK --set-xmark 0x64/0xffffffff
> iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT
> --to-ports 50000
> iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 80 -j REDIRECT
> --to-ports 50000

        Do you have CONFIG_IP_VS_NFCT enabled and sysctl var
/proc/sys/net/ipv4/vs/conntrack set to 1? IIRC, it is needed 
when packets modified by IPVS need to be inspected later by netfilter
in POST_ROUTING or INPUT chain as is the case with the LocalNode
feature. If "conntrack" is not enabled, IPVS marks the packets as 
untracked to avoid consuming memory for conntracks, the conntracks
are created and destroyed with every packet.

        If consuming memory is not a problem, you can find that
conntrack=1 with tuned nf_conntrack_max is faster than conntrack=0.

        Let me know if conntrack=1 does not help, so that
we can track the problem further.

> Issue:
> curl -v 'http://172.17.0.24'
> * About to connect() to 172.17.0.24 port 80 (#0)
> *   Trying 172.17.0.24...
> 
> 00:41:44.503581 IP 172.17.0.2.46099 > 172.17.0.24.80: Flags [S], seq
> 1066084928, win 14600, options [mss 1460,sackOK,TS val 2520815062 ecr
> 0,nop,wscale 7], length 0
> 00:41:44.503581 IP 172.17.0.2.46099 > 172.17.0.24.80: Flags [S], seq
> 1066084928, win 14600, options [mss 1460,sackOK,TS val 2520815062 ecr
> 0,nop,wscale 7], length 0
> 00:41:44.503658 IP 172.17.0.16.50000 > 172.17.0.2.46099: Flags [S.], seq
> 824291086, ack 1066084929, win 14480, options [mss 1460,sackOK,TS val
> 9521949 ecr 2520815062,nop,wscale 7], length 0
> 00:41:44.503663 IP 172.17.0.16.50000 > 172.17.0.2.46099: Flags [S.], seq
> 824291086, ack 1066084929, win 14480, options [mss 1460,sackOK,TS val
> 9521949 ecr 2520815062,nop,wscale 7], length 0
> 
> So the problem I'm having is that the source ip is not being translated by
> iptables but sent via lvs as the RIP.  Is there a kernel option, iptables
> option or ipvsadm option that would allow it to change it back to the VIP?
> 
> Any help would be very appreciated!
> 
> Jacoby

Regards

--
Julian Anastasov <ja@xxxxxx>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>