LVS
lvs-users
[Top] [All Lists]
Google
 
Web LinuxVirtualServer.org
<prev [Date] next> <prev [Thread] next>

Re: [lvs-users] Port mapping with LVS-DR using fwmark

from [Jacoby Hickerson] [Permanent Link]
To: Julian Anastasov <ja@xxxxxx>
Subject: Re: [lvs-users] Port mapping with LVS-DR using fwmark
Cc: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
From: Jacoby Hickerson <hickersonjl@xxxxxxxxx>
Date: Mon, 27 Jan 2014 18:13:36 -0800 
Apologies, the debug output showing port 50130 should be 50000

ex:
IPVS: lookup/in TCP 172.17.0.24:*50130*->172.17.0.2:42816 not hit
should be:
IPVS: lookup/in TCP 172.17.0.24:*50000*->172.17.0.2:42816 not hit

I have attached the file 'ipvs_debug_output' with correct debug so that
this thread is not cluttered.

Thanks!

Jacoby


On Mon, Jan 27, 2014 at 4:00 PM, Jacoby Hickerson <hickersonjl@xxxxxxxxx>wrote:

> Certainly and that makes sense, I will consolidate what I've emailed
> before with the additional information here.
>
> # PC info: Linux 3.12.5 for real servers 1 and 2, and Linux 3.9.10 for the
> client box.
>
> There are 3 boxes total, client box, director/RIP1( real server 1) and
> RIP2 (real server 2):
> - client box:
> inet 172.17.0.2/16 brd 172.17.255.255 scope global eth1   #CIP
>
> - director which is the same as real server 1 (RIP1).  The client is on a
> separate box.
> inet 172.17.0.16/16 brd 172.17.255.255 scope global bond0
> #RIP1
> inet 172.17.0.24/16 brd 172.17.255.255 scope global secondary bond0:2
> #VIP
>
> - real server 2 (RIP2)
> inet 172.17.0.24/32 scope global lo:0                      #VIP on
> loopback
> inet 172.17.0.17/16 brd 172.17.255.255 scope global bond0  #RIP2
>
> # ipvs setup on real server 1 (RIP1) only
> ipvsadm -C
> ipvsadm -A -f 100 -s rr
> ipvsadm -a -f 100 -r 172.17.0.16 -w 100
> ipvsadm -a -f 100 -r 172.17.0.17 -w 100
>
> # iptable rules (these rules are set for both real server 1 and real
> server 2)
> iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp
> --dport 80 -j MARK --set-xmark 0x64/0xffffffff
> iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT
> --to-ports 50000
> iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 80 -j REDIRECT
> --to-ports 50000
>
> The test I'm conducting is an http get from the client box connecting to
> the VIP:
> - Issue the following command on the client box:
> curl -v 'http://172.17.0.24'
>
> On both real servers there is an nginx webserver listening on port 50000
>
> I also turned on debugging and ran the curl command with port mapping
> using level 12 debug (this is output when the issue occurs of no load
> balancing).
> Debug output on real server 1 after executing the curl command the first
> time:
>
> Jan 24 23:05:44 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0
> refcnt 1 weight 100
> Jan 24 23:05:44 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37455 v:
> 172.17.0.16:50130 d:172.17.0.17:50130 fwd:R s:65276 conn->flags:183
> conn->refcnt:1 dest->refcnt:2
> Jan 24 23:05:44 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37455 v:
> 172.17.0.16:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2
> Jan 24 23:05:44 pc01 kernel: IPVS: TCP input  [S...] 172.17.0.17:50130->
> 172.17.0.2:37455 state: NONE->SYN_RECV conn->refcnt:2
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
> 172.17.0.2:37455 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130->
> 172.17.0.2:37455 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup service: fwm 0 TCP
> 172.17.0.2:37455 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455->
> 172.17.0.16:50130 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455->
> 172.17.0.16:50130 hit
> Jan 24 23:05:44 pc01 kernel: IPVS: TCP input  [..A.] 172.17.0.17:50130->
> 172.17.0.2:37455 state: SYN_RECV->ESTABLISHED conn->refcnt:2
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455->
> 172.17.0.16:50130 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455->
> 172.17.0.16:50130 hit
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
> 172.17.0.2:37455 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130->
> 172.17.0.2:37455 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
> 172.17.0.2:37455 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130->
> 172.17.0.2:37455 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455->
> 172.17.0.16:50130 not hit
> Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455->
> 172.17.0.16:50130 hit
> Jan 24 23:05:44 pc01 kernel: IPVS: TCP input  [.FA.] 172.17.0.17:50130->
> 172.17.0.2:37455 state: ESTABLISHED->FIN_WAIT conn->refcnt:2
>
> Debug output on real server 1 after executing the curl command a second
> time:
>
> Jan 24 23:05:45 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling...
> Jan 24 23:05:45 pc01 kernel: IPVS: RR: server 172.17.0.16:0 activeconns 0
> refcnt 1 weight 100
> Jan 24 23:05:45 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37456 v:
> 172.17.0.16:50130 d:172.17.0.16:50130 fwd:R s:65276 conn->flags:183
> conn->refcnt:1 dest->refcnt:2
> Jan 24 23:05:45 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37456 v:
> 172.17.0.16:50130 d:172.17.0.16:50130 conn->flags:101C3 conn->refcnt:2
> Jan 24 23:05:45 pc01 kernel: IPVS: TCP input  [S...] 172.17.0.16:50130->
> 172.17.0.2:37456 state: NONE->SYN_RECV conn->refcnt:2
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
> 172.17.0.2:37456 hit
> Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response,
> net/netfilter/ipvs/ip_vs_core.c line 1094
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456->
> 172.17.0.16:50130 not hit
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456->
> 172.17.0.16:50130 hit
> Jan 24 23:05:45 pc01 kernel: IPVS: TCP input  [..A.] 172.17.0.16:50130->
> 172.17.0.2:37456 state: SYN_RECV->ESTABLISHED conn->refcnt:2
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456->
> 172.17.0.16:50130 not hit
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456->
> 172.17.0.16:50130 hit
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
> 172.17.0.2:37456 hit
> Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response,
> net/netfilter/ipvs/ip_vs_core.c line 1094
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
> 172.17.0.2:37456 hit
> Jan 24 23:05:45 pc01 kernel: IPVS: TCP output  [.FA.] 172.17.0.16:50130->
> 172.17.0.2:37456 state: ESTABLISHED->FIN_WAIT conn->refcnt:2
> Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response,
> net/netfilter/ipvs/ip_vs_core.c line 1094
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456->
> 172.17.0.16:50130 not hit
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456->
> 172.17.0.16:50130 hit
> Jan 24 23:05:45 pc01 kernel: IPVS: TCP input  [.FA.] 172.17.0.16:50130->
> 172.17.0.2:37456 state: FIN_WAIT->TIME_WAIT conn->refcnt:2
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
> 172.17.0.2:37456 hit
> Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response,
> net/netfilter/ipvs/ip_vs_core.c line 1094
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out UDP 172.17.0.16:50014->
> 239.192.0.1:50015 not hit
> Jan 24 23:05:45 pc01 kernel: IPVS: packet type=2 proto=17
> daddr=239.192.0.1 ignored in hook 1
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out UDP 127.0.0.1:45176->
> 127.0.0.1:53 not hit
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in UDP 127.0.0.1:45176->
> 127.0.0.1:53 not hit
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup service: fwm 0 UDP 127.0.0.1:53not 
> hit
>
> Below is an example of good results when connecting directly to port
> 50000.  For this scenario I removed port 80 and updated iptables with
> fwmark for port 50000:
> iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp
> --dport 50000 -j MARK --set-xmark 0x64/0xffffffff
>
> Debug output on real server 1 when not port mapping first test (curl -v
> 'http://172.17.0.24:50000'):
>
> Jan 25 00:19:37 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling...
> Jan 25 00:19:37 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0
> refcnt 1 weight 100
> Jan 25 00:19:37 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:42815 v:
> 172.17.0.24:50130 d:172.17.0.17:50130 fwd:R s:4 conn->flags:183
> conn->refcnt:1 dest->refcnt:2
> Jan 25 00:19:37 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:42815 v:
> 172.17.0.24:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2
> Jan 25 00:19:37 pc01 kernel: IPVS: TCP input  [S...] 172.17.0.17:50130->
> 172.17.0.2:42815 state: NONE->SYN_RECV conn->refcnt:2
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:37 pc01 kernel: IPVS: new dst 172.17.0.17, src 172.17.0.16,
> refcnt=1
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
> 172.17.0.24:50130 not hit
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
> 172.17.0.24:50130 hit
> Jan 25 00:19:37 pc01 kernel: IPVS: TCP input  [..A.] 172.17.0.17:50130->
> 172.17.0.2:42815 state: SYN_RECV->ESTABLISHED conn->refcnt:2
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
> 172.17.0.24:50130 not hit
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
> 172.17.0.24:50130 hit
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
> 172.17.0.24:50130 not hit
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
> 172.17.0.24:50130 hit
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
> 172.17.0.24:50130 not hit
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
> 172.17.0.24:50130 hit
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
> 172.17.0.24:50130 not hit
> Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
> 172.17.0.24:50130 hit
> Jan 25 00:19:37 pc01 kernel: IPVS: TCP input  [.FA.] 172.17.0.17:50130->
> 172.17.0.2:42815 state: ESTABLISHED->FIN_WAIT conn->refcnt:2
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
>
> Debug output on real server 1 when not port mapping second test (curl -v
> 'http://172.17.0.24:50000'):
>
> Jan 25 00:19:39 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling...
> Jan 25 00:19:39 pc01 kernel: IPVS: RR: server 172.17.0.16:0 activeconns 0
> refcnt 1 weight 100
> Jan 25 00:19:39 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:42816 v:
> 172.17.0.24:50130 d:172.17.0.16:50130 fwd:R s:65276 conn->flags:183
> conn->refcnt:1 dest->refcnt:2
> Jan 25 00:19:39 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:42816 v:
> 172.17.0.24:50130 d:172.17.0.16:50130 conn->flags:101C3 conn->refcnt:2
> Jan 25 00:19:39 pc01 kernel: IPVS: TCP input  [S...] 172.17.0.16:50130->
> 172.17.0.2:42816 state: NONE->SYN_RECV conn->refcnt:2
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:39 pc01 kernel: IPVS: new dst 172.17.0.16, src 172.17.0.16,
> refcnt=1
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130->
> 172.17.0.2:42816 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130->
> 172.17.0.2:42816 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup service: fwm 0 TCP
> 172.17.0.2:42816 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816->
> 172.17.0.24:50130 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816->
> 172.17.0.24:50130 hit
> Jan 25 00:19:39 pc01 kernel: IPVS: TCP input  [..A.] 172.17.0.16:50130->
> 172.17.0.2:42816 state: SYN_RECV->ESTABLISHED conn->refcnt:2
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816->
> 172.17.0.24:50130 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816->
> 172.17.0.24:50130 hit
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130->
> 172.17.0.2:42816 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130->
> 172.17.0.2:42816 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130->
> 172.17.0.2:42816 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130->
> 172.17.0.2:42816 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816->
> 172.17.0.24:50130 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816->
> 172.17.0.24:50130 hit
> Jan 25 00:19:39 pc01 kernel: IPVS: TCP input  [.FA.] 172.17.0.16:50130->
> 172.17.0.2:42816 state: ESTABLISHED->FIN_WAIT conn->refcnt:2
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130->
> 172.17.0.2:42816 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130->
> 172.17.0.2:42816 not hit
> Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:39545->
> 172.17.0.16:3306 not hit
>
> The tcpdump command that was used was as follows on real server 1:
> tcpdump -iany -nn port 80 or port 50000
>
> I realized later that using 'any' device isn't as helpful when trying to
> pinpoint loopback traffic, so that's what my follow up email was referring
> to.
>
> Thanks again for the support, feel free to ask for any additional
> information to help debug.
>
> Jacoby
>
>
> On Sat, Jan 25, 2014 at 6:25 AM, Julian Anastasov <ja@xxxxxx> wrote:
>
>>
>>         Hello,
>>
>> On Thu, 23 Jan 2014, Jacoby Hickerson wrote:
>>
>> > Just to clarify the packets are going to the loopback of node 1, when
>> they
>> > should be going to node 2.  This is shown in the tcpdump output:Here is
>> the
>> > output from the lo device of the first node:
>> > 02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack
>> > 2970678458, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
>> length
>> > 0
>> > 02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [P.], seq
>> > 0:173, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
>> > length 173
>> > 02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack
>> 2,
>> > win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
>> > 02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [F.], seq
>> > 173, ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484],
>> length
>> > 0
>>
>> ...
>>
>> > Packets are being sent from the RIP of the first node only.  From my
>> > understanding when using DR OutPkts should always be zero.
>>
>>         When LocalNode (local RIP) is used, we can see
>> the local reply in LOCAL_OUT hook. It happens for NAT but
>> also for DR. So, it is normal. But we see these replies
>> after DNAT in LOCAL_OUT, see ip_vs_ops[] for reference.
>>
>> > The end result is that the packets are always coming from the first
>> > node and never balanced to the second node.
>> >
>> > Thanks for any further help, seems the solution is really close!
>>
>>         Can you provide more understandable description
>> for the test, for example:
>>
>> - client box:
>> IP1: X.X.X.X/N dev DEV
>> IP2: ...
>>
>> - director:
>> IP1: ...
>> VIP: XXX
>> are client and director same box
>>
>> - real server:
>> IP1: ...
>>
>>         iptable rules used. By this way I can try to
>> duplicate the problem. Now I see some IPs in tcpdump
>> output but I'm not sure what kind of traffic is shown,
>> where is started the tcpdump, on what box, on what
>> interface, external, internal...
>>
>> Regards
>>
>> --
>> Julian Anastasov <ja@xxxxxx>
>>
>
>

Attachment: ipvs_debug_output
Description: Binary data

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
[More with this subject...]
<Prev in Thread] Current Thread [Next in Thread>
Previous by Date:  Re: [lvs-users] Port mapping with LVS-DR using fwmark, Jacoby Hickerson
Next by Date:  Re: [lvs-users] Port mapping with LVS-DR using fwmark, Julian Anastasov
Previous by Thread:  Re: [lvs-users] Port mapping with LVS-DR using fwmark, Jacoby Hickerson
Next by Thread:  Re: [lvs-users] Port mapping with LVS-DR using fwmark, Julian Anastasov
Indexes:  [Date] [Thread] [Top] [All Lists]