LVS
lvs-users
[Top] [All Lists]
Google
 
Web LinuxVirtualServer.org
<prev [Date] next> <prev [Thread] next>

Re: [lvs-users] Port mapping with LVS-DR using fwmark

from [Jacoby Hickerson] [Permanent Link]
To: Julian Anastasov <ja@xxxxxx>
Subject: Re: [lvs-users] Port mapping with LVS-DR using fwmark
Cc: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
From: Jacoby Hickerson <hickersonjl@xxxxxxxxx>
Date: Mon, 27 Jan 2014 16:00:48 -0800 
Certainly and that makes sense, I will consolidate what I've emailed before
with the additional information here.

# PC info: Linux 3.12.5 for real servers 1 and 2, and Linux 3.9.10 for the
client box.

There are 3 boxes total, client box, director/RIP1( real server 1) and RIP2
(real server 2):
- client box:
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth1   #CIP

- director which is the same as real server 1 (RIP1).  The client is on a
separate box.
inet 172.17.0.16/16 brd 172.17.255.255 scope global bond0
#RIP1
inet 172.17.0.24/16 brd 172.17.255.255 scope global secondary bond0:2   #VIP

- real server 2 (RIP2)
inet 172.17.0.24/32 scope global lo:0                      #VIP on loopback
inet 172.17.0.17/16 brd 172.17.255.255 scope global bond0  #RIP2

# ipvs setup on real server 1 (RIP1) only
ipvsadm -C
ipvsadm -A -f 100 -s rr
ipvsadm -a -f 100 -r 172.17.0.16 -w 100
ipvsadm -a -f 100 -r 172.17.0.17 -w 100

# iptable rules (these rules are set for both real server 1 and real server
2)
iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp
--dport 80 -j MARK --set-xmark 0x64/0xffffffff
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT
--to-ports 50000
iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 80 -j REDIRECT
--to-ports 50000

The test I'm conducting is an http get from the client box connecting to
the VIP:
- Issue the following command on the client box:
curl -v 'http://172.17.0.24'

On both real servers there is an nginx webserver listening on port 50000

I also turned on debugging and ran the curl command with port mapping using
level 12 debug (this is output when the issue occurs of no load balancing).
Debug output on real server 1 after executing the curl command the first
time:

Jan 24 23:05:44 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0
refcnt 1 weight 100
Jan 24 23:05:44 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37455 v:
172.17.0.16:50130 d:172.17.0.17:50130 fwd:R s:65276 conn->flags:183
conn->refcnt:1 dest->refcnt:2
Jan 24 23:05:44 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37455 v:
172.17.0.16:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2
Jan 24 23:05:44 pc01 kernel: IPVS: TCP input  [S...] 172.17.0.17:50130->
172.17.0.2:37455 state: NONE->SYN_RECV conn->refcnt:2
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1031
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
172.17.0.2:37455 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130->
172.17.0.2:37455 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: lookup service: fwm 0 TCP
172.17.0.2:37455 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455->
172.17.0.16:50130 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455->
172.17.0.16:50130 hit
Jan 24 23:05:44 pc01 kernel: IPVS: TCP input  [..A.] 172.17.0.17:50130->
172.17.0.2:37455 state: SYN_RECV->ESTABLISHED conn->refcnt:2
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1031
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455->
172.17.0.16:50130 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455->
172.17.0.16:50130 hit
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1031
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
172.17.0.2:37455 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130->
172.17.0.2:37455 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
172.17.0.2:37455 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130->
172.17.0.2:37455 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455->
172.17.0.16:50130 not hit
Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455->
172.17.0.16:50130 hit
Jan 24 23:05:44 pc01 kernel: IPVS: TCP input  [.FA.] 172.17.0.17:50130->
172.17.0.2:37455 state: ESTABLISHED->FIN_WAIT conn->refcnt:2

Debug output on real server 1 after executing the curl command a second
time:

Jan 24 23:05:45 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling...
Jan 24 23:05:45 pc01 kernel: IPVS: RR: server 172.17.0.16:0 activeconns 0
refcnt 1 weight 100
Jan 24 23:05:45 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37456 v:
172.17.0.16:50130 d:172.17.0.16:50130 fwd:R s:65276 conn->flags:183
conn->refcnt:1 dest->refcnt:2
Jan 24 23:05:45 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37456 v:
172.17.0.16:50130 d:172.17.0.16:50130 conn->flags:101C3 conn->refcnt:2
Jan 24 23:05:45 pc01 kernel: IPVS: TCP input  [S...] 172.17.0.16:50130->
172.17.0.2:37456 state: NONE->SYN_RECV conn->refcnt:2
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
172.17.0.2:37456 hit
Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response,
net/netfilter/ipvs/ip_vs_core.c line 1094
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456->
172.17.0.16:50130 not hit
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456->
172.17.0.16:50130 hit
Jan 24 23:05:45 pc01 kernel: IPVS: TCP input  [..A.] 172.17.0.16:50130->
172.17.0.2:37456 state: SYN_RECV->ESTABLISHED conn->refcnt:2
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456->
172.17.0.16:50130 not hit
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456->
172.17.0.16:50130 hit
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
172.17.0.2:37456 hit
Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response,
net/netfilter/ipvs/ip_vs_core.c line 1094
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
172.17.0.2:37456 hit
Jan 24 23:05:45 pc01 kernel: IPVS: TCP output  [.FA.] 172.17.0.16:50130->
172.17.0.2:37456 state: ESTABLISHED->FIN_WAIT conn->refcnt:2
Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response,
net/netfilter/ipvs/ip_vs_core.c line 1094
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456->
172.17.0.16:50130 not hit
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456->
172.17.0.16:50130 hit
Jan 24 23:05:45 pc01 kernel: IPVS: TCP input  [.FA.] 172.17.0.16:50130->
172.17.0.2:37456 state: FIN_WAIT->TIME_WAIT conn->refcnt:2
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130->
172.17.0.2:37456 hit
Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response,
net/netfilter/ipvs/ip_vs_core.c line 1094
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out UDP 172.17.0.16:50014->
239.192.0.1:50015 not hit
Jan 24 23:05:45 pc01 kernel: IPVS: packet type=2 proto=17 daddr=239.192.0.1
ignored in hook 1
Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out UDP 127.0.0.1:45176->
127.0.0.1:53 not hit
Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in UDP 127.0.0.1:45176->
127.0.0.1:53 not hit
Jan 24 23:05:45 pc01 kernel: IPVS: lookup service: fwm 0 UDP 127.0.0.1:53not hit

Below is an example of good results when connecting directly to port 50000.
 For this scenario I removed port 80 and updated iptables with fwmark for
port 50000:
iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp
--dport 50000 -j MARK --set-xmark 0x64/0xffffffff

Debug output on real server 1 when not port mapping first test (curl -v
'http://172.17.0.24:50000'):

Jan 25 00:19:37 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling...
Jan 25 00:19:37 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0
refcnt 1 weight 100
Jan 25 00:19:37 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:42815 v:
172.17.0.24:50130 d:172.17.0.17:50130 fwd:R s:4 conn->flags:183
conn->refcnt:1 dest->refcnt:2
Jan 25 00:19:37 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:42815 v:
172.17.0.24:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2
Jan 25 00:19:37 pc01 kernel: IPVS: TCP input  [S...] 172.17.0.17:50130->
172.17.0.2:42815 state: NONE->SYN_RECV conn->refcnt:2
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:37 pc01 kernel: IPVS: new dst 172.17.0.17, src 172.17.0.16,
refcnt=1
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1031
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
172.17.0.24:50130 not hit
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
172.17.0.24:50130 hit
Jan 25 00:19:37 pc01 kernel: IPVS: TCP input  [..A.] 172.17.0.17:50130->
172.17.0.2:42815 state: SYN_RECV->ESTABLISHED conn->refcnt:2
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1031
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
172.17.0.24:50130 not hit
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
172.17.0.24:50130 hit
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1031
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
172.17.0.24:50130 not hit
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
172.17.0.24:50130 hit
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1031
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
172.17.0.24:50130 not hit
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
172.17.0.24:50130 hit
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1031
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815->
172.17.0.24:50130 not hit
Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815->
172.17.0.24:50130 hit
Jan 25 00:19:37 pc01 kernel: IPVS: TCP input  [.FA.] 172.17.0.17:50130->
172.17.0.2:42815 state: ESTABLISHED->FIN_WAIT conn->refcnt:2
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116

Debug output on real server 1 when not port mapping second test (curl -v
'http://172.17.0.24:50000'):

Jan 25 00:19:39 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling...
Jan 25 00:19:39 pc01 kernel: IPVS: RR: server 172.17.0.16:0 activeconns 0
refcnt 1 weight 100
Jan 25 00:19:39 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:42816 v:
172.17.0.24:50130 d:172.17.0.16:50130 fwd:R s:65276 conn->flags:183
conn->refcnt:1 dest->refcnt:2
Jan 25 00:19:39 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:42816 v:
172.17.0.24:50130 d:172.17.0.16:50130 conn->flags:101C3 conn->refcnt:2
Jan 25 00:19:39 pc01 kernel: IPVS: TCP input  [S...] 172.17.0.16:50130->
172.17.0.2:42816 state: NONE->SYN_RECV conn->refcnt:2
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:39 pc01 kernel: IPVS: new dst 172.17.0.16, src 172.17.0.16,
refcnt=1
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130->
172.17.0.2:42816 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130->
172.17.0.2:42816 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: lookup service: fwm 0 TCP
172.17.0.2:42816 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816->
172.17.0.24:50130 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816->
172.17.0.24:50130 hit
Jan 25 00:19:39 pc01 kernel: IPVS: TCP input  [..A.] 172.17.0.16:50130->
172.17.0.2:42816 state: SYN_RECV->ESTABLISHED conn->refcnt:2
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816->
172.17.0.24:50130 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816->
172.17.0.24:50130 hit
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130->
172.17.0.2:42816 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130->
172.17.0.2:42816 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130->
172.17.0.2:42816 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130->
172.17.0.2:42816 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816->
172.17.0.24:50130 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816->
172.17.0.24:50130 hit
Jan 25 00:19:39 pc01 kernel: IPVS: TCP input  [.FA.] 172.17.0.16:50130->
172.17.0.2:42816 state: ESTABLISHED->FIN_WAIT conn->refcnt:2
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
net/netfilter/ipvs/ip_vs_xmit.c line 1009
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130->
172.17.0.2:42816 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130->
172.17.0.2:42816 not hit
Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out,
net/netfilter/ipvs/ip_vs_core.c line 1116
Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:39545->
172.17.0.16:3306 not hit

The tcpdump command that was used was as follows on real server 1:
tcpdump -iany -nn port 80 or port 50000

I realized later that using 'any' device isn't as helpful when trying to
pinpoint loopback traffic, so that's what my follow up email was referring
to.

Thanks again for the support, feel free to ask for any additional
information to help debug.

Jacoby


On Sat, Jan 25, 2014 at 6:25 AM, Julian Anastasov <ja@xxxxxx> wrote:

>
>         Hello,
>
> On Thu, 23 Jan 2014, Jacoby Hickerson wrote:
>
> > Just to clarify the packets are going to the loopback of node 1, when
> they
> > should be going to node 2.  This is shown in the tcpdump output:Here is
> the
> > output from the lo device of the first node:
> > 02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack
> > 2970678458, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
> length
> > 0
> > 02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [P.], seq
> > 0:173, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483],
> > length 173
> > 02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack
> 2,
> > win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
> > 02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [F.], seq
> > 173, ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484],
> length
> > 0
>
> ...
>
> > Packets are being sent from the RIP of the first node only.  From my
> > understanding when using DR OutPkts should always be zero.
>
>         When LocalNode (local RIP) is used, we can see
> the local reply in LOCAL_OUT hook. It happens for NAT but
> also for DR. So, it is normal. But we see these replies
> after DNAT in LOCAL_OUT, see ip_vs_ops[] for reference.
>
> > The end result is that the packets are always coming from the first
> > node and never balanced to the second node.
> >
> > Thanks for any further help, seems the solution is really close!
>
>         Can you provide more understandable description
> for the test, for example:
>
> - client box:
> IP1: X.X.X.X/N dev DEV
> IP2: ...
>
> - director:
> IP1: ...
> VIP: XXX
> are client and director same box
>
> - real server:
> IP1: ...
>
>         iptable rules used. By this way I can try to
> duplicate the problem. Now I see some IPs in tcpdump
> output but I'm not sure what kind of traffic is shown,
> where is started the tcpdump, on what box, on what
> interface, external, internal...
>
> Regards
>
> --
> Julian Anastasov <ja@xxxxxx>
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
[More with this subject...]
<Prev in Thread] Current Thread [Next in Thread>
Previous by Date:  [lvs-users] prasunb25@xxxxxxxxx has been waiting for 6 days. Accept?, prasunb25
Next by Date:  Re: [lvs-users] Port mapping with LVS-DR using fwmark, Jacoby Hickerson
Previous by Thread:  Re: [lvs-users] Port mapping with LVS-DR using fwmark, Julian Anastasov
Next by Thread:  Re: [lvs-users] Port mapping with LVS-DR using fwmark, Jacoby Hickerson
Indexes:  [Date] [Thread] [Top] [All Lists]