|
Hello,
On Mon, 27 Jan 2014, Jacoby Hickerson wrote:
> Certainly and that makes sense, I will consolidate what I've emailed before
> with the additional information here.
>
> # PC info: Linux 3.12.5 for real servers 1 and 2, and Linux 3.9.10 for the
> client box.
>
> There are 3 boxes total, client box, director/RIP1( real server 1) and RIP2
> (real server 2):
> - client box:
> inet 172.17.0.2/16 brd 172.17.255.255 scope global eth1 #CIP
>
> - director which is the same as real server 1 (RIP1). The client is on a
> separate box.
> inet 172.17.0.16/16 brd 172.17.255.255 scope global bond0
> #RIP1
> inet 172.17.0.24/16 brd 172.17.255.255 scope global secondary bond0:2 #VIP
>
> - real server 2 (RIP2)
> inet 172.17.0.24/32 scope global lo:0 #VIP on loopback
> inet 172.17.0.17/16 brd 172.17.255.255 scope global bond0 #RIP2
>
> # ipvs setup on real server 1 (RIP1) only
> ipvsadm -C
> ipvsadm -A -f 100 -s rr
> ipvsadm -a -f 100 -r 172.17.0.16 -w 100
> ipvsadm -a -f 100 -r 172.17.0.17 -w 100
>
> # iptable rules (these rules are set for both real server 1 and real server
> 2)
> iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp
> --dport 80 -j MARK --set-xmark 0x64/0xffffffff
> iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT
> --to-ports 50000
> iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 80 -j REDIRECT
> --to-ports 50000
>
> The test I'm conducting is an http get from the client box connecting to the
> VIP:
> - Issue the following command on the client box:
> curl -v 'http://172.17.0.24'
>
> On both real servers there is an nginx webserver listening on port 50000
>
> I also turned on debugging and ran the curl command with port mapping using
> level 12 debug (this is output when the issue occurs of no load balancing).
> Debug output on real server 1 after executing the curl command the first
> time:
>
> Jan 24 23:05:44 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0
> refcnt 1 weight 100
The debug output was very helpful.
Looks like -j REDIRECT combined with DR is a bad idea.
When packet comes to IPVS the daddr is already 172.17.0.16,
see the "v:172.17.0.16" line below:
> Jan 24 23:05:44 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37455
> v:172.17.0.16:50130 d:172.17.0.17:50130 fwd:R s:65276 conn->flags:183
> conn->refcnt:1 dest->refcnt:2
The remote real server 2 is not configured for
such VIP (172.17.0.16). I don't remember when was
-j REDIRECT used for IPVS setups, may be for transparent
proxy setups.
Why not just use NAT method for both servers
without any REDIRECT rules?
Even -j DNAT --to-destination VIP:50000 has better
chance to use VIP instead of first IP.
> Jan 24 23:05:44 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37455
> v:172.17.0.16:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2
> Jan 24 23:05:44 pc01 kernel: IPVS: TCP input [S...]
> 172.17.0.17:50130->172.17.0.2:37455 state: NONE->SYN_RECV conn->refcnt:2
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
Above "ip_vs_xmit.c line 1031" means packet was
sent to remote real server 2 (172.17.0.17) but due to
-j REDIRECT the daddr is 172.17.0.16.
...
> Debug output on real server 1 after executing the curl command a second
> time:
>
> Jan 24 23:05:45 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling...
> Jan 24 23:05:45 pc01 kernel: IPVS: RR: server 172.17.0.16:0 activeconns 0
> refcnt 1 weight 100
> Jan 24 23:05:45 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37456
> v:172.17.0.16:50130 d:172.17.0.16:50130 fwd:R s:65276 conn->flags:183
> conn->refcnt:1 dest->refcnt:2
> Jan 24 23:05:45 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37456
> v:172.17.0.16:50130 d:172.17.0.16:50130 conn->flags:101C3 conn->refcnt:2
> Jan 24 23:05:45 pc01 kernel: IPVS: TCP input [S...]
> 172.17.0.16:50130->172.17.0.2:37456 state: NONE->SYN_RECV conn->refcnt:2
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
No "ip_vs_xmit.c line 1031" here, packet was delivered
locally with NF_ACCEPT, so it goes to local real server
as per the "d:172.17.0.16" info.
> Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP
> 172.17.0.16:50130->172.17.0.2:37456 hit
...
> Below is an example of good results when connecting directly to port 50000.
So, no -j REDIRECT => no problem?
> For this scenario I removed port 80 and updated iptables with fwmark for
> port 50000:
> iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp
> --dport 50000 -j MARK --set-xmark 0x64/0xffffffff
>
> Debug output on real server 1 when not port mapping first test (curl -v
> 'http://172.17.0.24:50000'):
>
> Jan 25 00:19:37 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling...
> Jan 25 00:19:37 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0
> refcnt 1 weight 100
> Jan 25 00:19:37 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:42815
> v:172.17.0.24:50130 d:172.17.0.17:50130 fwd:R s:4 conn->flags:183
> conn->refcnt:1 dest->refcnt:2
Yep, "v:172.17.0.24" means no -j REDIRECT was used.
> Jan 25 00:19:37 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:42815
> v:172.17.0.24:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2
> Jan 25 00:19:37 pc01 kernel: IPVS: TCP input [S...]
> 172.17.0.17:50130->172.17.0.2:42815 state: NONE->SYN_RECV conn->refcnt:2
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1009
> Jan 25 00:19:37 pc01 kernel: IPVS: new dst 172.17.0.17, src 172.17.0.16,
> refcnt=1
> Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out,
> net/netfilter/ipvs/ip_vs_core.c line 1116
> Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit,
> net/netfilter/ipvs/ip_vs_xmit.c line 1031
Regards
--
Julian Anastasov <ja@xxxxxx>_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
