|
Hello,
On Tue, 11 Dec 2018, Bogdan Rudas wrote:
> I try to do load-balance the service which is listening both TCP and UDP
> using same port number *1111*. There are set of real servers, each of them
> host few application instances listening on different ports (1111/udp,
> 1116/udp, 1115/tcp, 1117/tcp)
> I used fwmark to group connection by protocol, i.e. 1111/udp have *fwmark 1*,
> 1111/tcp have *fwmark 2.*
> LVS-NAT mode is in use.
> What I see is when the client connection coming to 1111/*TCP* the LVS try
> to pass it to RTI:1111/*TCP *while the RIP have TCP listeners on ports 1115
> and 1117 and LVS was configured accordingly.
This is a long-standing behaviour to ignore rport for real servers
added to persistent fwmark-based virtual service. What can be changed is
to depend on rport being 0 or not:
# Map services 1:1 (port is preserved)
-r RIP:0 => translate only daddr=VIP->RIP
# Map all vports to same rport
-r RIP:RPORT => translate both address and port
But we risk to change behaviour to existing installations.
> Here is test connection info:
> # ipvsadm -L -n -c
> IPVS connection entries
> pro expire state source virtual destination
> *TCP* 00:55 SYN_RECV 113.25.79.34:38204 115.5.6.138*:1111 *10.1.2.13
> *:1111* *<<<< It should be RIP:1115 or RIP:1117 HERE!*
> IP 00:49 NONE 113.25.79.34:0 0.0.0.2:0 10.1.2.13:0
> TCP 01:00 SYN_RECV 113.25.79.34:38206 115.5.6.138:1111 10.1.2.13:1111
>
> Below are saved firewall and ipvs configurations:
>
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *mangle
> :PREROUTING ACCEPT [9507:869411]
> :INPUT ACCEPT [7637:700414]
> :FORWARD ACCEPT [70:10945]
> :OUTPUT ACCEPT [8781:1045717]
> :POSTROUTING ACCEPT [8851:1056662]
> COMMIT
> # Completed on Tue Dec 11 17:02:29 2018
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *filter
> :INPUT ACCEPT [0:0]
> :FORWARD ACCEPT [82:11111]
> :OUTPUT ACCEPT [16923:2797363]
> -A INPUT -p icmp -m comment --comment "000 accept all icmp" -j ACCEPT
> -A INPUT -i lo -m comment --comment "001 accept all to lo interface" -j
> ACCEPT
> -A INPUT -d 127.0.0.0/8 ! -i lo -m comment --comment "002 reject local
> traffic not on loopback interface" -j REJECT --reject-with
> icmp-port-unreachable
> -A INPUT -m comment --comment "003 accept related established rules" -m
> state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT -p tcp -m multiport --dports 22 -m comment --comment "100 accept
> ssh connection" -m state --state NEW -j ACCEPT
> -A INPUT -d 224.0.0.0/8 -p vrrp -m comment --comment "104 Accept VRRP
> proto" -j ACCEPT
> -A INPUT -m comment --comment "999 drop all" -j REJECT --reject-with
> icmp-port-unreachable
> COMMIT
> # Completed on Tue Dec 11 17:02:29 2018
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *nat
> :PREROUTING ACCEPT [42:2928]
> :INPUT ACCEPT [0:0]
> :OUTPUT ACCEPT [4:360]
> :POSTROUTING ACCEPT [4:360]
> -A PREROUTING -p tcp -m multiport --dports 1111 -j MARK --set-xmark
> 0x2/0xffffffff
> -A PREROUTING -p *udp* -m multiport --dports 1111 -m comment --comment "101
> MARK udp 1111 traffic from everywhere" -j MARK --set-xmark *0x1*/0xffffffff
> -A PREROUTING -p *tcp* -m multiport --dports 1111 -m comment --comment "101
> MARK tcp 1111 traffic from everywhere" -j MARK --set-xmark *0x2*/0xffffffff
> COMMIT
> # Completed on Tue Dec 11 17:02:29 2018
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *raw
> :PREROUTING ACCEPT [17569:3914462]
> :OUTPUT ACCEPT [16925:2798776]
> COMMIT
>
> # ipvsadm -S
> -A -f 1 -s wrr -p 50
> -a -f 1 -r 10.1.2.12:1111 -m -w 2
> -a -f 1 -r 10.1.2.12:1116 -m -w 2
This is not supported: many real services on same real IP
when persistence is used for fwmark. Because: packets to different vport
can be marked with same fwmark to be handled by same virtual service.
As result, the fwmark template <IPPROTO_IP,caddr,0,fwmark,0,daddr,0> does
not use dport from the packet and we can not distinguish the real services
that differ just by rport.
In short, virtual service that can handle traffic to different
vports can not use persistence together with many real services on same
IP.
> -a -f 1 -r 10.1.2.13:1111 -m -w 2
> -a -f 1 -r 10.1.2.13:1116 -m -w 2
> -a -f 1 -r 10.1.2.14:1111 -m -w 2
> -a -f 1 -r 10.1.2.14:1116 -m -w 2
> -A -f 2 -s wrr -p 50
> -a -f 2 -r 10.1.2.12:1115 -m -w 2
> -a -f 2 -r 10.1.2.12:1117 -m -w 2
> -a -f 2 -r 10.1.2.13:1115 -m -w 2
> -a -f 2 -r 10.1.2.13:1117 -m -w 2
> -a -f 2 -r 10.1.2.14:1115 -m -w 2
> -a -f 2 -r 10.1.2.14:1117 -m -w 2
>
>
> # ipvsadm -L -n
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> FWM 1 wrr persistent 50
> -> 10.1.2.12:1111 Masq 2 0 0
> -> 10.1.2.12:1116 Masq 2 0 0
> -> 10.1.2.13:1111 Masq 2 0 0
> -> 10.1.2.13:1116 Masq 2 0 0
> -> 10.1.2.14:1111 Masq 2 0 0
> -> 10.1.2.14:1116 Masq 2 0 0
> FWM 2 wrr persistent 50
> -> 10.1.2.12:1115 Masq 2 0 0
> -> 10.1.2.12:1117 Masq 2 0 0
> -> 10.1.2.13:1115 Masq 2 0 0
> -> 10.1.2.13:1117 Masq 2 0 1
> -> 10.1.2.14:1115 Masq 2 0 0
> -> 10.1.2.14:1117 Masq 2 0 0
>
>
> OS is CentOS 7 with kernel *3.10*
>
> Thank you.
Regards
--
Julian Anastasov <ja@xxxxxx>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
