|
LVS users,
I have a "one arm NAT" configuration with so-called "local node" that I'm
trying to make work with LVS.
I have two servers, syslog servers (call them A and B), and one client (we'll
call C). Both A and B also run LVS with keepalived. There is a VIP address
managed by keepalived. The idea is that when client C contacts the VIP, it will
hit either A or B based on LVS scheduling. I'm doing both UDP and TCP
protocols. For purposes of this discussion with respect to packets and the IP
addresses, I'll use the format (source IP, destination IP).
When client contacts the VIP, a packet is sent with (C-IP, VIP). It arrives at
A that holds the VIP at the moment, and proceeds to LOCAL_IN and LVS-NAT
changes the packet to either (C-IP, A-IP) or (C-IP, B-IP). In the former case
(C-IP, A-IP) case, it gets processed by the local syslog process. In the later
case (C-IP, B-IP), it gets sent back out to B. However, because the packet is
(C-IP, B-IP), B's replies go to (B-IP, C-IP) and doesn't get processed by the
LVS-NAT anymore. Connection hangs... as the C is expecting reply from (VIP,
C-IP).
I know in reading the various FAQ and postings about "one arm NAT" LVS-NAT
situations is to have the "real servers" be the default GW. This won't work for
me since the real servers are also the LVS servers. However, I was thinking
that if I can use netfilter/iptables to source NAT the load balanced packets, I
can change the (C-IP,B-IP) show up as (A-IP, B-IP) instead, and hence B's
replies would go back to A (the current LVS-NAT) and sent back to C as (VIP,
C-IP). However, the packet doesn't seem to traverse the "netfliter stack" as
expected:
(i setup a -j LOG rule within every chain [PREROUTING, INPUT, FORWARD, OUTPUT,
POSTROUTING] of every table [mangle, nat, filter], and watch the packet
traverse the "netfilter stack")
Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-nat-PRER: IN=eth0 OUT=
MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35
LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16
Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-INPU: IN=eth0 OUT=
MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35
LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16
Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-INPU: IN=eth0 OUT=
MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35
LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16
Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-OUTP: IN= OUT=eth1
SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF
PROTO=UDP SPT=39178 DPT=514 LEN=16
Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-OUTP: IN= OUT=eth1
SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF
PROTO=UDP SPT=39178 DPT=514 LEN=16
Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-POSR: IN= OUT=eth1
SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF
PROTO=UDP SPT=39178 DPT=514 LEN=16
This is a packet from (C-IP, VIP), goes through
NAT/PREROUTING->ROUTING->MANGLE/INPUT->FILTER/INPUT->(LVS-NAT?)->ROUTING->MANGLE/OUTPUT->FILTER/OUTPUT->MANGLE/POSTROUTING.
What I see missing are MANGLE/PREROUTING and particularly interested in the
missing NAT/POSTROUTING, where I was hoping to SNAT the packet.
According to this post:
http://archive.linuxvirtualserver.org/html/lvs-users/2011-03/msg00019.html
It would appear that at some point, LVS-NAT packets did traverse
NAT/POSTROUTING and one could SNAT the LVS outbound packets?
I'm using:
RHEL 6.5 with RH kernel 2.6.32-431.17.1.el6.x86_64
ipvsadm-1.26-2.el6.x86_64
keepalived-1.2.13-1.el6.x86_64
here are the -j LOG iptables rules I'm using:
[root@sprdrsyas400 ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 293K packets, 29M bytes)
pkts bytes target prot opt in out source destination
4 144 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-PRER: '
Chain INPUT (policy ACCEPT 293K packets, 29M bytes)
pkts bytes target prot opt in out source destination
4 144 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-INPU: '
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-FORW: '
Chain OUTPUT (policy ACCEPT 362K packets, 18M bytes)
pkts bytes target prot opt in out source destination
2 72 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-OUTP: '
Chain POSTROUTING (policy ACCEPT 362K packets, 18M bytes)
pkts bytes target prot opt in out source destination
2 72 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-POSR: '
[root@sprdrsyas400 ~]# iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 8739 packets, 500K bytes)
pkts bytes target prot opt in out source destination
7 252 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-PRER: '
Chain POSTROUTING (policy ACCEPT 25612 packets, 1374K bytes)
pkts bytes target prot opt in out source destination
0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-POSR: '
Chain OUTPUT (policy ACCEPT 25612 packets, 1374K bytes)
pkts bytes target prot opt in out source destination
0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-OUTP: '
[root@sprdrsyas400 ~]# iptables -vnL
Chain INPUT (policy ACCEPT 292K packets, 29M bytes)
pkts bytes target prot opt in out source destination
4 144 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-INPU: '
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-FORW: '
Chain OUTPUT (policy ACCEPT 361K packets, 18M bytes)
pkts bytes target prot opt in out source destination
2 72 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-OUTP: '
And this is LVS on A:
[root@sprdrsyas400 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP 10.180.88.35:514 rr ops
-> 10.180.88.29:514 Local 1 0 0
-> 10.180.88.30:514 Masq 1 0 0
I would appreciate any help...
Thanks in advance,
-Bond
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
