LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

[lvs-users] so-called "one arm NAT" LVS-NAT and interaction with netfilt

To: "lvs-users@xxxxxxxxxxxxxxxxxxxxxx" <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: [lvs-users] so-called "one arm NAT" LVS-NAT and interaction with netfilter/iptables
From: "Masuda, Bond" <Bond_Masuda@xxxxxxxxxx>
Date: Thu, 26 Jun 2014 18:18:28 +0000
LVS users,

I have a "one arm NAT" configuration with so-called "local node" that I'm 
trying to make work with LVS.

I have two servers, syslog servers (call them A and B), and one client (we'll 
call C). Both A and B also run LVS with keepalived. There is a VIP address 
managed by keepalived. The idea is that when client C contacts the VIP, it will 
hit either A or B based on LVS scheduling. I'm doing both UDP and TCP 
protocols. For purposes of this discussion with respect to packets and the IP 
addresses, I'll use the format (source IP, destination IP).

When client contacts the VIP, a packet is sent with (C-IP, VIP). It arrives at 
A that holds the VIP at the moment, and proceeds to LOCAL_IN and LVS-NAT 
changes the packet to either (C-IP, A-IP) or (C-IP, B-IP). In the former case 
(C-IP, A-IP) case, it gets processed by the local syslog process. In the later 
case (C-IP, B-IP), it gets sent back out to B. However, because the packet is 
(C-IP, B-IP), B's replies go to (B-IP, C-IP) and doesn't get processed by the 
LVS-NAT anymore. Connection hangs... as the C is expecting reply from (VIP, 
C-IP).

I know in reading the various FAQ and postings about "one arm NAT" LVS-NAT 
situations is to have the "real servers" be the default GW. This won't work for 
me since the real servers are also the LVS servers. However, I was thinking 
that if I can use netfilter/iptables to source NAT the load balanced packets, I 
can change the (C-IP,B-IP) show up as (A-IP, B-IP) instead, and hence B's 
replies would go back to A (the current LVS-NAT) and sent back to C as (VIP, 
C-IP). However, the packet doesn't seem to traverse the "netfliter stack" as 
expected:

(i setup a -j LOG rule within every chain [PREROUTING, INPUT, FORWARD, OUTPUT, 
POSTROUTING] of every table [mangle, nat, filter], and watch the packet 
traverse the "netfilter stack")


Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-nat-PRER: IN=eth0 OUT= 
MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 
LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-INPU: IN=eth0 OUT= 
MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 
LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-INPU: IN=eth0 OUT= 
MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 
LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-OUTP: IN= OUT=eth1 
SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF 
PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-OUTP: IN= OUT=eth1 
SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF 
PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-POSR: IN= OUT=eth1 
SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF 
PROTO=UDP SPT=39178 DPT=514 LEN=16

This is a packet from (C-IP, VIP), goes through 
NAT/PREROUTING->ROUTING->MANGLE/INPUT->FILTER/INPUT->(LVS-NAT?)->ROUTING->MANGLE/OUTPUT->FILTER/OUTPUT->MANGLE/POSTROUTING.

What I see missing are MANGLE/PREROUTING and particularly interested in the 
missing NAT/POSTROUTING, where I was hoping to SNAT the packet.

According to this post:

http://archive.linuxvirtualserver.org/html/lvs-users/2011-03/msg00019.html

It would appear that at some point, LVS-NAT packets did traverse 
NAT/POSTROUTING and one could SNAT the LVS outbound packets?

I'm using:

RHEL 6.5 with RH kernel 2.6.32-431.17.1.el6.x86_64
ipvsadm-1.26-2.el6.x86_64
keepalived-1.2.13-1.el6.x86_64

here are the -j LOG iptables rules I'm using:

[root@sprdrsyas400 ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 293K packets, 29M bytes)
pkts bytes target     prot opt in     out     source               destination
    4   144 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-PRER: '

Chain INPUT (policy ACCEPT 293K packets, 29M bytes)
pkts bytes target     prot opt in     out     source               destination
    4   144 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-INPU: '

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-FORW: '

Chain OUTPUT (policy ACCEPT 362K packets, 18M bytes)
pkts bytes target     prot opt in     out     source               destination
    2    72 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-OUTP: '

Chain POSTROUTING (policy ACCEPT 362K packets, 18M bytes)
pkts bytes target     prot opt in     out     source               destination
    2    72 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-POSR: '
[root@sprdrsyas400 ~]# iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 8739 packets, 500K bytes)
pkts bytes target     prot opt in     out     source               destination
    7   252 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-PRER: '

Chain POSTROUTING (policy ACCEPT 25612 packets, 1374K bytes)
pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-POSR: '

Chain OUTPUT (policy ACCEPT 25612 packets, 1374K bytes)
pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-OUTP: '
[root@sprdrsyas400 ~]# iptables -vnL
Chain INPUT (policy ACCEPT 292K packets, 29M bytes)
pkts bytes target     prot opt in     out     source               destination
    4   144 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-INPU: '

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-FORW: '

Chain OUTPUT (policy ACCEPT 361K packets, 18M bytes)
pkts bytes target     prot opt in     out     source               destination
    2    72 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0   
        udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-OUTP: '


And this is LVS on A:

[root@sprdrsyas400 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
UDP  10.180.88.35:514 rr ops
  -> 10.180.88.29:514            Local   1      0          0
  -> 10.180.88.30:514            Masq    1      0          0

I would appreciate any help...

Thanks in advance,
-Bond

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>