[lvs-users] packets not seen or dropped by ipvs on lo interface in client-director LVS-DR setup

[campbell mcleay <campbell.mcleay@xxxxxxxxxxxxxx>]

Hi,

In a client-director (LVS-DR) setup, packets being sent to the VIP from the
same box do not get load balanced by IPVS. I am basing this setup from
the LVS-HOWTO. The patch to ip_vs_core.c as mentioned in the HOWTO should be
present in the kernel version I am using (RHEL 2.6.18 standard kernel).

I can see the connection on loopback, with the source and destination as the
VIP address:

17:59:38.053548 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4
(0x0800), length 66: 192.168.10.13.56148 > 192.168.10.13.80: S
2993458444:2993458444(0) win 32792 <mss 16396,nop,nop,sackOK,nop,wscale 7>
E..4..@.@.....

 ...and the packets go no further. I thus get 'connection refused'.

I can connect to port 80 ok on the realserver real IPs from the director.

I can see the iptables fwmark rule (-A OUTPUT -d 172.16.1.10 -p tcp -m tcp
--dport 80 -j MARK --set-mark 0x2) incrementing as it marks the packet.

I am using kernel 2.6.18 (RHEL) and ipvsadm v1.24 2005/12/10 (compiled with
popt and IPVS v1.2.1).

The config is as follows:

*Director/client box:*

eth0         192.168.10.140 (real IP)
eth0:1      192.168.10.13 (VIP)

iptables rules:

*mangle
-A OUTPUT -d 192.168.10.13 -p tcp -m tcp --dport 80 -j MARK --set-mark 0x2
COMMIT
*nat
-A PREROUTING -d 192.168.10.140 -p tcp -m tcp --dport 80 -j REDIRECT
--to-ports 3128

ip rule show:

0:    from all lookup 255
100:    from all fwmark 0x2 lookup VIP
32766:    from all lookup main
32767:    from all lookup default

ip route list table 100
local default dev lo  scope host


I'm using keepalived to configure. Config:

-------------8<----------------

! Configuration File for keepalived

global_defs {
   router_id LVS_ANON
}

virtual_server 192.168.10.13 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    nat_mask 255.255.255.0
    !persistence_timeout 50
    protocol TCP

    real_server 192.168.10.41 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
    }
}

    real_server 192.168.10.208 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 300000
            delay_before_retry 300000
        }
    }
}

-----------------8<--------


*Realservers:*

realserver1:

eth0          192.168.10.41
lo:0            192.168.10.13

realserver2:

eth0          192.168.10.208
lo:0            192.168.10.41

On both realservers, I have set arp to '1' in ignore for
/proc/sys/net/ipv4/conf/all/arp_ignore and
/proc/sys/net/ipv4/conf/eth0/arp_ignore
and '2' for /proc/sys/net/ipv4/conf/all/arp_announce
/proc/sys/net/ipv4/conf/eth0/arp_announce

I have tried with and without rp_filter set to '0' on the director. I assume
that it would not need to be enabled, though, since if packets did leave and
return on eth0:1, they
would be using the right interface for their src and dst. Though I am not
sure about this.


Clearly I am doing something wrong here if others have got this working, so
any further suggestions would be very welcome!
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users