LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

ldirectord still leaving fallback server in pool v1.88 fwmark configurat

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: ldirectord still leaving fallback server in pool v1.88 fwmark configuration
From: Jonathan Trott <jtrott@xxxxxxxxxxx>
Date: Fri, 28 May 2004 12:43:52 +1000
I'm looking at moving my LVS config from virtual IP to fwmark setup. I have two configuration files for ldirectord, one with virtual ip and one with fwmark.

Virtual IP:

# Global Directives
checktimeout=6
checkinterval=2
autoreload=yes
#logfile="/var/log/ldirectord.log"
logfile="local0"
quiescent=no

virtual=192.168.100.1:80
        real=192.168.11.11:80 masq
        real=192.168.11.12:80 masq
        real=192.168.11.13:80 masq
        real=192.168.11.14:80 masq
        fallback=127.0.0.1:80
        service=http
        request="/serverstate/"
        receive="server up"
        scheduler=lc
        persistent=7200
        #netmask=255.255.255.255
        protocol=tcp

virtual=192.168.0.1:80
        real=192.168.11.11:80 masq
        real=192.168.11.12:80 masq
        real=192.168.11.13:80 masq
        real=192.168.11.14:80 masq
        fallback=127.0.0.1:80
        service=http
        request="/serverstate/"
        receive="server up"
        scheduler=lc
        persistent=7200
        #netmask=255.255.255.255
        protocol=tcp

virtual=192.168.100.1:443
        real=192.168.11.11:443 masq
        real=192.168.11.12:443 masq
        real=192.168.11.13:443 masq
        real=192.168.11.14:443 masq
        fallback=127.0.0.1:443
        service=https
        request="/serverstate/"
        receive="server up"
        scheduler=lc
        persistent=7200
        #netmask=255.255.255.255
        protocol=tcp

virtual=192.168.0.1:443
        real=192.168.11.11:443 masq
        real=192.168.11.12:443 masq
        real=192.168.11.13:443 masq
        real=192.168.11.14:443 masq
        fallback=127.0.0.1:443
        service=https
        request="/serverstate/"
        receive="server up"
        scheduler=lc
        persistent=7200
        #netmask=255.255.255.255
        protocol=tcp

Fwmark:

# Global Directives
checktimeout=6
checkinterval=2
autoreload=yes
#logfile="/var/log/ldirectord.log"
logfile="local0"
quiescent=no


virtual=1
        real=192.168.11.11 masq
        real=192.168.11.12 masq
        real=192.168.11.13 masq
        real=192.168.11.14 masq
        fallback=127.0.0.1
        checkport=80
        service=http
        request="/serverstate/"
        receive="server up"
        scheduler=lc
        persistent=7200
        #netmask=255.255.255.255
        protocol=fwm


virtual=100
        real=192.168.11.11 masq
        real=192.168.11.12 masq
        real=192.168.11.13 masq
        real=192.168.11.14 masq
        fallback=127.0.0.1
        checkport=80
        service=http
        request="/serverstate/"
        receive="server up"
        scheduler=lc
        persistent=7200
        #netmask=255.255.255.255
        protocol=fwm

virtual=64
        real=192.168.11.11 masq
        real=192.168.11.12 masq
        real=192.168.11.13 masq
        real=192.168.11.14 masq
        fallback=127.0.0.1
        checkport=80
        service=http
        request="/serverstate/"
        receive="server up"
        scheduler=lc
        persistent=7200
        #netmask=255.255.255.255
        protocol=fwm

I have both 64 and 100 in there as I'm not sure whether virtual needs to be in hex or decimal and the documentation doesn't make that clear. I was going to see which one was getting all the connections then remove the other one.
iptables mark table is as below:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
MARK tcp -- 0.0.0.0/0 192.168.100.1 tcp multiport dports 80,443 MARK set 0x64 MARK tcp -- 0.0.0.0/0 192.168.0.1 tcp multiport dports 80,443 MARK set 0x1

The mark was setup with --set-mark 100, and it shows as a hex value. Not really important but would be nice to see an update to ipvsadm(8) to say whether it should be hex or decimal. Replacing the virtual ip config with the fwmark config results in the following in the /var/log/messages:

May 28 12:33:37 osacon2 ldirectord[30290]: Configuration file '/etc/ha.d/conf/ldirectord.cf' has changed on disk
May 28 12:33:38 osacon2 ldirectord[30290]:  - reread new configuration
May 28 12:33:38 osacon2 ldirectord[30290]: Added virtual server: 1
May 28 12:33:38 osacon2 ldirectord[30290]: Added fallback server: 127.0.0.1 (0 x 1) (Weight set to 1)
May 28 12:33:38 osacon2 ldirectord[30290]: Added virtual server: 100
May 28 12:33:38 osacon2 ldirectord[30290]: Added fallback server: 127.0.0.1 (0 x 100) (Weight set to 1)
May 28 12:33:38 osacon2 ldirectord[30290]: Added virtual server: 64
May 28 12:33:38 osacon2 ldirectord[30290]: Added fallback server: 127.0.0.1 (0 x 64) (Weight set to 1) May 28 12:33:38 osacon2 ldirectord[30290]: Removed virtual server: 192.168.100.1:80 May 28 12:33:38 osacon2 ldirectord[30290]: Removed virtual server: 192.168.0.1:80 May 28 12:33:38 osacon2 ldirectord[30290]: Removed virtual server: 192.168.100.1:443 May 28 12:33:38 osacon2 ldirectord[30290]: Removed virtual server: 192.168.0.1:443 May 28 12:33:38 osacon2 ldirectord[30290]: Added real server: 192.168.11.11:0 (1 x 1) (Weight set to 1) May 28 12:33:38 osacon2 ldirectord[30290]: Added real server: 192.168.11.11:0 (1 x 100) (Weight set to 1) May 28 12:33:38 osacon2 ldirectord[30290]: Added real server: 192.168.11.11:0 (1 x 64) (Weight set to 1)

With the resulting ipvsadm -L -n:

IP Virtual Server version 1.0.11 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  1 lc persistent 7200
  -> 192.168.11.11:0              Masq    1      0          0
  -> 127.0.0.1:0                  Local   1      0          0
FWM  64 lc persistent 7200
  -> 192.168.11.11:0              Masq    1      0          0
  -> 127.0.0.1:0                  Local   1      0          0
FWM  100 lc persistent 7200
  -> 192.168.11.11:0              Masq    1      0          0
  -> 127.0.0.1:0                  Local   1      0          4

Obviously not what we were looking for.... The fallback should be removed from the pool as soon as a real server is available. If we swap back the config files for the virtual ip config file I get the following in /var/log/messages:

May 28 12:34:16 osacon2 ldirectord[30290]: Configuration file '/etc/ha.d/conf/ldirectord.cf' has changed on disk
May 28 12:34:16 osacon2 ldirectord[30290]:  - reread new configuration
May 28 12:34:16 osacon2 ldirectord[30290]: Added virtual server: 192.168.100.1:80 May 28 12:34:16 osacon2 ldirectord[30290]: Added fallback server: 127.0.0.1:80 (0 x 192.168.100.1:80) (Weight set to 1) May 28 12:34:16 osacon2 ldirectord[30290]: Added virtual server: 192.168.0.1:80 May 28 12:34:16 osacon2 ldirectord[30290]: Added fallback server: 127.0.0.1:80 (0 x 192.168.0.1:80) (Weight set to 1) May 28 12:34:16 osacon2 ldirectord[30290]: Added virtual server: 192.168.100.1:443 May 28 12:34:16 osacon2 ldirectord[30290]: Added fallback server: 127.0.0.1:443 (0 x 192.168.100.1:443) (Weight set to 1) May 28 12:34:16 osacon2 ldirectord[30290]: Added virtual server: 192.168.0.1:443 May 28 12:34:16 osacon2 ldirectord[30290]: Added fallback server: 127.0.0.1:443 (0 x 192.168.0.1:443) (Weight set to 1)
May 28 12:34:16 osacon2 ldirectord[30290]: Removed virtual server: 1
May 28 12:34:16 osacon2 ldirectord[30290]: Removed virtual server: 100
May 28 12:34:16 osacon2 ldirectord[30290]: Removed virtual server: 64
May 28 12:34:16 osacon2 ldirectord[30290]: Added real server: 192.168.11.11:80 (1 x 192.168.100.1:80) (Weight set to 1) May 28 12:34:16 osacon2 ldirectord[30290]: Deleted fallback server: 127.0.0.1:80 (1 x 192.168.100.1:80) May 28 12:34:16 osacon2 ldirectord[30290]: Added real server: 192.168.11.11:80 (1 x 192.168.0.1:80) (Weight set to 1) May 28 12:34:16 osacon2 ldirectord[30290]: Deleted fallback server: 127.0.0.1:80 (1 x 192.168.0.1:80) May 28 12:34:22 osacon2 ldirectord[30290]: Added real server: 192.168.11.11:443 (1 x 192.168.100.1:443) (Weight set to 1) May 28 12:34:22 osacon2 ldirectord[30290]: Deleted fallback server: 127.0.0.1:443 (1 x 192.168.100.1:443) May 28 12:34:22 osacon2 ldirectord[30290]: Added real server: 192.168.11.11:443 (1 x 192.168.0.1:443) (Weight set to 1) May 28 12:34:22 osacon2 ldirectord[30290]: Deleted fallback server: 127.0.0.1:443 (1 x 192.168.0.1:443)

Notice that the fallback servers have been removed from all 4 virtual servers. ipvsadm -L -n is as below:

IP Virtual Server version 1.0.11 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.1:80 lc persistent 7200
  -> 192.168.11.11:80             Masq    1      0          0
TCP  192.168.100.1:80 lc persistent 7200
  -> 192.168.11.11:80             Masq    1      0          0
TCP  192.168.0.1:443 lc persistent 7200
  -> 192.168.11.11:443            Masq    1      0          0
TCP  192.168.100.1:443 lc persistent 7200
  -> 192.168.11.11:443            Masq    1      0          4

This is the same problem we saw intermittently during the testing phase and what I was trying out the new builds to hopefully get fixed. Something has to be different in the code between the fwmark handling and the virtual ip handling.
Any ideas?
Thanks,
JT

<Prev in Thread] Current Thread [Next in Thread>
  • ldirectord still leaving fallback server in pool v1.88 fwmark configuration, Jonathan Trott <=