|
I'm looking at moving my LVS config from virtual IP to fwmark setup. I
have two configuration files for ldirectord, one with virtual ip and
one with fwmark.
Virtual IP:
# Global Directives
checktimeout=6
checkinterval=2
autoreload=yes
#logfile="/var/log/ldirectord.log"
logfile="local0"
quiescent=no
virtual=192.168.100.1:80
real=192.168.11.11:80 masq
real=192.168.11.12:80 masq
real=192.168.11.13:80 masq
real=192.168.11.14:80 masq
fallback=127.0.0.1:80
service=http
request="/serverstate/"
receive="server up"
scheduler=lc
persistent=7200
#netmask=255.255.255.255
protocol=tcp
virtual=192.168.0.1:80
real=192.168.11.11:80 masq
real=192.168.11.12:80 masq
real=192.168.11.13:80 masq
real=192.168.11.14:80 masq
fallback=127.0.0.1:80
service=http
request="/serverstate/"
receive="server up"
scheduler=lc
persistent=7200
#netmask=255.255.255.255
protocol=tcp
virtual=192.168.100.1:443
real=192.168.11.11:443 masq
real=192.168.11.12:443 masq
real=192.168.11.13:443 masq
real=192.168.11.14:443 masq
fallback=127.0.0.1:443
service=https
request="/serverstate/"
receive="server up"
scheduler=lc
persistent=7200
#netmask=255.255.255.255
protocol=tcp
virtual=192.168.0.1:443
real=192.168.11.11:443 masq
real=192.168.11.12:443 masq
real=192.168.11.13:443 masq
real=192.168.11.14:443 masq
fallback=127.0.0.1:443
service=https
request="/serverstate/"
receive="server up"
scheduler=lc
persistent=7200
#netmask=255.255.255.255
protocol=tcp
Fwmark:
# Global Directives
checktimeout=6
checkinterval=2
autoreload=yes
#logfile="/var/log/ldirectord.log"
logfile="local0"
quiescent=no
virtual=1
real=192.168.11.11 masq
real=192.168.11.12 masq
real=192.168.11.13 masq
real=192.168.11.14 masq
fallback=127.0.0.1
checkport=80
service=http
request="/serverstate/"
receive="server up"
scheduler=lc
persistent=7200
#netmask=255.255.255.255
protocol=fwm
virtual=100
real=192.168.11.11 masq
real=192.168.11.12 masq
real=192.168.11.13 masq
real=192.168.11.14 masq
fallback=127.0.0.1
checkport=80
service=http
request="/serverstate/"
receive="server up"
scheduler=lc
persistent=7200
#netmask=255.255.255.255
protocol=fwm
virtual=64
real=192.168.11.11 masq
real=192.168.11.12 masq
real=192.168.11.13 masq
real=192.168.11.14 masq
fallback=127.0.0.1
checkport=80
service=http
request="/serverstate/"
receive="server up"
scheduler=lc
persistent=7200
#netmask=255.255.255.255
protocol=fwm
I have both 64 and 100 in there as I'm not sure whether virtual needs
to be in hex or decimal and the documentation doesn't make that clear.
I was going to see which one was getting all the connections then
remove the other one.
iptables mark table is as below:
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- 0.0.0.0/0 192.168.100.1 tcp
multiport dports 80,443 MARK set 0x64
MARK tcp -- 0.0.0.0/0 192.168.0.1 tcp
multiport dports 80,443 MARK set 0x1
The mark was setup with --set-mark 100, and it shows as a hex value.
Not really important but would be nice to see an update to ipvsadm(8)
to say whether it should be hex or decimal.
Replacing the virtual ip config with the fwmark config results in the
following in the /var/log/messages:
May 28 12:33:37 osacon2 ldirectord[30290]: Configuration file
'/etc/ha.d/conf/ldirectord.cf' has changed on disk
May 28 12:33:38 osacon2 ldirectord[30290]: - reread new configuration
May 28 12:33:38 osacon2 ldirectord[30290]: Added virtual server: 1
May 28 12:33:38 osacon2 ldirectord[30290]: Added fallback server:
127.0.0.1 (0 x 1) (Weight set to 1)
May 28 12:33:38 osacon2 ldirectord[30290]: Added virtual server: 100
May 28 12:33:38 osacon2 ldirectord[30290]: Added fallback server:
127.0.0.1 (0 x 100) (Weight set to 1)
May 28 12:33:38 osacon2 ldirectord[30290]: Added virtual server: 64
May 28 12:33:38 osacon2 ldirectord[30290]: Added fallback server:
127.0.0.1 (0 x 64) (Weight set to 1)
May 28 12:33:38 osacon2 ldirectord[30290]: Removed virtual server:
192.168.100.1:80
May 28 12:33:38 osacon2 ldirectord[30290]: Removed virtual server:
192.168.0.1:80
May 28 12:33:38 osacon2 ldirectord[30290]: Removed virtual server:
192.168.100.1:443
May 28 12:33:38 osacon2 ldirectord[30290]: Removed virtual server:
192.168.0.1:443
May 28 12:33:38 osacon2 ldirectord[30290]: Added real server:
192.168.11.11:0 (1 x 1) (Weight set to 1)
May 28 12:33:38 osacon2 ldirectord[30290]: Added real server:
192.168.11.11:0 (1 x 100) (Weight set to 1)
May 28 12:33:38 osacon2 ldirectord[30290]: Added real server:
192.168.11.11:0 (1 x 64) (Weight set to 1)
With the resulting ipvsadm -L -n:
IP Virtual Server version 1.0.11 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 lc persistent 7200
-> 192.168.11.11:0 Masq 1 0 0
-> 127.0.0.1:0 Local 1 0 0
FWM 64 lc persistent 7200
-> 192.168.11.11:0 Masq 1 0 0
-> 127.0.0.1:0 Local 1 0 0
FWM 100 lc persistent 7200
-> 192.168.11.11:0 Masq 1 0 0
-> 127.0.0.1:0 Local 1 0 4
Obviously not what we were looking for.... The fallback should be
removed from the pool as soon as a real server is available. If we swap
back the config files for the virtual ip config file I get the
following in /var/log/messages:
May 28 12:34:16 osacon2 ldirectord[30290]: Configuration file
'/etc/ha.d/conf/ldirectord.cf' has changed on disk
May 28 12:34:16 osacon2 ldirectord[30290]: - reread new configuration
May 28 12:34:16 osacon2 ldirectord[30290]: Added virtual server:
192.168.100.1:80
May 28 12:34:16 osacon2 ldirectord[30290]: Added fallback server:
127.0.0.1:80 (0 x 192.168.100.1:80) (Weight set to 1)
May 28 12:34:16 osacon2 ldirectord[30290]: Added virtual server:
192.168.0.1:80
May 28 12:34:16 osacon2 ldirectord[30290]: Added fallback server:
127.0.0.1:80 (0 x 192.168.0.1:80) (Weight set to 1)
May 28 12:34:16 osacon2 ldirectord[30290]: Added virtual server:
192.168.100.1:443
May 28 12:34:16 osacon2 ldirectord[30290]: Added fallback server:
127.0.0.1:443 (0 x 192.168.100.1:443) (Weight set to 1)
May 28 12:34:16 osacon2 ldirectord[30290]: Added virtual server:
192.168.0.1:443
May 28 12:34:16 osacon2 ldirectord[30290]: Added fallback server:
127.0.0.1:443 (0 x 192.168.0.1:443) (Weight set to 1)
May 28 12:34:16 osacon2 ldirectord[30290]: Removed virtual server: 1
May 28 12:34:16 osacon2 ldirectord[30290]: Removed virtual server: 100
May 28 12:34:16 osacon2 ldirectord[30290]: Removed virtual server: 64
May 28 12:34:16 osacon2 ldirectord[30290]: Added real server:
192.168.11.11:80 (1 x 192.168.100.1:80) (Weight set to 1)
May 28 12:34:16 osacon2 ldirectord[30290]: Deleted fallback server:
127.0.0.1:80 (1 x 192.168.100.1:80)
May 28 12:34:16 osacon2 ldirectord[30290]: Added real server:
192.168.11.11:80 (1 x 192.168.0.1:80) (Weight set to 1)
May 28 12:34:16 osacon2 ldirectord[30290]: Deleted fallback server:
127.0.0.1:80 (1 x 192.168.0.1:80)
May 28 12:34:22 osacon2 ldirectord[30290]: Added real server:
192.168.11.11:443 (1 x 192.168.100.1:443) (Weight set to 1)
May 28 12:34:22 osacon2 ldirectord[30290]: Deleted fallback server:
127.0.0.1:443 (1 x 192.168.100.1:443)
May 28 12:34:22 osacon2 ldirectord[30290]: Added real server:
192.168.11.11:443 (1 x 192.168.0.1:443) (Weight set to 1)
May 28 12:34:22 osacon2 ldirectord[30290]: Deleted fallback server:
127.0.0.1:443 (1 x 192.168.0.1:443)
Notice that the fallback servers have been removed from all 4 virtual
servers. ipvsadm -L -n is as below:
IP Virtual Server version 1.0.11 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.1:80 lc persistent 7200
-> 192.168.11.11:80 Masq 1 0 0
TCP 192.168.100.1:80 lc persistent 7200
-> 192.168.11.11:80 Masq 1 0 0
TCP 192.168.0.1:443 lc persistent 7200
-> 192.168.11.11:443 Masq 1 0 0
TCP 192.168.100.1:443 lc persistent 7200
-> 192.168.11.11:443 Masq 1 0 4
This is the same problem we saw intermittently during the testing phase
and what I was trying out the new builds to hopefully get fixed.
Something has to be different in the code between the fwmark handling
and the virtual ip handling.
Any ideas?
Thanks,
JT
| 