|
Today while testing my LVS-DR I have been running into the situation
where keepalived gets itself into some kind of sync issue. Basically
both directors end up as master and then nothing works after that. Here
is a snip from logs:
DIRECTOR 1:
Jul 23 21:52:50 grp-01-00-50 Keepalived: Starting Keepalived v1.1.13
(03/26,2007)
Jul 23 21:52:50 grp-01-00-50 Keepalived: Starting Healthcheck child
process, pid=29059
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Using MII-BMSR NIC polling
thread...
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Netlink reflector reports
IP 192.168.1.150 added
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Registering Kernel netlink
reflector
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Registering Kernel netlink
command channel
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Registering gratutious ARP
shared channel
Jul 23 21:52:50 grp-01-00-50 Keepalived: Starting VRRP child process,
pid=29060
Jul 23 21:52:50 grp-01-00-50 kernel: IPVS: sync thread stopped!
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Configuration is using :
34940 Bytes
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Using MII-BMSR
NIC polling thread...
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Netlink
reflector reports IP 192.168.1.150 added
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Registering
Kernel netlink reflector
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Registering
Kernel netlink command channel
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Configuration is
using : 9125 Bytes
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Activating
healtchecker for service [192.168.1.200:80]
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Activating
healtchecker for service [192.168.1.201:80]
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: VRRP sockpool:
[ifindex(2), proto(112), fd(8,9)]
Jul 23 21:52:50 grp-01-00-50 kernel: IPVS: sync thread started: state =
MASTER, mcast_ifn = eth0, syncid = 25
Jul 23 21:52:52 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Transition to MASTER STATE
Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Entering MASTER STATE
Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
setting protocol VIPs.
Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: Netlink: skipping nl_cmd
msg...
Jul 23 21:52:54 grp-01-00-50 Keepalived_healthcheckers: Netlink
reflector reports IP 192.168.1.240 added
Jul 23 21:52:54 grp-01-00-50 : /etc/keepalived/ip_localhost (called
from: keepalived): rsh 192.168.1.200 /sbin/ip addr add 192.168.1.240/32
dev lo brd + scope host
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called
from: keepalived): RTNETLINK answers: File exists
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called
from: keepalived): rsh 192.168.1.201 /sbin/ip addr add 192.168.1.240/32
dev lo brd + scope host
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called
from: keepalived): RTNETLINK answers: File exists
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called
from: keepalived): ip addr del 192.168.1.240/32 dev lo
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called
from: keepalived): RTNETLINK answers: Cannot assign requested address
Jul 23 21:52:59 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:39 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Received lower prio advert, forcing new election
Jul 23 21:53:39 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:41 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Received lower prio advert, forcing new election
Jul 23 21:53:41 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:43 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Received lower prio advert, forcing new election
Jul 23 21:53:43 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:45 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Received lower prio advert, forcing new election
Jul 23 21:53:45 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:47 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Received lower prio advert, forcing new election
Jul 23 21:53:47 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:49 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1)
Received lower prio advert, forcing new election
... forever
DIRECTOR 2:
Jul 23 21:53:32 grp-01-00-51 Keepalived: Starting Keepalived v1.1.13
(03/26,2007)
Jul 23 21:53:32 grp-01-00-51 Keepalived: Starting Healthcheck child
process, pid=27215
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Using MII-BMSR NIC polling
thread...
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Netlink reflector reports
IP 192.168.1.151 added
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Registering Kernel netlink
reflector
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Registering Kernel netlink
command channel
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Registering gratutious ARP
shared channel
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Configuration is using :
34940 Bytes
Jul 23 21:53:33 grp-01-00-51 Keepalived: Starting VRRP child process,
pid=27217
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Using MII-BMSR
NIC polling thread...
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Netlink
reflector reports IP 192.168.1.151 added
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Registering
Kernel netlink reflector
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Registering
Kernel netlink command channel
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Configuration is
using : 9125 Bytes
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Activating
healtchecker for service [192.168.1.200:80]
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Activating
healtchecker for service [192.168.1.201:80]
Jul 23 21:53:33 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1)
Entering BACKUP STATE
Jul 23 21:53:33 grp-01-00-51 Keepalived_vrrp: VRRP sockpool:
[ifindex(2), proto(112), fd(8,9)]
Jul 23 21:53:33 grp-01-00-51 : /etc/keepalived/ip_localhost (called
from: keepalived): ip addr add 192.168.1.240/32 dev lo brd + scope host
Jul 23 21:53:33 grp-01-00-51 kernel: IPVS: sync thread started: state =
BACKUP, mcast_ifn = eth0, syncid = 25
Jul 23 21:53:39 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1)
Transition to MASTER STATE
Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1)
Entering MASTER STATE
Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1)
setting protocol VIPs.
Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1)
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:41 grp-01-00-51 Keepalived_healthcheckers: Netlink
reflector reports IP 192.168.1.240 added
Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: Netlink: skipping nl_cmd
msg...
Jul 23 21:53:41 grp-01-00-51 avahi-daemon[2086]: Registering new address
record for 192.168.1.240 on eth0.IPv4.
Jul 23 21:53:41 grp-01-00-51 kernel: IPVS: stopping sync thread 27223 ...
Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called
from: keepalived): rsh 192.168.1.200 /sbin/ip addr add 192.168.1.240/32
dev lo brd + scope host
Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called
from: keepalived): poll: protocol failure in circuit setup
Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called
from: keepalived): rsh 192.168.1.201 /sbin/ip addr add 192.168.1.240/32
dev lo brd + scope host
Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called
from: keepalived): poll: protocol failure in circuit setup
Jul 23 21:53:42 grp-01-00-51 : /etc/keepalived/ip_localhost (called
from: keepalived): ip addr del 192.168.1.240/32 dev lo
Jul 23 21:53:42 grp-01-00-51 kernel: IPVS: sync thread stopped!
Jul 23 21:53:42 grp-01-00-51 kernel: IPVS: sync thread started: state =
MASTER, mcast_ifn = eth0, syncid = 25
Jul 23 21:53:46 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1)
Sending gratuitous ARPs on eth0 for 192.168.1.240
[root@grp-01-00-51 keepalived]#
Anyway I found that this was a firewall issue. Once that was fixed this
sync problem went away. Just thought I would post it in case someone
else runs into this.
Gerry
|
