LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

[lvs-users] Keepalived sync war

To: "lvs >> \"LinuxVirtualServer.org users mailing list.\"" <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: [lvs-users] Keepalived sync war
From: Gerry Reno <greno@xxxxxxxxxxx>
Date: Mon, 23 Jul 2007 22:51:40 -0400
Today while testing my LVS-DR I have been running into the situation 
where keepalived gets itself into some kind of sync issue.  Basically 
both directors end up as master and then nothing works after that.  Here 
is a snip from logs:

DIRECTOR 1:
Jul 23 21:52:50 grp-01-00-50 Keepalived: Starting Keepalived v1.1.13 
(03/26,2007)
Jul 23 21:52:50 grp-01-00-50 Keepalived: Starting Healthcheck child 
process, pid=29059
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Using MII-BMSR NIC polling 
thread...
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Netlink reflector reports 
IP 192.168.1.150 added
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Registering Kernel netlink 
reflector
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Registering Kernel netlink 
command channel
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Registering gratutious ARP 
shared channel
Jul 23 21:52:50 grp-01-00-50 Keepalived: Starting VRRP child process, 
pid=29060
Jul 23 21:52:50 grp-01-00-50 kernel: IPVS: sync thread stopped!
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: Configuration is using : 
34940 Bytes
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Using MII-BMSR 
NIC polling thread...
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Netlink 
reflector reports IP 192.168.1.150 added
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Registering 
Kernel netlink reflector
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Registering 
Kernel netlink command channel
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Configuration is 
using : 9125 Bytes
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Activating 
healtchecker for service [192.168.1.200:80]
Jul 23 21:52:50 grp-01-00-50 Keepalived_healthcheckers: Activating 
healtchecker for service [192.168.1.201:80]
Jul 23 21:52:50 grp-01-00-50 Keepalived_vrrp: VRRP sockpool: 
[ifindex(2), proto(112), fd(8,9)]
Jul 23 21:52:50 grp-01-00-50 kernel: IPVS: sync thread started: state = 
MASTER, mcast_ifn = eth0, syncid = 25
Jul 23 21:52:52 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Transition to MASTER STATE
Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Entering MASTER STATE
Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
setting protocol VIPs.
Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:52:54 grp-01-00-50 Keepalived_vrrp: Netlink: skipping nl_cmd 
msg...
Jul 23 21:52:54 grp-01-00-50 Keepalived_healthcheckers: Netlink 
reflector reports IP 192.168.1.240 added
Jul 23 21:52:54 grp-01-00-50 : /etc/keepalived/ip_localhost (called 
from: keepalived): rsh 192.168.1.200 /sbin/ip addr add 192.168.1.240/32 
dev lo brd + scope host
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called 
from: keepalived): RTNETLINK answers: File exists
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called 
from: keepalived): rsh 192.168.1.201 /sbin/ip addr add 192.168.1.240/32 
dev lo brd + scope host
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called 
from: keepalived): RTNETLINK answers: File exists
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called 
from: keepalived): ip addr del 192.168.1.240/32 dev lo
Jul 23 21:52:55 grp-01-00-50 : /etc/keepalived/ip_localhost (called 
from: keepalived): RTNETLINK answers: Cannot assign requested address
Jul 23 21:52:59 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:39 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Received lower prio advert, forcing new election
Jul 23 21:53:39 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:41 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Received lower prio advert, forcing new election
Jul 23 21:53:41 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:43 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Received lower prio advert, forcing new election
Jul 23 21:53:43 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:45 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Received lower prio advert, forcing new election
Jul 23 21:53:45 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:47 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Received lower prio advert, forcing new election
Jul 23 21:53:47 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:49 grp-01-00-50 Keepalived_vrrp: VRRP_Instance(VI_1) 
Received lower prio advert, forcing new election
... forever

DIRECTOR 2:
Jul 23 21:53:32 grp-01-00-51 Keepalived: Starting Keepalived v1.1.13 
(03/26,2007)
Jul 23 21:53:32 grp-01-00-51 Keepalived: Starting Healthcheck child 
process, pid=27215
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Using MII-BMSR NIC polling 
thread...
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Netlink reflector reports 
IP 192.168.1.151 added
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Registering Kernel netlink 
reflector
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Registering Kernel netlink 
command channel
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Registering gratutious ARP 
shared channel
Jul 23 21:53:32 grp-01-00-51 Keepalived_vrrp: Configuration is using : 
34940 Bytes
Jul 23 21:53:33 grp-01-00-51 Keepalived: Starting VRRP child process, 
pid=27217
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Using MII-BMSR 
NIC polling thread...
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Netlink 
reflector reports IP 192.168.1.151 added
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Registering 
Kernel netlink reflector
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Registering 
Kernel netlink command channel
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Configuration is 
using : 9125 Bytes
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Activating 
healtchecker for service [192.168.1.200:80]
Jul 23 21:53:33 grp-01-00-51 Keepalived_healthcheckers: Activating 
healtchecker for service [192.168.1.201:80]
Jul 23 21:53:33 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) 
Entering BACKUP STATE
Jul 23 21:53:33 grp-01-00-51 Keepalived_vrrp: VRRP sockpool: 
[ifindex(2), proto(112), fd(8,9)]
Jul 23 21:53:33 grp-01-00-51 : /etc/keepalived/ip_localhost (called 
from: keepalived): ip addr add 192.168.1.240/32 dev lo brd + scope host
Jul 23 21:53:33 grp-01-00-51 kernel: IPVS: sync thread started: state = 
BACKUP, mcast_ifn = eth0, syncid = 25
Jul 23 21:53:39 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) 
Transition to MASTER STATE
Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) 
Entering MASTER STATE
Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) 
setting protocol VIPs.
Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) 
Sending gratuitous ARPs on eth0 for 192.168.1.240
Jul 23 21:53:41 grp-01-00-51 Keepalived_healthcheckers: Netlink 
reflector reports IP 192.168.1.240 added
Jul 23 21:53:41 grp-01-00-51 Keepalived_vrrp: Netlink: skipping nl_cmd 
msg...
Jul 23 21:53:41 grp-01-00-51 avahi-daemon[2086]: Registering new address 
record for 192.168.1.240 on eth0.IPv4.
Jul 23 21:53:41 grp-01-00-51 kernel: IPVS: stopping sync thread 27223 ...
Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called 
from: keepalived): rsh 192.168.1.200 /sbin/ip addr add 192.168.1.240/32 
dev lo brd + scope host
Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called 
from: keepalived): poll: protocol failure in circuit setup
Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called 
from: keepalived): rsh 192.168.1.201 /sbin/ip addr add 192.168.1.240/32 
dev lo brd + scope host
Jul 23 21:53:41 grp-01-00-51 : /etc/keepalived/ip_localhost (called 
from: keepalived): poll: protocol failure in circuit setup
Jul 23 21:53:42 grp-01-00-51 : /etc/keepalived/ip_localhost (called 
from: keepalived): ip addr del 192.168.1.240/32 dev lo
Jul 23 21:53:42 grp-01-00-51 kernel: IPVS: sync thread stopped!
Jul 23 21:53:42 grp-01-00-51 kernel: IPVS: sync thread started: state = 
MASTER, mcast_ifn = eth0, syncid = 25
Jul 23 21:53:46 grp-01-00-51 Keepalived_vrrp: VRRP_Instance(VI_1) 
Sending gratuitous ARPs on eth0 for 192.168.1.240
[root@grp-01-00-51 keepalived]#


Anyway I found that this was a firewall issue.  Once that was fixed this 
sync problem went away.  Just thought I would post it in case someone 
else runs into this.

Gerry



<Prev in Thread] Current Thread [Next in Thread>
  • [lvs-users] Keepalived sync war, Gerry Reno <=