|
On Wed, August 30, 2006, Todd Lyons wrote:
>
> Heartbeat typically brings up the virtual IP when it starts a
> service that uses it, and brings down the virtual IP after it
> stops a service.
> It sounds like the two heartbeats are having one or more of
> the following potential problems:
> 1) can't talk to each other (split brain) so both think the
> other is dead, and both bring up IP's.
Yeah I've looked at this extensively in the logs and via tcpdump on the
heartbeat interface, as far as I can tell everything looks ok.
> 2) your interfaces are configured to start at boot time
> instead of being controlled by heartbeat
Well, my ethernet interfaces both come up at boot with the server IP, all
other virtual IPs end up as an lo:xxx interface until heartbeat decides
which server is going to become the active director and enables ldirectord.
Unless I've misread something (possible), this is the way it should be
correct?
> 3) some previous instance of heartbeat didn't shut down
> virtual interfaces properly (but it sounds like you've
> rebooted since then).
Yeah I had quite a few problems when I first set the systems up with
heartbeat not communicating properly, so looking for extra instances of
heartbeat/ldirectord is usually the first thing I look at.
> What is the mac address for the ip's in question at the
> router/firewall?
>
> Which physical machine does that correspond to?
I was just thinking this myself as I was going through various config files
unrelated to the LVS. At one point I rebuilt one of the machines due to a
corrupted RAID from a crash. Since I first replicated the machines by
swapping out the hard drives and using the swapped out drive to create the
second server (and several more soon), I did the same thing after the crash
and changed appropriate things as needed. By the looks of it the MAC
address of the interface was being defined the same for both servers.
Hopefully this solves my problem although I probably wont be able to test it
for another 10 hours or so however.
> Post your heartbeat config and your ldirector config for both
> machines (ideally, they are identical between the two
> machines). Also, make sure that each machine can resolve and
> speak to each other. Also, which version of heartbeat and
> which version of ldirector would be helpful.
ldirectord,v 1.77.2.32 2005/09/21 04:00:41 horms
heartbeat.in,v 1.33.2.1 2004/04/20 05:05:28 alan
------------------Heartbeat (ha.cf):
# ha.cf - seahawk: comments removed
debugfile /var/log/ha-debug
logfile /var/log/ha-log
keepalive 5
deadtime 20
warntime 10
initdead 20
udpport 694
ucast eth2 10.0.0.1
auto_failback off
node nitehawk.thezoo
node seahawk.thezoo
respawn hacluster /usr/lib64/heartbeat/ipfail
deadping 30
debug 1
apiauth ipfail uid=hacluster gid=haclient
# ha.cf - nitehawk - only difference
ucast eth2 10.0.0.2
------------------ /etc/ha.d/haresources
Yeah I'm gonna get lots of "WTF!?" with this one, but it works when it
works.
It just complicates debugging things greatly. I think I have only ~60 IPs
enabled at the moment. My arptables_jf file is worse.
# no public name for LVS, use name of one of the servers.
seahawk.thezoo \
ldirectord \
LVSSyncDaemonSwap::master \
IPaddr2::192.168.0.3/24/eth1/192.168.0.255 \
IPaddr2::216.94.150.11/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.15/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.16/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.17/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.18/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.20/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.22/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.23/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.24/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.26/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.28/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.30/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.32/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.33/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.34/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.35/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.36/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.37/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.38/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.42/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.45/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.47/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.48/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.49/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.50/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.52/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.53/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.57/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.58/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.59/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.61/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.64/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.65/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.66/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.67/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.69/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.70/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.71/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.73/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.75/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.80/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.81/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.84/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.85/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.86/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.89/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.90/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.91/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.92/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.93/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.95/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.96/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.97/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.98/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.100/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.101/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.104/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.113/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.115/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.116/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.117/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.118/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.119/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.121/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.123/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.124/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.125/24/eth0/216.94.150.127 \
IPaddr2::216.94.150.126/24/eth0/216.94.150.127
------------------ldirectord.cf (no difference between servers)
# Global Settings
checktimeout=10
checkinterval=5
autoreload=no
logfile="/var/log/ldirectord.log"
quiescent=no
# Per Site Settings
virtual=192.168.0.3:80
real=192.168.0.4:80 gate
real=192.168.0.5:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.11:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.15:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.16:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.17:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.18:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.20:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.22:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
persistent=600
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.23:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.24:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.26:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
persistent=600
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.28:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
persistent=600
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.30:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.32:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.33:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.34:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.35:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
persistent=600
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.36:21
real=216.94.150.8:21 gate
real=216.94.150.7:21 gate
service=ftp
request=".lvs.msg"
receive="LVS OK"
login="zu_client"
passwd="zu!client^uploads"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.37:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.38:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.42:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
persistent=1200
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.45:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.47:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.48:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
persistent=600
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.49:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.50:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.52:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.53:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
persistent=600
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.57:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.59:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.61:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.64:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.65:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.66:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
persistent=600
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.67:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
persistent=600
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.69:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.70:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.71:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.73:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.75:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.80:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.81:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.84:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.85:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.86:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.89:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.90:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.91:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.92:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.93:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.95:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.96:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.97:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.98:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.100:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.101:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.104:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
persistent=3600
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.113:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.115:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
persistent=600
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.116:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.117:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
persistent=600
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.118:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.119:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.121:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
persistent=600
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.123:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.124:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.125:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
virtual=216.94.150.126:80
real=216.94.150.8:80 gate
real=216.94.150.7:80 gate
service=http
request=".lvs.html"
receive="Test Message"
scheduler=rr
protocol=tcp
checktype=negotiate
|
