|
Hi all,
Here is a "recipe" I cooked up (!) on creating LVS clusters with
machines that support
redundant networking.
Note: This is a work in progess so any feedback is welcome. Typos and/or
errors are
certainly present. I'm no gated or LVS expert, so maybe there's a better
way of doing
this, but I spent a lot of time working on it so I figured it could be
of use to other
folks as well.
I have performed many tests with this setup and it performed properly.
If you find
otherwise please let it be known.
SCENARIO
========
Our production environment is fully redundant at the network level (each
machine has two
network interfaces, each connected to a different network). All machine
are connected to
both these networks and data can come from either network. On each
machine, service run
on a local network address and gated announces the route for these
networks via both
network interface.
My task was to create an LVS cluster of 2 such machines (each a
potential director and
realserver as well).
NETWORK SETUP
=============
Network 1 is 192.168.100.0/24
Network 2 is 192.168.101.0/24
Machine 1:
- eth0: 192.168.100.1
- eth1: 192.168.101.1
- local network on loopback (lo:real): 192.168.200.1/32
Machine 2:
- eth0: 192.168.100.2
- eth1: 192.168.101.2
- local network on loopback (lo:real): 192.168.201.1/32
Virtual IP is 192.168.300.1
GATED SETUP
===========
Have gated annonce (and accept) the following routes:
Machine 1:
- announce 192.168.200.1/32
- accept routes from 192.168.100.2 and 192.168.101.2
Machine 2:
- announce 192.168.201.1/32
- accept routes from 192.168.100.1 and 192.168.101.1
These routes will be used by ldirectord to monitor the realservers.
RECIPE
======
1) Install UltraMonkey as usual, but:
- Make sure to configure ping nodes in both networks.
- Create the virtual IP alias as 192.168.300.1
- A virtual service definition in ldirectord.cf should look something
like this:
virtual=192.168.300.1:80
real=192.168.200.1:80 gate
real=192.168.201.1:80 gate
service=http
checkport=80
request="/test.html"
receive="test"
scheduler=rr
protocol=tcp
In a normal setup, heartbeat manages the virtual IP alias and brings it up
on the active director. If I understand correctly, an arp request is then
sent, making the other machines in the local network aware that the active
director is now the machine to be reached for the virtual IP.
In this setup we will tell heartbeat to leave the virtual IP alias alone
and have it tell gated to announce the route for the 192.168.300.1/32
network
instead. Therefore ONLY the active director will anounce the routes to
reach
the virtual IP network.
2) Change your haresources line to something like this:
node1.cluster.tld gated-toggle ldirectord
3) Place the following (or equivalent) code in a file called
/etc/ha.d/resource.d/gated-toggle:
--------8<--------
#!/bin/bash
#
# This gated control script should only be called by heartbeat!
#
# start: RESTART gated with the original (non-director config)
# stop: RESTART gated with the director config
#
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0
gdc=/usr/sbin/gdc
gated=/usr/sbin/gated
prog=gated
if [ ! -f /etc/gated.conf -o ! -f $gdc ] ; then
action $"Not starting $prog: " true
exit 0
fi
PATH=$PATH:/usr/bin:/usr/sbin
RETVAL=0
start() {
echo -n $"Starting $prog: "
CFG=$1
if [ "$CFG" != "" ] ; then
RES='$2$3'
RE="s/^(\s*\#+)(.*)(\#\s*heartbeat-toggle\s*)$/$RES/"
/usr/bin/perl -p -e "$RE" /etc/gated.conf > $CFG
daemon $gated -f $CFG
else
daemon $gated
fi
RETVAL=$?
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/gated
echo
return $RETVAL
}
stop() {
# Stop daemons.
action $"Stopping $prog" $gdc stop
RETVAL=$?
if [ $RETVAL -eq 0 ] ; then
rm -f /var/lock/subsys/gated
fi
return $RETVAL
}
# See how we were called.
case "$1" in
start)
stop
start "/etc/gated-heartbeat.conf"
;;
stop)
stop
start
;;
*)
echo $"Usage: $0 {start|stop}"
exit 1
esac
exit $RETVAL
-------->8--------
What this script does is:
On resource acquisition: Copy the gated configuration file (/etc/gated.conf)
to another file (/etc/gated-heartbeat.conf), activate the route for the
virtual IP
network and restart gated using the new file.
On resource loss: Restart gated using the original configuration.
Note: gated must always be running and must start at boot time using the
non-active
(default) config.
4) Modify /etc/gated.conf accordingly. Here is the /etc/gated.conf file
for machine 1:
--------8<--------
options syslog upto debug;
smux off;
bgp off;
egp off;
ospf off;
rip yes{
interface all noripin noripout;
interface eth0 ripin ripout version 2 multicast;
interface eth1 ripin ripout version 2 multicast;
trustedgateways 192.168.100.2 192.168.101.2 (...) # other routers in
the network ;
};
static {
192.168.200.1 masklen 32 interface 127.0.0.1 preference 0 retain;
192.168.300.1 masklen 32 interface 127.0.0.1 preference 0 retain;
};
import proto rip{
all;
};
# On exporte differentes affaires, en concordance avec le mode de
fonctionnement (prod/releve)
export proto rip{
proto static{
host 192.168.200.1 metric 1;
# host 192.168.300.1 metric 1; # heartbeat-toggle
};
};
-------->8--------
The gated-toggle script will look for all lines ending with "#
heartbeat-toggle" and turn them
on (or off) depending on the cluster state.
CONCLUSION
==========
That's about it. I suspect you could do something similar with zebra or
some other routing
software, as long you can restart it with a different config or (even
better) change it's
config dynamically (maybe you can even do this with gated, but I'm not
aware of this. Like
I said I'm no gated expert...)
Thanks and good luck,
Patrick LeBoutillier
--
=====================
Patrick LeBoutillier
Laval, Quebec, Canada
| 