LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

HOWTO: LVS and dynamic routing (using gated)

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: HOWTO: LVS and dynamic routing (using gated)
From: Patrick LeBoutillier <patl@xxxxxxxxxxxx>
Date: Wed, 26 May 2004 13:29:30 -0400
Hi all,

Here is a "recipe" I cooked up (!) on creating LVS clusters with machines that support
redundant networking.

Note: This is a work in progess so any feedback is welcome. Typos and/or errors are certainly present. I'm no gated or LVS expert, so maybe there's a better way of doing this, but I spent a lot of time working on it so I figured it could be of use to other
folks as well.

I have performed many tests with this setup and it performed properly. If you find
otherwise please let it be known.


SCENARIO
========
Our production environment is fully redundant at the network level (each machine has two network interfaces, each connected to a different network). All machine are connected to both these networks and data can come from either network. On each machine, service run on a local network address and gated announces the route for these networks via both
network interface.

My task was to create an LVS cluster of 2 such machines (each a potential director and
realserver as well).


NETWORK SETUP
=============
Network 1 is 192.168.100.0/24
Network 2 is 192.168.101.0/24

Machine 1:
 - eth0: 192.168.100.1
 - eth1: 192.168.101.1
 - local network on loopback (lo:real): 192.168.200.1/32

Machine 2:
 - eth0: 192.168.100.2
 - eth1: 192.168.101.2
 - local network on loopback (lo:real): 192.168.201.1/32

Virtual IP is 192.168.300.1


GATED SETUP
===========
Have gated annonce (and accept) the following routes:

Machine 1:
 - announce 192.168.200.1/32
 - accept routes from 192.168.100.2 and 192.168.101.2

Machine 2:
 - announce 192.168.201.1/32
 - accept routes from 192.168.100.1 and 192.168.101.1

These routes will be used by ldirectord to monitor the realservers.


RECIPE
======
1) Install UltraMonkey as usual, but:
  - Make sure to configure ping nodes in both networks.
  - Create the virtual IP alias as 192.168.300.1
- A virtual service definition in ldirectord.cf should look something like this:
    virtual=192.168.300.1:80
            real=192.168.200.1:80 gate
            real=192.168.201.1:80 gate
            service=http
            checkport=80
            request="/test.html"
            receive="test"
            scheduler=rr
            protocol=tcp

In a normal setup, heartbeat manages the virtual IP alias and brings it up
on the active director. If I understand correctly, an arp request is then
sent, making the other machines in the local network aware that the active
director is now the machine to be reached for the virtual IP.

In this setup we will tell heartbeat to leave the virtual IP alias alone
and have it tell gated to announce the route for the 192.168.300.1/32 network instead. Therefore ONLY the active director will anounce the routes to reach
the virtual IP network.

2) Change your haresources line to something like this:

  node1.cluster.tld gated-toggle ldirectord


3) Place the following (or equivalent) code in a file called /etc/ha.d/resource.d/gated-toggle:

--------8<--------
#!/bin/bash
#
# This gated control script should only be called by heartbeat!
#
# start: RESTART gated with the original (non-director config)
# stop:  RESTART gated with the director config
#

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0

gdc=/usr/sbin/gdc
gated=/usr/sbin/gated
prog=gated

if [ ! -f /etc/gated.conf -o ! -f $gdc ] ; then
       action $"Not starting $prog: " true
       exit 0
fi

PATH=$PATH:/usr/bin:/usr/sbin

RETVAL=0

start() {
       echo -n $"Starting $prog: "
       CFG=$1
       if [ "$CFG" != "" ] ; then
RES='$2$3' RE="s/^(\s*\#+)(.*)(\#\s*heartbeat-toggle\s*)$/$RES/"
               /usr/bin/perl -p -e "$RE" /etc/gated.conf > $CFG
               daemon $gated -f $CFG
       else
               daemon $gated
       fi
       RETVAL=$?
       [ $RETVAL -eq 0 ] && touch /var/lock/subsys/gated
       echo
       return $RETVAL
}

stop() {
       # Stop daemons.
       action $"Stopping $prog" $gdc stop
       RETVAL=$?
       if [ $RETVAL -eq 0 ] ; then
               rm -f /var/lock/subsys/gated
       fi
       return $RETVAL
}

# See how we were called.
case "$1" in
 start)
       stop
       start "/etc/gated-heartbeat.conf"
       ;;
 stop)
       stop
       start
       ;;
 *)
       echo $"Usage: $0 {start|stop}"
       exit 1
esac

exit $RETVAL
-------->8--------

What this script does is:

On resource acquisition: Copy the gated configuration file (/etc/gated.conf)
to another file (/etc/gated-heartbeat.conf), activate the route for the virtual IP
network and restart gated using the new file.

On resource loss: Restart gated using the original configuration.

Note: gated must always be running and must start at boot time using the non-active
(default) config.


4) Modify /etc/gated.conf accordingly. Here is the /etc/gated.conf file for machine 1:
--------8<--------
options syslog upto debug;

smux off;
bgp off;
egp off;
ospf off;

rip yes{
 interface all noripin noripout;
 interface eth0 ripin ripout version 2 multicast;
 interface eth1 ripin ripout version 2 multicast;
trustedgateways 192.168.100.2 192.168.101.2 (...) # other routers in the network ;
};


static {
       192.168.200.1 masklen 32 interface 127.0.0.1 preference 0 retain;
       192.168.300.1 masklen 32 interface 127.0.0.1 preference 0 retain;
};

import proto rip{
 all;
};

# On exporte differentes affaires, en concordance avec le mode de fonctionnement (prod/releve)
export proto rip{
 proto static{
         host 192.168.200.1 metric 1;
#          host 192.168.300.1 metric 1; # heartbeat-toggle
 };
};
-------->8--------

The gated-toggle script will look for all lines ending with "# heartbeat-toggle" and turn them
on (or off) depending on the cluster state.



CONCLUSION
==========
That's about it. I suspect you could do something similar with zebra or some other routing software, as long you can restart it with a different config or (even better) change it's config dynamically (maybe you can even do this with gated, but I'm not aware of this. Like
I said I'm no gated expert...)


Thanks and good luck,

Patrick LeBoutillier
--

=====================
Patrick LeBoutillier
Laval, Quebec, Canada

<Prev in Thread] Current Thread [Next in Thread>