LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] is it possible tohave ldirectorand realcluster server on

To: 'LinuxVirtualServer.org users mailing list.' <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] is it possible tohave ldirectorand realcluster server on same physical machine?
From: Mrvka Andreas <mrv@xxxxxx>
Date: Mon, 6 Dec 2010 17:01:52 +0100
Hello Darren,

you are the best.

With your config I've got it up and running within one hour.
I do not use stonith becauce both servers are inside Vmware but I am happy.

Additionally I have tomcat (standalone and not in pacemaker) on both nodes 
running
and there also exists a manual how to build a tomcat cluster
(http://tomcat.apache.org/tomcat-6.0-doc/cluster-howto.html) with Session 
Replication
and I hope that this one also does the job.

Thank you Darren, for this time.

Best regards
Andrew



-----Ursprüngliche Nachricht-----
Von: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] Im Auftrag von 
Darren.Mansell@xxxxxxxxxxxx
Gesendet: Montag, 6. Dezember 2010 13:09
An: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Betreff: Re: [lvs-users] is it possible tohave ldirectorand realcluster server 
on same physical machine?

This is all of my network config, and it works fine here:

/etc/sysctl.conf
# Disable response to broadcasts.
# You don't want yourself becoming a Smurf amplifier.
net.ipv4.icmp_echo_ignore_broadcasts = 1
# enable route verification on all interfaces
net.ipv4.conf.all.rp_filter = 1
# enable ipV6 forwarding
#net.ipv6.conf.all.forwarding = 1
# increase the number of possible inotify(7) watches
fs.inotify.max_user_watches = 65536
# avoid deleting secondary IPs on deleting the primary IP
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
net.ipv4.ip_forward = 1
#net.ipv4.conf.default.rp_filter = 1
#net.ipv4.conf.default.accept_source_route = 0
#net.ipv4.ip_nonlocal_bind=1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2


/etc/sysconfig/network/ifcfg-eth0
BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR='10.167.20.10/24'
MTU=''
NAME='NetXtreme II BCM5709 Gigabit Ethernet'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'


/etc/sysconfig/network/ifcfg-lo
# Loopback (lo) configuration
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
BROADCAST=127.255.255.255
IPADDR_2=127.0.0.2/8
STARTMODE=onboot
USERCONTROL=no
FIREWALL=no
IPADDR_VIP=10.167.20.100
NETMASK_VIP=255.255.255.255
NETWORK_VIP=10.167.20.0
BROADCAST_VIP=10.167.20.255
LABEL_VIP='VIP'


crm configure show
node NODE-MYSQL-02 \
        attributes standby="off"
node NODE-MYSQL-01 \
        attributes standby="off"
primitive STONITH-2 stonith:external/ibmrsa-telnet \
        params ip_address="10.167.20.22" nodename="NODE-MYSQL-02" 
password="PASSWORD" username="USERID" \
        meta target-role="Started"
primitive STONITH-1 stonith:external/ibmrsa-telnet \
        params ip_address="10.167.20.12" nodename="NODE-MYSQL-01" 
password="PASSWORD" username="USERID" \
        meta target-role="Started"
primitive Virtual-IP ocf:heartbeat:IPaddr2 \
        params lvs_support="true" ip="10.167.20.100" cidr_netmask="24" 
broadcast="10.167.20.255" \
        op monitor interval="1m" timeout="10s" \
        meta migration-threshold="10"
primitive ldirectord ocf:heartbeat:ldirectord \
        params configfile="/etc/ha.d/ldirectord.cf" \
        op monitor interval="2m" timeout="20s" \
        meta migration-threshold="10"
primitive MySQL ocf:heartbeat:mysql \
        params binary="/usr/bin/mysqld_safe" config="/etc/my.cnf" 
datadir="/data/mysql" user="mysql" pid="/var/lib/mysql/mysql.pid" 
socket="/var/lib/mysql/mysql.sock" test_passwd="password" 
test_table="ldirectord.connectioncheck" test_user="servicecheck" \
        op monitor interval="20s" timeout="10s" \
        meta migration-threshold="10" target-role="Started"
group Load-Balancing Virtual-IP ldirectord
clone MySQL-Clone MySQL
location Prefer-Node1 ldirectord \
        rule $id="prefer-node1-rule" 100: #uname eq NODE-MYSQL-01
location l-st-1 STONITH-1 -inf: NODE-MYSQL-01
location l-st-2 STONITH-2 -inf: NODE-MYSQL-02
property $id="cib-bootstrap-options" \
        dc-version="1.0.3-0080ec086ae9c20ad5c4c3562000c0ad68374f0a" \
        expected-quorum-votes="2" \
        no-quorum-policy="ignore" \
        start-failure-is-fatal="false" \
        stonith-action="reboot" \
        last-lrm-refresh="1291319704" \
        stonith-enabled="false"


/etc/ha.d/ldirectord.cf
checktimeout=5
checkinterval=7
autoreload=yes
logfile="/var/log/ldirectord.log"
quiescent=no
emailalert=alerts@xxxxxxxxxxxxx
virtual=10.167.20.100:3306
        fallback=10.167.20.20:3306
        real=10.167.20.10:3306 gate 1000
        #real=10.167.20.20:3306 gate 1
        service=mysql
        login="servicecheck"
        passwd="password"
        database="ldirectord"
        request="SELECT * from connectioncheck;"
        scheduler=sh
        protocol=tcp
        checktype=negotiate


Finally, ensure you have the cluster-network-kmp-default package installed for 
LVS on SLES.

Regards,
Darren Mansell

-----Original Message-----
From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Mrvka Andreas
Sent: 06 December 2010 11:05
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] is it possible tohave ldirectorand realcluster server 
on same physical machine?

I've forgot to explain:

if I do everything the same (referring sysctl for example)  then the servers 
are not responding in the network anymore.

Which causes this feature?

Best regards
Andrew


-----Original Message-----
From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of 
Darren.Mansell@xxxxxxxxxxxx
Sent: Freitag, 3. Dezember 2010 14:53
To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: Re: [lvs-users] is it possible to have ldirectorand realcluster server 
on same physical machine?

Glad it helped. This is my original howto for this kind of setup:

http://www.clusterlabs.org/wiki/Load_Balanced_MySQL_Replicated_Cluster

darren


-----Original Message-----
From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Mrvka Andreas
Sent: 03 December 2010 13:46
To: 'LinuxVirtualServer.org users mailing list.'
Subject: Re: [lvs-users] is it possible to have ldirectorand realcluster server 
on same physical machine?

Hi Darren,

thank you for the detailed infos.
I've read out of your messages that in sysctl.conf is nothing special for the 
lo-interface.
This is a very important information! Thank you.

I know that there is a parameter no-quorum=ignore but I've already run into it 
something.
It would be nice to know how to restore quorum if it happens again (what I 
don't believe after your HowTo) because 2-node it just for testing and I will 
also add more nodes.

Thank you at this time.
I am motivated to finish building the cluster.

Regards
Andrew


-----Ursprüngliche Nachricht-----
Von: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] Im Auftrag von 
Darren.Mansell@xxxxxxxxxxxx
Gesendet: Freitag, 3. Dezember 2010 14:20
An: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Betreff: Re: [lvs-users] is it possible to have ldirector and realcluster 
server on same physical machine?

1. Yes, ldirectord, cluster, real server, resources all on the same machine.

2. /etc/sysctl.conf is (appended to existing default) :
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2

You will need to put non-arping interfaces for VIPs on all nodes:
MYSQL-01:~ # cat /etc/sysconfig/network/ifcfg-lo # Loopback (lo) configuration
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
BROADCAST=127.255.255.255
IPADDR_2=127.0.0.2/8
STARTMODE=onboot
USERCONTROL=no
FIREWALL=no
IPADDR_VIP=10.167.20.100
NETMASK_VIP=255.255.255.255
NETWORK_VIP=10.167.20.0
BROADCAST_VIP=10.167.20.255
LABEL_VIP='VIP'

The cluster config for the VIP resource requires the lvs_support parameter set:
primitive Virtual-IP ocf:heartbeat:IPaddr2 \
        params lvs_support="true" ip="10.167.20.100" cidr_netmask="24" 
broadcast="10.167.20.255" \
        op monitor interval="1m" timeout="10s" \
        meta migration-threshold="10"

3. STONITH is there for fencing, but failover happens when the node is 
unavailabe, or the resource hits the failcount limit depending on your own 
settings. The Pacemaker documentation at Clusterlabs is excellent for this.

4. Not sure I understand the question. Quorum is taken care of by OpenAIS. If 
you only have a 2-node cluster you need to set the CRM property 
'no-quorum-policy' to 'ignore'.

Regards,
Darren

-----Original Message-----
From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Mrvka Andreas
Sent: 03 December 2010 11:53
To: 'LinuxVirtualServer.org users mailing list.'
Subject: Re: [lvs-users] is it possible to have ldirector and realcluster 
server on same physical machine?

Hi Darren,

pleased to get notice of someone who uses the same product.
I have lots of questions ;-)

1) Do do you arrange ldirector and real cluster on the same machine?
   Novell documentation do never go into detail so I believe it works.

2) How do you arrange sysctl configs (arp settings) in detail on all servers?

3) How do you arrange fail-over? via stonith ressource? how?

4) I often got the price to run into quorum (at pacemaker). How do you prevent 
it?
   And if the cluster crashes (unexpected shutdown) how do you restore its sync?


Best regards
Andrew
GoogleTalk: andreas.mrvka@xxxxxxxxx


-----Ursprüngliche Nachricht-----
Von: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] Im Auftrag von 
Darren.Mansell@xxxxxxxxxxxx
Gesendet: Freitag, 3. Dezember 2010 12:34
An: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Betreff: Re: [lvs-users] is it possible to have ldirector and real cluster 
server on same physical machine?

Hello.

We have around 20 nodes of SLES 11 HAE servers running in 2 or 3 node clusters.

All of them are configured with local resources and ldirectord to load-balance 
to the local nodes with a highly available ldirectord resource being able to 
move on failover.

I can give more info if required.


-----Original Message-----
From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx 
[mailto:lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Mrvka Andreas
Sent: 03 December 2010 10:43
To: 'lvs-users@xxxxxxxxxxxxxxxxxxxxxx'
Subject: Re: [lvs-users] is it possible to have ldirector and real cluster 
server on same physical machine?

Hi,

I've misstped the arp_announce entries. It should be 2 instead of 1.
After a reboot it looks quite good having only one ldirector.

I am satisfied at the moment.
Novell HAE (High Availability Extension) shipps pacemaker.
I will try to configure it managing ldirector service on both machines but only 
be explicitly active on one node.
And in the backup-case pacemaker should switch.

This sounds clear for me. But in fact, that the sysctl.cf file differs - how 
should this be updated?

Thank you very much till now.
Andrew


-----Ursprüngliche Nachricht-----
Von: Mrvka Andreas
Gesendet: Freitag, 3. Dezember 2010 11:09
An: 'lvs-users@xxxxxxxxxxxxxxxxxxxxxx'
Betreff: AW: [lvs-users] is it possible to have ldirector and real cluster 
server on same physical machine?

Hello Simon,

thank you for your fast response! I am happy for every feedback.
Actually I thought I can have more than one active ldirector (for failover and 
ldirector load balancing).


Well, okay let's do ldirector with one node.
I refer to your Ultramonkey Howto.

----- On my first node I run ldirector and cluster node --- eth0 is real IP and 
lo:0 is virtual IP. So I do have to arp_announce on lo:0 butnot arp_ignore.
net.ipv4.conf.all.arp_announce = 1 # enable the feature 
net.ipv4.conf.lo.arp_anounce = 1

----- On my second and n+1 node I run only cluster nodes --- eth0 is real IP 
and lo:0 is virtual IP. So I do have to arp_ignore on lo:0 but not arp_announce.
net.ipv4.conf.all.arp_ignore = 1 # enable the feature 
net.ipv4.conf.lo.arp_ignore = 1


Is this correct?

Thank for directing me into right direction.
Andrew


-----Ursprüngliche Nachricht-----
Von: Simon Horman [mailto:horms@xxxxxxxxxxxx]
Gesendet: Freitag, 3. Dezember 2010 10:39
An: Mrvka Andreas
Cc: 'lvs-users@xxxxxxxxxxxxxxxxxxxxxx'
Betreff: Re: [lvs-users] is it possible to have ldirector and real cluster 
server on same physical machine?

On Fri, Dec 03, 2010 at 09:31:19AM +0100, Mrvka Andreas wrote:
> Hello list,
>
> I hope you can help me.
> I am using a simple 2 node cluster environment on SUSE Linux Enterprise 
> Server 11 SP1 HAE.
>
> I only use ldiretord/ipvs and an apache service on both machines.
> My virtual IP address on interface lo:0 and real cluster IP on eth0.
>
> Regarding to the documentation of Novell it should be possible to have 
> ldirectord etc. on the same machines.
> Do you agree?
>
> I ask because if I do a tcpdump I see lots of traffic from the client
> browser to the server (http port) and the tcp session will never end.
> ( client side has finished the transfer but on the server side tcpdump
> runs and runs....)

Hi Andreas,

if you want a pair of machines and you want one of them to be the active linux 
director + a real server, and you want the other to be the backup-linux 
director + a real server then yes it is possible, but you need to take special 
care.

1. On the active linux director the VIP should be on lo and
   On the backup-linux director the VIP should be on ethX
   - That is, you need to move the VIP when a machine moves
     from being active to backup and vice versa.
2. The backup-linux director must not have LVS rules in place for the
   service being handled by the active linux director.
   - That is, you need to have ldirectord running on only the active
     linux director. Or in other words, you need to start it
     when a machine becomes active and stop it when it becomes a backup.
3. You have to make sure that arp_ignore and arp_announce are set correctly.

Its a bit old, but I have some notes on this at 
http://www.ultramonkey.org/3/topologies/sl-ha-lb-eg.html

(In relation to ARP you want the "Debian" notes unless  you have a really 
ancient kernel.)

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send 
requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send 
requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send 
requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send 
requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send 
requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send 
requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx Send 
requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>