LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

[lvs-users] Source Hashing (Again)

To: <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: [lvs-users] Source Hashing (Again)
From: <Darren.Mansell@xxxxxxxxxxxx>
Date: Wed, 29 Sep 2010 11:36:04 +0100
Hello everyone.

 

We have a 2 node load-balanced MySQL cluster running in active-active
mode. Linux-HA is running on each with LVS balancing connections on the
nodes themselves.

 

This works great for other production clusters but the MySQL nodes
require connection jailing, i.e. a single client that connects to the
cluster IP must always only connect to a single node and not balance
requests to the other MySQL node due to replication race conditions.
(Foreign Key issues etc.)

 

So for this we use source hashing investigating after looking through
the archives of this list to see that as long as you give the weighting
a fairly high number it should be able to balance connections when one
node is unavailable and the weighting is set to 0. Unfortunately this
doesn't seem to happen. These are my current rules:

 

OGG-MYSQL-01:~ # ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  10.167.20.100:6446 sh

  -> 10.167.20.10:6446            Local   100    3          1         

  -> 10.167.20.20:6446            Route   110    0          0         

TCP  10.167.20.100:3306 sh

  -> 10.167.20.10:3306            Local   100    32         0         

  -> 10.167.20.20:3306            Route   110    0          0

 

When ldirectord detects that MySQL has died it changes the weighting
where connections should then go to the other node, but they don't seem
to:

 

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  OGG-MYSQL-NLB:6446 sh

  -> OGG-MYSQL-01:6446            Local   0      12         0         

  -> OGG-MYSQL-02:6446            Route   110    0          0         

TCP  OGG-MYSQL-NLB:mysql sh

  -> OGG-MYSQL-01:mysql           Local   0      74         0         

  -> OGG-MYSQL-02:mysql           Route   110    0          0

 

And new connections can't access the VIP:

 

CLIENT-01:~ # mysql -pxxxxxx -h10.167.20.100

ERROR 2003 (HY000): Can't connect to MySQL server on '10.167.20.100'
(111)

 

But if I then use WLC it's fine to balance:

 

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  10.167.20.100:6446 wlc

  -> 10.167.20.10:6446            Local   0      1          0

  -> 10.167.20.20:6446            Route   110    1          0

TCP  10.167.20.100:3306 wlc

  -> 10.167.20.10:3306            Local   0      1          0

  -> 10.167.20.20:3306            Route   110    0          1

 

CLIENT-01:~ # mysql -pxxxxxx -h10.167.20.100

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 545052

Server version: 5.1.38-enterprise-gpl-advanced-log MySQL Enterprise
Server - Advanced Edition (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the current input
statement.

mysql>

 

I could use WLC with persistence but then you could have 1 client node
with a connection to each MySQL node and it's no better off. I believe
only source hashing can ensure the source IP gets jailed to one node
therefore I have to try and make that work unless anyone can suggest a
better way?

 

Further info can be supplied upon request. Thanks in advance for any
help.

 

Darren

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>