|
Hi guys,
I'm having major difficulty at the moment with LVS and I figured I should
probably see if any of you guys can shed any light on my problems. Firstly, we
have a set up like the following (public IP addresses have been substituted):
clients
|
router (Router for 1.2.3.0/24 is 1.2.3.254)
|
__________ |
| | | VIP 1.2.3.4
| director |--- DIP 1.1.1.1
|__________| |
|
|
------------------------------------
| | |
| | |
RIP1 1.2.3.1 RIP2 1.2.3.2 RIP3 1.2.3.3
VIP 1.2.3.4 VIP 1.2.3.4 VIP 1.2.3.4
_____________ _____________ _____________
| | | | | |
| realserver | | realserver | | realserver |
|_____________| |_____________| |_____________|
There are several clusters running on this LVS system on their own different
VLANS, and they're all running beautifully. The problem has arisen when we've
setup our first cluster with Debian Lenny nodes. There are three nodes in this
cluster in total, each with a RIP on eth0, a VIP (on a loopback interface,
lo:1) and another IP on eth1. The problem that has arisen is that ldirectord
is constantly deleting/re-adding nodes from the cluster, even though the nodes
are up and running perfectly. This results in a partially working setup, in
that it actually works, but is very unstable/unreliable due to the constant
dropping/re-adding. We're running ping and HTTP/HTTPS checks both internally
and externally to the RIP's so that we know the nodes are stable. All three
nodes are set up identically, so when we fix one node, we'll be able to fix the
other two in exactly the same way.
As per the example layout above, you'll see that the VIP and the RIP's are all
on the same subnet, and as such they're all on the same VLAN. We've isolated
the issue we're having as an ARP problem. I've proven this by doing a tcpdump
on all of the nodes showing that they don't reply to ARP requests coming from
1.2.3.4, the VIP. After the director tries requesting this information
repeatedly using its VIP (usually around 15-20 times), it then seems to
fall-back to making the request via its DIP of 1.1.1.1 which is then promptly
replied to.
Here is an example of such a tcpdump from 1.2.3.2 which shows this behavior:
# tcpdump -ni eth0 | grep arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:47:48.549989 arp who-has 1.2.3.2 tell 1.2.3.254
20:47:48.549996 arp reply 1.2.3.2 is-at 00:18:8b:f7:cc:35 <--- Responds to the
router OK
20:51:05.964539 arp who-has 1.2.3.2 tell 1.2.3.4
20:51:06.964322 arp who-has 1.2.3.2 tell 1.2.3.4
20:51:07.230061 arp who-has 1.2.3.3 tell 1.2.3.4
20:51:07.963226 arp who-has 1.2.3.2 tell 1.2.3.4
20:51:08.233043 arp who-has 1.2.3.3 tell 1.2.3.4 <--- Doesn't respond to the
VIP on the director
20:51:08.966106 arp who-has 1.2.3.2 tell 1.2.3.4
20:51:09.232216 arp who-has 1.2.3.3 tell 1.2.3.4
20:51:09.965515 arp who-has 1.2.3.2 tell 1.2.3.4
20:51:10.968835 arp who-has 1.2.3.2 tell 1.2.3.4
<more erroneous requests snipped>
20:51:14.931208 arp who-has 1.2.3.2 tell 1.2.3.254
20:51:14.931215 arp reply 1.2.3.2 is-at 00:18:8b:f7:cc:35 <--- Responds to the
router OK
20:51:15.013430 arp who-has 1.2.3.2 tell 1.1.1.1
20:51:15.013443 arp reply 1.2.3.2 is-at 00:18:8b:f7:cc:35 <--- Responds to the
directors DIP OK
Futher more, we cannot arping the realservers from a terminal on the director,
unless the lo:1 interface on the realserver is taken down. If it's ifdown'd
the arpings work fine, but cease as soon as we issue an 'ifup lo:1' on the
realserver. It almost seems as if the lo:1 interface on the realservers is
working as if it's a normal interface, which of course it isn't. Could it be
that the arp replies are disappearing into lo:1 on the realservers, never to be
seen again?
As I noted previously, several other customers of ours have this exact same
setup (ablit with Windows realservers). We can ping all of their RIP's from
the director perfectly well, which leads us to believe that this is a problem
somewhere in the setup of these Debian realservers. Please find below output
of ifconfig, 'cat /etc/sysconfig.conf' and 'route -n' on our realservers:
# ifconfig
eth0 Link encap:Ethernet HWaddr 00:18:8b:f7:cc:35
inet addr:1.2.3.2 Bcast:1.2.3.255 Mask:255.255.255.0
inet6 addr: fe80::218:8bff:fef7:cc35/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:692317 errors:0 dropped:0 overruns:0 frame:0
TX packets:938914 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:301829193 (287.8 MiB) TX bytes:1075675350 (1.0 GiB)
Interrupt:16
eth1 Link encap:Ethernet HWaddr 00:18:8b:f7:cc:36
inet addr:192.168.0.1 Bcast:192.168.0.15 Mask:255.255.255.240
inet6 addr: fe80::218:8bff:fef7:cc36/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2237477 errors:0 dropped:0 overruns:0 frame:0
TX packets:1528671 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2792808412 (2.6 GiB) TX bytes:130718722 (124.6 MiB)
Interrupt:17
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:96993 errors:0 dropped:0 overruns:0 frame:0
TX packets:96993 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:11393978 (10.8 MiB) TX bytes:11393978 (10.8 MiB)
lo:1 Link encap:Local Loopback
inet addr:1.2.3.4 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
# cat /etc/sysctl.conf | grep -v "#"
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.eth0.arp_announce = 2
net.ipv4.conf.eth1.arp_ignore = 1
net.ipv4.conf.eth1.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
1.2.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.240 U 0 0 0 eth1
0.0.0.0 1.2.3.254 0.0.0.0 UG 0 0 0 eth0
Hopefully someone out there might have some idea what is happening with our
setup! :) Thanks for getting through my post in one piece!
Kind regards,
Chris Merrett
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
