|
Hi!
I finally found some time to write down a couple of notes
about ARP releated problems with LVS and 2.2 kernels.
Before we start, let's assume we have following network
configuration:
192.168.1.1 gateway
192.168.1.10 load balancer
192.168.1.2 virtual address
192.168.1.11 real server
10.10.10.10 client
Or goal is:
Virtual server should respond to arp requests for both
192.168.1.2 and 192.168.1.10
Real server should respond to arp requests for 192.168.1.11
but NOT 192.168.1.2.
Gateway should send packets for 192.168.1.2 to the
load balancer no matter what.
Problem 1: Interface aliases
Real server and load balancer need to have an interface
alias with the virtual address (192.168.1.2) in order to
respond to packets for virtual server. This interface alias
could normally be either eth0:0 or lo:0.
On the 2.0 kernels, the ARP responding ability of an
interface alias could either be enabled or disabled. If you
wanted eth0:0 not to respond to ARP requests, you could
simply say:
ifconfig eth0:0 192.168.1.2 -arp up
In the 2.2 kernels this doesn't work any more. Whether ARP
requests are responded or not, depends only on the way the real
interface is configured. So if eth0 responds to ARP requests
(which it normally should), eth0:0 will also respond to ARP
requests no matter what.
This means an ethernet alias (eth0:0) is not permitted on
real servers, because real servers should not respond ARP
requests.
On the other hand, loopback aliases never respond ARP
requests, which means, loopback alias (lo:0) must not be
used on a load balancer machine.
Problem 2: Loopback aliases
I haven't done much checking on loopback interface problem,
but it seems that if loopback alias is used on a real server,
the whole ARP gets screwed.
It appears that loopback interfaces get special ARP treatment
in the kernel, so I suggest avoiding the loopback aliases as
whole.
The question now is: What kind of an interface can I use on
real servers?
As I already noted, eth0:0 alias can not be used, because
such aliases respond to ARP requests. lo:0 aliases can not
be used, because they make ARP problems too.
In case of tunneling VS configuration, the answer is
trivial: tunl0. But to be honest, tunl0 interface can also
be used for direct routing.
With direct routing, the only thing we need an interface
for is to let kernel know we posses an additional IP
address. This means, we can set up any kind of an interface,
as long as it doesn't respond ARP requests. Instead of
tunl0, you could also set up a ppp0, slip0, eth1 or
whatever. I suggest setting up a tunl0:
ifconfig tunl0 192.168.1.2 -arp up
Problem 3: Real server ARP requests.
Suppose we have set up a virtual server as described at
the beginning. All computers are running, but no requests
have been made.
Then the client (10.10.10.10) sends a request to the
virtual address (192.168.1.2).
When the packet arrives to gateway, the gateway makes
an ARP query for the 192.168.1.2 and the load balancer
responds.
Gateway remembers the balancer's MAC address and sends
the packet to the balancer.
Balancer receives the packet, chooses the real server
and forwards the packet to the real server by direct
routing or tunneling method.
Real server receives the packet and generates a response
packet.
(until now everything works correctly)
When real server wants to send the response packet to
the gateway, it finds out, that it does not know the
gateway's MAC address.
It sends an ARP request to the local network and asks for
the gateway MAC address. This should look like:
ARP, who has 192.168.1.1, tell 192.168.1.11
But in reality, real server asks something like:
ARP, who has 192.168.1.1, tell 192.168.1.2,
because it takes the source addres from the packet
it wants to send.
Here the problems come in.
Gateway receives the packet and responds to it, which
is correct. But at the same time, gatweay does a little
optimization. It finds out, that the real server's MAC
address is not listed in it's ARP tables and adds the
entry into the table, just in case it might need that
address in the near future.
The ARP request contained the virtual IP address and
the real server's MAC address, so from now on, the
gateway will send all packets destined for the virtual
server to the real server instead (due to MAC address).
This means all packets that follow will avoid the
virtual server as whole and get responded by the
real server.
If the real server's ARP request would be:
ARP, who has 192.168.1.1, tell 192.168.1.11
all this would not have happened. Therefore I have
patched the 2.2 VS kernel in such a way, that it
composes ARP requests based on the address of the
interface selected by the routing tables instead of
the address taken from the packet itself.
In order for virtual server to work correctly, the
real servers should have patched kernels as well,
or at least copy the patched
/usr/src/linux/net/ipv4/arp.c
file to the real servers before compiling the kernels.
Conclusion
Those were my experience with ARP problems, and the
2.2 kernel virtual server.
I think it would be wise to add this letter to the
web site and notify the network developers about
our findings at some point in time.
Here are some golden rules I stick to, when I do
virtual server configuration:
Rule 1:
Do not use lo:0 alias on the load balancer.
Use eth0:0 alias instead.
Rule 2:
Avoid using lo:0 alias, not even on real servers.
Use tunl0 or some other simulated interface
on real servers instead.
Rule 3:
Apply the VS patch to kernels on real servers.
|
