LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: LVS DR increased "system CPU" and Excessive Network Traffic

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: LVS DR increased "system CPU" and Excessive Network Traffic
From: Graeme Fowler <graeme@xxxxxxxxxxx>
Date: Thu, 05 Apr 2007 16:58:36 +0100
On Wed, 2007-04-04 at 20:35 +0100, Iain Young wrote:
> I've just installed a couple of machines with LVS on, using the
> DR method. I also used the localnode feature, and started the
> sync daemon as master on one, and backup on the other.
> 
> The nodes are functioning both as Director, and Real Servers,
> and now, both machines are:
> 
> a) showing "System CPU" at around 20%, which vanishes if I kill
> LVS.
> 
> b) Sending out an awful lot of LAN traffic (120,000 packets an
> hour!), which I'm guessing is multicast [which is to be expected],
> but surely it doesnt need *that* many packets! Again, it dissapears
> when I shutdown LVS.

For the moment, just try it without the sync daemon at all. What you're
probably seeing is packets being "reflected" between servers because of
the following hypothetical scenario:

1. New connection arrives on Server1, is handled by localnode on
Server1. Sync daemon sends established info to Server2. Connection gets
handled and cleared from LVS table.
2. Next new connection arrives on Server1 from same client, is
round-robinned to Server2, which has connection in its' table yet to
expire from Server1 and sends the packet back. Server1 looks up in its'
table and finds that it sent the packet to Server2, so sends it back
again, and so on, ad infinitum.

If dropping the sync daemon clears the problem, you need to investigate
using netfilter marks (fwmarks) on inbound connections and then
configure your LVS VSes to match mark values rather than IP addresses.
It helps if you have two NICs per server, as you can send the
inter-server traffic via the "backend" non-client facing LAN and
simplify the marking of packets quite significantly.

Graeme


<Prev in Thread] Current Thread [Next in Thread>