LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: Again: LVS/Apache cluster freezes from time to time

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: Again: LVS/Apache cluster freezes from time to time
From: Joseph Mack <mack.joseph@xxxxxxx>
Date: Wed, 05 Nov 2003 10:10:00 -0500
Jan Abraham wrote:
> 
> > how do you know Apache isn't handling it? 

> The number of ESTABLISHED connections increases during the freeze, and
> their receive queue get filled, but the send queue isn't. To my
> understanding, this is a clear signal that Apache doesn't handle the
> request (of course it does, after the freeze).
> KeepAlive is turned off, so the connection will close after processing
> the request, lingering in TIME_WAIT state for a couple of seconds.

OK, thanks

> > Has php/apache been configured to allow enough memory, processes....
> 
> Yes, this issue was checked many times :)

had to check :-) thanks


> > Do you have any idea why this doesn't happen if the machine is outside
> > LVS?
> 
> Not really, I'm afraid. I've checked packets for unusual behaviour and
> contents...
> 
> > how about trying different schedulers (lc, rr) or a different forwarder
> > (LVS-NAT, LVS-DR). (I'm just shooting in the dark here, trying to collect
> > more data).
> 
> I've already tried this... :(

oh, well
 
> May other equipment interfere with LVS? Our housing company uses two
> core routers with STP for failover... I see their announcements (802.1d
> and VRRP) in the tcpdump.
> 
> Can LVS confuse the ARP table of the switch? 


> I noticed that the switch
> sends IP packets to all hosts sometimes (and as far as I understand,
> this is to get the MAC address into its ARP cache).

A thread recently (search for "BUCHE" on the list archive) resulted in the
following posting

http://marc.theaimsgroup.com/?l=linux-virtual-server&m=106743944016355&w=2

where Frederic found that the switch was caching the results of arp requests.
LVS-NAT doesn't play games with the arp tables, so if the freeze 
occurs with LVS-NAT then the switch isn't your problem. However if it
occurs only with LVS-DR, then possibly your switch is involved.
When the router asks "who has VIP, tell router" then only the director
should reply and the switch should only see the directors MAC address.
When packets are being send from the realserver:VIP to the router,
then the switches might notice that packets from the VIP are coming
from different MACs. For this reason LVS-DR doesn't work with ATM.
Even though there is nothing in the tcpip spec to say that packets
from a particular IP must always come from the same MAC address,
ATM checks whether the source MAC address has changed and if
it has disallows the packet. Thus with ATM the packets from the 
realservers can't get back to the client. According to people 
here, the ATM spec is broken.

I'm not sure how you could check whether this is a problem for you.

Do you have a simple unmanaged switch, and can you run with just
one realserver?

Joe
-- 
Joseph Mack PhD, High Performance Computing & Scientific Visualization
SAIC, Supporting the EPA Research Triangle Park, NC 919-541-0007
Federal Contact - John B. Smith 919-541-1087 - smith.johnb@xxxxxxx
<Prev in Thread] Current Thread [Next in Thread>