LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: Again: LVS/Apache cluster freezes from time to time

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: Again: LVS/Apache cluster freezes from time to time
From: Jan Abraham <jan_abraham@xxxxxxx>
Date: Wed, 05 Nov 2003 15:49:36 +0100
> how do you know Apache isn't handling it? (I can't tell whether it is or 
> isn't,
> I'm just asking). If apache had handled the request, you wouldn't be able
> to tell from ESTABLISHED. The ESTABLISHED state will stay there till the 
> client
> does a close or apache closes it, but apache has a delayed close in case
> the client makes another request. So the connection could be in ESTABLISHED
> with the request fulfilled a while ago (about 2mins I think). 

The number of ESTABLISHED connections increases during the freeze, and
their receive queue get filled, but the send queue isn't. To my
understanding, this is a clear signal that Apache doesn't handle the
request (of course it does, after the freeze).
KeepAlive is turned off, so the connection will close after processing
the request, lingering in TIME_WAIT state for a couple of seconds.

> Has php/apache been configured to allow enough memory, processes.... 

Yes, this issue was checked many times :)
There is a maximum of 300 Apache processes at the moment, they consume
about 300MB of the RAM. There's still so much memory left that I can
turn off swap and system has still 500MB for caching files...

> Do you have any idea why this doesn't happen if the machine is outside
> LVS?

Not really, I'm afraid. I've checked packets for unusual behaviour and
contents... 

> how about trying different schedulers (lc, rr) or a different forwarder
> (LVS-NAT, LVS-DR). (I'm just shooting in the dark here, trying to collect
> more data).

I've already tried this... :(

May other equipment interfere with LVS? Our housing company uses two
core routers with STP for failover... I see their announcements (802.1d
and VRRP) in the tcpdump.

Can LVS confuse the ARP table of the switch? I noticed that the switch
sends IP packets to all hosts sometimes (and as far as I understand,
this is to get the MAC address into its ARP cache).

Jan


<Prev in Thread] Current Thread [Next in Thread>