LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: Our LVS/DR backends freezes

To: Olle Östlund <olle@xxxxxxxxxxx>
Subject: Re: Our LVS/DR backends freezes
Cc: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
From: Joseph Mack NA3T <jmack@xxxxxxxx>
Date: Mon, 27 Nov 2006 16:56:43 -0800 (PST)
On Mon, 27 Nov 2006, Olle Östlund wrote:

Remote logon via ssh does not work (hangs or will get a "connection closed by remote server" type of message). Logging in via the console does not work (hangs after password has been typed in), using su will produce the same result.

I get this when I don't have enough entries in

/etc/securetty

so it sounds like a resource problem but it's hard to think of a resource that would be short in a realserver but not an otherwise identical standalone server.

nothing strange with `ulimit -a` ?

Nothing monotonically increasing as far as we have seen, no.

:-(

We are running a "weighted round robin" load balancer algoritm, and the director will set weight 0 (= no traffic) on a backend once it has frozen (not responding to the directors status requests). Then the number of active connections sloooowly drops.

They shouldn't slowly drop.

All connections should be gone in the time the clients are connected + FIN_WAIT. How long does a client hold a tomcat connection? seconds, minutes, hours?

I haven't setup a tomcat server, but I assume the connection time is comparable to connecting to a regular html/http server, ie get your hit(s) and you're gone (seconds).

Where are you measuring the number of connections? with ipvsadm on the director or with netstat on the realserver?

Netstat typically reports a normal number of connections (~ 100) in various states.

"Runnig out of ports" sounds interesting. It sure looks like we are running out of something. How would one go about detecting if we are running out of ports?



Remeber, if looks like we are running out of something which causes
applications to wait instead of bailing out with an error message. Would running out of ports cause applications to wait?

actually you aren't going to run out of ports, you've got the server end, which is going to have multiple processes listening on port 80. The client can (in principle) run out of ports.

do the realservers talk to each other (eg have a common disk
system)?

Well, no disk- or file-sharing. The Tomcats are communicating HTTP session-replication data between backends. Our HTTP-sessions contain very little data, so the traffic should not be that big.

it's the only thing different to operating as a standalone server that you've mentioned.

Anyone out there having a good idea where to look for clues to what may
be wrong?

no, still just going through obvious checklists here

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!




<Prev in Thread] Current Thread [Next in Thread>