LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: Our LVS/DR backends freezes

To: olle@xxxxxxxxxxx, <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: Our LVS/DR backends freezes
From: Joseph Mack NA3T <jmack@xxxxxxxx>
Date: Mon, 27 Nov 2006 08:12:34 -0800 (PST)
On Mon, 27 Nov 2006, Olle ~Vstlund wrote:

Our LVS-backends typically freezes after 5-7 days of normal operation.
These freezes are not system-crashes, but it seems like all new
TCP-connections towards the servers will hang forever. It is impossible
to logon or perform an su (they will hang), but existing sessions will
function fine as long as you don't issue 'critical commands' (commands
which perform a tcp-connection?).

such as loging in remotely, but not from the console...?

The syslogd stops writing the syslog,
etc. Looking at the servers activity using top reveals nothing abnormal
-- there is no swapping, cpu-usage is low, etc.

what about various outputs from ipvsadm on the director? Anything monotonically increasing there (I know the problem is on the realservers)?

Anything monotonically increasing on the realservers (eg look with netstat to see if running out of ports or all connections in FIN_WAIT)?


The cluster is hosting 14 websites, typically serving 1.5 million
request a normal day. As I said the cluster may run fine for a week and
then suddly the backends freezes. The funny thing is that both backend
usually freezes at roughly the same time.

presumably they've been evenly balanced.

I take it that this has something to do with LVS, ie you don't get the same behaviour with a bare single server?

The only cure we have come up with so far is to reboot the servers. Once
rebooted a server will run for days again. It has ocationally happed
that the second (frozen) server has recoverd once the first server is
rebooted.

do the realservers talk to each other (eg have a common disk system)?

Anyone out there having a good idea where to look for clues to what may
be wrong?

not a clue sorry

Joe
--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!




<Prev in Thread] Current Thread [Next in Thread>