LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: ESTABLISHED Connection Spike (OT?)

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: ESTABLISHED Connection Spike (OT?)
From: Jacob Coby <jcoby@xxxxxxxxxxxxxxx>
Date: Fri, 19 Aug 2005 15:53:06 -0400
Jacob Coby wrote:
Hi all,

I've got an IP addr that, on occassion, takes up a mass of connections and leaves them in an ESTABLISHED state. The IP addr is of a business that uses our website, but it's causing a DOS of sorts. I don't know if this is a bug in LVS or not. Any pointers are appreciated.

As an update, we traced this back to a problem with the firewall the company is using. Both firewalls are made by Watchguard (http://www.watchguard.com). They are different models - one is a Firebox and I'm not sure what the other is. The Firebox apparently can operate in two NAT modes: direct (?) and proxy. In direct mode, it is leaving hundreds of connections in the EST state. In proxy mode, it works correctly.

The other firewall can only act in direct mode, so it is still causing a problem.

Before I try contacting the company, are there any known issues with the version of LVS I'm running that would cause this behavior? Are they any known issues with this brand of firewall?


Software:
ipvsadm v1.21 2002/07/09 (compiled with popt and IPVS v1.0.4)
kernel-2.4.20-28.7.um.3
redhat 7.3

Symptoms:
1. Connections jump from 50-100 up to 300-600.
2. A single IP address takes up 80-90% of those connections.
3. All of the connections from that ip address are in the ESTABLISHED state. 4. Very few of them are actually sending/receiving data (when using tcpdump -xX -s 1024 "host bad.ip.addr"). I see a few packets with the F and S flags set.

Because these are all ESTABLISHED connections to our website, they're taking up an apache process, and eventually locking everyone else out.

Any ideas?  snort logs don't show anything malicious from the ip.

Thanks,


--
-Jacob

<Prev in Thread] Current Thread [Next in Thread>