|
On Apr 18, 2006, at 1:22 PM, Roberto Nibali wrote:
[...]
Most of our customer's httpd show RSS between 800KB and 2MB; some
of them it's including mod_perl or mod_php.
The mod_python and mod_php applications currently under my care are
at 38-44MB resident on 64-bit. On a minimal 64-bit box, I'm seeing
6MB resident. I've honestly never seen an application, either CGI-
or mod-based, use less than 2MB on 32-bit including the CGI, and most
in the 15-45MB range. As you say, I think the application is a huge
variable, but therein lies the weakness of the process model.
Disabling KeepAlives will drop your working process count by
roughly an order of magnitude, and unless you're just serving
static content it's generally worth disabling. But for your case
of 150 idle connections, it doesn't help.
Or you set your timeouts correctly, or you implement proper state
mapping using a reverse proxy and a cookie engine.
Timeouts certainly help, but that's somewhat akin to saying that if
you set your synrecv timeout low enough, the DoS won't hurt you. :)
KeepAlives by their nature will increase the simultanous connection
count, but I apologize if I came across as advocating turning them
off as a knee-jerk fix to connection-count problems.
Whether they're beneficial or not (for scalability reasons) depends
on whether you bottleneck on CPU or RAM first, and whether you're
willing to scale wider to keep the behavior change due to keepalives.
Netfilter has various matching modules that can limit connections
from and/or to specific IPs, for example:
iptables --table filter --append VIP_ACCEPT --match dstlimit --
dstlimit 666 --dstlimit-mode srcipdstip-dstport --dstlimit-name
VIP_LIMIT --jump ACCEPT
No wonder you have no memory left on your box :).
:) The rule was just an example, and 666 is my numeric "foo".
LVS has some DoS prevention settings which you should consider
(drop_entry, drop_packet, secure_tcp) but they're generally only
useful for SYN floods.
I would not call dropping a certain amount of illegitimate _and_
legitimate connections to be too useful when you're running on a
strict SLA. QoS approaches based on window scaling help a bit more.
Agreed, I was just mentioning them, not advocating them. drop_packet
and secure_tcp, set to 1, seem decent choices. If LVS is out of RAM,
I think your SLA is doomed, only to be perhaps aided by these
settings. Having them on all the time is indeed Bad.
Regarding throttling, I reckon you use the TC framework available
in the Linux kernel since long before netfilter hit the tables.
Good point, I had forgotten about TC, though I'm not sure it can
throttle *connections* vs *throughput*.
, plus a more closely integrated and maintained health checking
system.
How would you improve it? Suggestions are always welcome.
I had to completely rewrite the LVS alert module for mon, in addition
to tweaking several of the other mon modules. Now, this was on a so-
last-year 2.4 distro -- I haven't worked with LVS under 2.6 yet or
more modern mon installs. I also wrote a simple CLI interface
wrapper to ipvsadm, since editing the ipvsadm rules file isn't
terribly operator-friendly and prone to error for simple host/service
disables.
I think all the parts are there for a Unix admin to complete an
install. But for a health-checking, stateful-failover, user-friendly-
interface setup, it's pretty piecemeal. And there's no L7 to my
knowledge. There are some commercial alternatives (that will appeal
to some admins for these reasons) that are likely inferior overall to
LVS. I think the work lies most in integration, both of the
documentation and testing, and perhaps patch integration.
Just my soapbox -- I'm using LVS under heavy load right now so I'm
certainly not complaining.
And source-routing support. ;)
Which you did :).
Eh, not really -- I just accidentally reiterated the same problem and
same patch that someone posted a year or two ago. :)
There are commercial products available that implement heavy-duty
DoS/intrusion protection.
True. And have you looked at how they do it? Traffic contracts and
the sorts, like TC for example.
The primary parts of the systems I alluded to are the attack
fingerprints and flood detection that intelligently blocks bad
traffic, not good traffic. Nothing is 100%, but in terms of
intelligent, low-false-positive malicious request / flood blocking,
they do extremely well at blocking the bad stuff and passing the good
stuff.
Is it worth the bank that they charge, or the added points of
failure? Depends on how big your company is I suppose. But I know
of no direct OSS alternative -- or I'd use it!
Cheers,
--
Ken.
|
