LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: DoS protection strategies

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: DoS protection strategies
From: Ken Brownfield <krb@xxxxxxxxxxx>
Date: Tue, 18 Apr 2006 14:19:43 -0700
On Apr 18, 2006, at 1:22 PM, Roberto Nibali wrote:
[...]
Most of our customer's httpd show RSS between 800KB and 2MB; some of them it's including mod_perl or mod_php.

The mod_python and mod_php applications currently under my care are at 38-44MB resident on 64-bit. On a minimal 64-bit box, I'm seeing 6MB resident. I've honestly never seen an application, either CGI- or mod-based, use less than 2MB on 32-bit including the CGI, and most in the 15-45MB range. As you say, I think the application is a huge variable, but therein lies the weakness of the process model.

Disabling KeepAlives will drop your working process count by roughly an order of magnitude, and unless you're just serving static content it's generally worth disabling. But for your case of 150 idle connections, it doesn't help.

Or you set your timeouts correctly, or you implement proper state mapping using a reverse proxy and a cookie engine.

Timeouts certainly help, but that's somewhat akin to saying that if you set your synrecv timeout low enough, the DoS won't hurt you. :) KeepAlives by their nature will increase the simultanous connection count, but I apologize if I came across as advocating turning them off as a knee-jerk fix to connection-count problems.

Whether they're beneficial or not (for scalability reasons) depends on whether you bottleneck on CPU or RAM first, and whether you're willing to scale wider to keep the behavior change due to keepalives.

Netfilter has various matching modules that can limit connections from and/or to specific IPs, for example: iptables --table filter --append VIP_ACCEPT --match dstlimit -- dstlimit 666 --dstlimit-mode srcipdstip-dstport --dstlimit-name VIP_LIMIT --jump ACCEPT

No wonder you have no memory left on your box :).

:)  The rule was just an example, and 666 is my numeric "foo".

LVS has some DoS prevention settings which you should consider (drop_entry, drop_packet, secure_tcp) but they're generally only useful for SYN floods.

I would not call dropping a certain amount of illegitimate _and_ legitimate connections to be too useful when you're running on a strict SLA. QoS approaches based on window scaling help a bit more.

Agreed, I was just mentioning them, not advocating them. drop_packet and secure_tcp, set to 1, seem decent choices. If LVS is out of RAM, I think your SLA is doomed, only to be perhaps aided by these settings. Having them on all the time is indeed Bad.

Regarding throttling, I reckon you use the TC framework available in the Linux kernel since long before netfilter hit the tables.

Good point, I had forgotten about TC, though I'm not sure it can throttle *connections* vs *throughput*.

, plus a more closely integrated and maintained health checking system.

How would you improve it? Suggestions are always welcome.

I had to completely rewrite the LVS alert module for mon, in addition to tweaking several of the other mon modules. Now, this was on a so- last-year 2.4 distro -- I haven't worked with LVS under 2.6 yet or more modern mon installs. I also wrote a simple CLI interface wrapper to ipvsadm, since editing the ipvsadm rules file isn't terribly operator-friendly and prone to error for simple host/service disables.

I think all the parts are there for a Unix admin to complete an install. But for a health-checking, stateful-failover, user-friendly- interface setup, it's pretty piecemeal. And there's no L7 to my knowledge. There are some commercial alternatives (that will appeal to some admins for these reasons) that are likely inferior overall to LVS. I think the work lies most in integration, both of the documentation and testing, and perhaps patch integration.

Just my soapbox -- I'm using LVS under heavy load right now so I'm certainly not complaining.

And source-routing support. ;)

Which you did :).

Eh, not really -- I just accidentally reiterated the same problem and same patch that someone posted a year or two ago. :)

There are commercial products available that implement heavy-duty DoS/intrusion protection.

True. And have you looked at how they do it? Traffic contracts and the sorts, like TC for example.

The primary parts of the systems I alluded to are the attack fingerprints and flood detection that intelligently blocks bad traffic, not good traffic. Nothing is 100%, but in terms of intelligent, low-false-positive malicious request / flood blocking, they do extremely well at blocking the bad stuff and passing the good stuff.

Is it worth the bank that they charge, or the added points of failure? Depends on how big your company is I suppose. But I know of no direct OSS alternative -- or I'd use it!

Cheers,
--
Ken.

<Prev in Thread] Current Thread [Next in Thread>