LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: DoS protection strategies

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: DoS protection strategies
From: Ken Brownfield <krb@xxxxxxxxxxx>
Date: Tue, 18 Apr 2006 11:46:34 -0700
This 150 connection limit is the default MaxClients setting in Apache, which in practice should be adjusted as high as you can without Apache using more memory than you want it to (e.g., 80-100% of available RAM -- no need for the DoS to swap-kill your box, too). Each Apache process will use several megabytes (higher or lower depending on 32- or 64-bit platforms, add-on modules, etc) so this can't be set too high. Disabling KeepAlives will drop your working process count by roughly an order of magnitude, and unless you're just serving static content it's generally worth disabling. But for your case of 150 idle connections, it doesn't help.

Netfilter has various matching modules that can limit connections from and/or to specific IPs, for example:

iptables --table filter --append VIP_ACCEPT --match dstlimit -- dstlimit 666 --dstlimit-mode srcipdstip-dstport --dstlimit-name VIP_LIMIT --jump ACCEPT

The reason DoS attacks are so successful (especially full-handshake attacks) is that something needs to be able to cache and handle incoming connections. And that is exactly where Apache is weakest -- the process model is terrible at handling a high number of simultaneous, quasi-idle connections.

LVS has some DoS prevention settings which you should consider (drop_entry, drop_packet, secure_tcp) but they're generally only useful for SYN floods. A full handshake will be passed on through LVS to the application, and that is where the resources must be available. And given persistence, a single-IP attack will be extremely effective if you only have one (or few) real servers.

Once a connection has been made to Apache, it will need to either relegate idle connections out of process (see Apache 2.2's new event MPM, not sure if it only works on idle keepalives) or limit based on IP with the modules you mention.

This problem is difficult to solve completely, and I agree that solving it in Apache is the least powerful, least convenient, and highest overhead solution. Given Netfilter functionality (2.6 and later), the absence of throttles or connection limits in LVS isn't fatal. But I do feel that LVS could be made a more comprehensive system if it rolled in even basic connection throttling/limiting, plus a more closely integrated and maintained health checking system. And source-routing support. ;)

There are commercial products available that implement heavy-duty DoS/ intrusion protection. They block the vast majority of simple attacks and are crucial for any large-scale public-facing services. But a good distributed full-handshake or even valid HTTP request DoS is almost impossible to fully block.

Good luck,
--
Ken.

On Apr 18, 2006, at 8:53 AM, Willem de Groot wrote:

Hi,

To my surprise, opening 150 tcp connections to a default apache
installation is enough to effectively DoS it for a few minutes (until
connections time out). This could be circumvented by using
mod_throttle, mod_bwshare or mod_limitipconn but imho a much better
place to solve this is at the LVS loadbalancer. Which already does
source IP tracking for the "persistency" feature.

Did anyone implement such a feature? Considerations?

A sample script to test your webhosting provider:

#!/usr/bin/perl
my $target = shift or die "Usage: $0 <target>\n";
use IO::Socket::INET;
for my $t (0..300) {
  print "Try $t... ";
  $cons[$t] = IO::Socket::INET->new( PeerAddr => "$target:80", Proto
=> 'tcp', Blocking => 1 )
  or die "Couldn't connect!";
        print "connected!\n";
}
print "Enter to drop connections...\n";
<STDIN>;

Regards,
Willem
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://www.in-addr.de/mailman/listinfo/lvs-users


<Prev in Thread] Current Thread [Next in Thread>