LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: DoS protection strategies

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: DoS protection strategies
From: Roberto Nibali <ratz@xxxxxxxxxxxx>
Date: Tue, 18 Apr 2006 22:22:50 +0200
Ken,

I reckon people have different setups and thus our views seem to differ a bit. Without proper numbers and test conduct configurations that are repeatable by other persons, we'll probably compare apples to oranges. That said, I should nevertheless like to comment on this based on my limited experience with web services.
This 150 connection limit is the default MaxClients setting in Apache, which in practice should be adjusted as high as you can without Apache using more memory than you want it to (e.g., 80-100% of available RAM -- no need for the DoS to swap-kill your box, too). Each Apache process will use several megabytes (higher or lower depending on 32- or 64-bit platforms, add-on modules, etc) so this can't be set too high.
Most of our customer's httpd show RSS between 800KB and 2MB; some of 
them it's including mod_perl or mod_php.
Disabling KeepAlives will drop your working process count by roughly an order of magnitude, and unless you're just serving static content it's generally worth disabling. But for your case of 150 idle connections, it doesn't help.
Or you set your timeouts correctly, or you implement proper state 
mapping using a reverse proxy and a cookie engine.
Netfilter has various matching modules that can limit connections from and/or to specific IPs, for example:
iptables --table filter --append VIP_ACCEPT --match dstlimit --dstlimit 
666 --dstlimit-mode srcipdstip-dstport --dstlimit-name VIP_LIMIT --jump 
ACCEPT
No wonder you have no memory left on your box :).

The reason DoS attacks are so successful (especially full-handshake attacks) is that something needs to be able to cache and handle incoming connections. And that is exactly where Apache is weakest -- the process model is terrible at handling a high number of simultaneous, quasi-idle connections.
100% agreed.

LVS has some DoS prevention settings which you should consider (drop_entry, drop_packet, secure_tcp) but they're generally only useful for SYN floods.
I would not call dropping a certain amount of illegitimate _and_ 
legitimate connections to be too useful when you're running on a strict 
SLA. QoS approaches based on window scaling help a bit more.
A full handshake will be passed on through LVS to the application, and that is where the resources must be available. And given persistence, a single-IP attack will be extremely effective if you only have one (or few) real servers.
Yep.

Once a connection has been made to Apache, it will need to either relegate idle connections out of process (see Apache 2.2's new event MPM, not sure if it only works on idle keepalives) or limit based on IP with the modules you mention.
This problem is difficult to solve completely, and I agree that solving 
it in Apache is the least powerful, least convenient, and highest 
overhead solution.  Given Netfilter functionality (2.6 and later), the 
absence of throttles or connection limits in LVS isn't fatal.  But I do 
feel that LVS could be made a more comprehensive system if it rolled in 
even basic connection throttling/limiting
There is the overflow mode, I haven't submitted the new code yet, but 
you can already limit the amount of connections per real server. My 
2.4.x IPVS patches regarding the overflow mode are done (in case you 
want to test), since we still deploy mostly 2.4.x kernels.
Regarding throttling, I reckon you use the TC framework available in the 
Linux kernel since long before netfilter hit the tables.
, plus a more closely integrated and maintained health checking system.
How would you improve it? Suggestions are always welcome.

And source-routing support. ;)
Which you did :).

There are commercial products available that implement heavy-duty DoS/intrusion protection.
True. And have you looked at how they do it? Traffic contracts and the 
sorts, like TC for example.
They block the vast majority of simple attacks and are crucial for any large-scale public-facing services.
Blocking or dropping is not acceptable, diverting or migrating is. The 
biggest issue on large-scale web services according to my experience is 
the detection of malicious requests.
But a good distributed full-handshake or even valid HTTP request DoS is almost impossible to fully block.
Agreed ;).

Best regards,
Roberto Nibali, ratz
--
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc




<Prev in Thread] Current Thread [Next in Thread>