|
Ken,
I reckon people have different setups and thus our views seem to differ
a bit. Without proper numbers and test conduct configurations that are
repeatable by other persons, we'll probably compare apples to oranges.
That said, I should nevertheless like to comment on this based on my
limited experience with web services.
This 150 connection limit is the default MaxClients setting in Apache,
which in practice should be adjusted as high as you can without Apache
using more memory than you want it to (e.g., 80-100% of available RAM --
no need for the DoS to swap-kill your box, too). Each Apache process
will use several megabytes (higher or lower depending on 32- or 64-bit
platforms, add-on modules, etc) so this can't be set too high.
Most of our customer's httpd show RSS between 800KB and 2MB; some of
them it's including mod_perl or mod_php.
Disabling KeepAlives will drop your working process count by roughly an
order of magnitude, and unless you're just serving static content it's
generally worth disabling. But for your case of 150 idle connections,
it doesn't help.
Or you set your timeouts correctly, or you implement proper state
mapping using a reverse proxy and a cookie engine.
Netfilter has various matching modules that can limit connections from
and/or to specific IPs, for example:
iptables --table filter --append VIP_ACCEPT --match dstlimit --dstlimit
666 --dstlimit-mode srcipdstip-dstport --dstlimit-name VIP_LIMIT --jump
ACCEPT
No wonder you have no memory left on your box :).
The reason DoS attacks are so successful (especially full-handshake
attacks) is that something needs to be able to cache and handle incoming
connections. And that is exactly where Apache is weakest -- the process
model is terrible at handling a high number of simultaneous, quasi-idle
connections.
100% agreed.
LVS has some DoS prevention settings which you should consider
(drop_entry, drop_packet, secure_tcp) but they're generally only useful
for SYN floods.
I would not call dropping a certain amount of illegitimate _and_
legitimate connections to be too useful when you're running on a strict
SLA. QoS approaches based on window scaling help a bit more.
A full handshake will be passed on through LVS to the
application, and that is where the resources must be available. And
given persistence, a single-IP attack will be extremely effective if you
only have one (or few) real servers.
Yep.
Once a connection has been made to Apache, it will need to either
relegate idle connections out of process (see Apache 2.2's new event
MPM, not sure if it only works on idle keepalives) or limit based on IP
with the modules you mention.
This problem is difficult to solve completely, and I agree that solving
it in Apache is the least powerful, least convenient, and highest
overhead solution. Given Netfilter functionality (2.6 and later), the
absence of throttles or connection limits in LVS isn't fatal. But I do
feel that LVS could be made a more comprehensive system if it rolled in
even basic connection throttling/limiting
There is the overflow mode, I haven't submitted the new code yet, but
you can already limit the amount of connections per real server. My
2.4.x IPVS patches regarding the overflow mode are done (in case you
want to test), since we still deploy mostly 2.4.x kernels.
Regarding throttling, I reckon you use the TC framework available in the
Linux kernel since long before netfilter hit the tables.
, plus a more closely
integrated and maintained health checking system.
How would you improve it? Suggestions are always welcome.
And source-routing
support. ;)
Which you did :).
There are commercial products available that implement heavy-duty
DoS/intrusion protection.
True. And have you looked at how they do it? Traffic contracts and the
sorts, like TC for example.
They block the vast majority of simple
attacks and are crucial for any large-scale public-facing services.
Blocking or dropping is not acceptable, diverting or migrating is. The
biggest issue on large-scale web services according to my experience is
the detection of malicious requests.
But
a good distributed full-handshake or even valid HTTP request DoS is
almost impossible to fully block.
Agreed ;).
Best regards,
Roberto Nibali, ratz
--
echo
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc
|
