|
Hello,
On Fri, 17 Nov 2000, Anush Elangovan wrote:
> But, I guess the current scheme stops the connection right at the lvs
> director rarther than sending them to the real server and then reseting it
> incase of it being suspected as a SYN flood. The advantage here is that you
> save per connection processing time as the connection is initially sent thru
> and then RST if it is a SYN flood.
Yes, drop_packet drops packets without sending them to the real
servers but this is not the case with drop_entry. Currently, there are
cases in LVS where the real server is informed about the missing entry
using ICMP.
> > 1. SYN comes
> > 2. You defer this packet before analyzing the source address or pass it
> > to a real server?
> > 2.1 If you defer this packet, then you remember so many fields from the
> > TCP packet. And for long time (as part of the connection entry may be).
> > 2.2 You pass it => the real server start its handshaking.
> >
>
> I would pass it on to the real server, and would be assigned an initial
> priority (source IP check could be done here). Let us say the "priority
> changing" mechanism could be a separate process (can even be a separate
> process in userland, working fulltime just prioritizing connections and
> changing their priority). All the ip_vs has to do is trigger the
> connection priority based drop mechanism once system capacity is going to be
> reached. (this could depend on a similar calculation using amemthresh ). This
> mechanism needs to just send RST packets to the real servers.
drop_entry works in the same way. The only difference is that
the reset to the real server is ICMP_PORT_UNREACH instead of TCP RST,
i.e. the initial information from the SYN packet is not known for LVS.
The other difference is that this works for any state and even not
under attack. But this ICMP message is well accepted from the real
server if it is in SYN_* state.
> > So, the main question is: do you want to guard the real servers
> > or you prefer to keep the LVS director not busy, just like the current
> > LVS defense strategies?
> >
>
> I am interested in total system availablity to legitimate users, this means I
> should guard the real servers. Since I RST the connection on the real server,
> the corresponding entry in the Director is also removed, the director will
> also be available to service more legitimate connections.
You have to check the standards what information should contain
this RST packet. I'm not sure if you can generate the correct values
in the TCP RST packet (seq numbers, etc.). IMO, the ICMP message to the
real servers reaches the same goal without any problems. It can't work
immediately for already established connections (the error report is very
slow, usually 2 minutes, security considerations) but this is not true
for the attacker's connections, we assume that they are not in established
state.
> > These method guarantee to keep the load in the real servers low
> > but why they are not perfect: there is no known method to distinguish
> > the real clients from the spoofed requests. If you take the role to
>
> Yes, I understand that there is no way to distinguish the real client, but do
> you think this priority based system could be a step in the right direction.
Your ICMP checks in the director can help only to detect
valid clients for real servers that don't implement SYN cookies. But I'm
not sure about the effect if you pass all SYNs to the real server. These
packets can be dropped from the real server without notifying the director
to terminate the checks. These packets can be for valid connections but the
director doesn't know this, the real server can keep only 128 SYNs in
the incoming queue. The director will learn whether one connection is valid
at the same time when the director learns it using ICMP. What timeout you
will select for receiving the ICMP reply? Shorter than the real server's
timeout for the client's ACK? You can alter ipv4/tcp_retries1 or any
analog networking setting to acheive the same effect but for the real
server. The difference is that the director tries to check all incoming
connections using ICMP but the real server checks only those that have
chance to be established, i.e. that are not dropped after exceeding the
incoming queue. Your can replace the ICMP checks just by setting
tcp_retries1 to 2 in the real servers, for example. This source checking
will generate lower number of replies.
The other variant is not to pass the incoming SYNs to the
real servers before the validation but this delays the handshake
under attack. When the ICMP reply is received you can pass the next
SYN packet. This can work only for clients that don't block the ICMP
echo messages.
> > validate the incoming requests you have to defer/drop the incoming
> > SYN packet after allocating memory for storing the initial information
> > while the connection is validated. You are moving the handshake from
> > the real servers to the director but we don't know what happens while
> > this is performed for the LVS/DR and LVS/TUN methods where the outgoing
> > traffic does not come to the director. The users can't remove the
>
> Though I didnot have in mind any idea about defering the packet in the
> director, I guess it could be used in VS-NAT, and in VS-DR where all the real
> server and director are in the same switched ethernet segment (if we can have
> some code to sniff out all the outgoing packets). But this is not my aim here.
For NAT you can rely on secure_tcp. There is a short timeout
for the SYN state that can help the connections that are not replied
from the real server (without SYN cookies support) to be dropped very
soon. Once they are dropped if the real server continues to retransmit
the SYN+ACK packets it is informed using ICMP about the missing entry.
> I would like to forward the connection immediately to the real
> server, while the validation is done. The connection is allowed to continue
> normally (until we find that the whole cluster is going to reach full capacity
> this could also be accounted for in the number of connections. Say, each real
> server can support x number of connections and we have 3 real servers, so
> when we are reaching like 98% of 3x we could start resetting lower priority
> connections. My point here is that we donot need to track the replies of the
> real server in DR/TUN. The initial SYN packet is what we mostly use to
> prioritize the connection. We will also use a little history here, like how
> many concurrent connections from one IP etc. The following packets like ACK
> etc will be used to adjust the connection priority.
Hm, I don't see anything better than the drop_entry. Both the
drop_entry and your proposal do:
- not guard the real servers from attack. IMO, it is too late to
inform the real server "RS, the ICMP timeout expired please forget
about this connection". This can work only for real servers without
SYN cookies support and without the ability to control the SYN_RECV
timeout. The real server is already too busy to drop even valid
requests. I agree that for some real servers the SYN_RECV states
can be reduced with the RST packet (if it can be generated) but
I'm not sure for the effect of the ICMP packets and the extra TCP
RST packets to the real server (for connections already dropped
from the real server).
- use priorities considering the TCP state
> > changing the states. But we move the load of these checks from the
> > moment of the attack to the packet handling code (i.e. even when there
> > is no attack). But smarter spoofing programs can fool the director
> > in DR and TUN mode by sending ACK packet after the initial SYN packet.
>
> Yes it is possible to fool the system, but it will be of no use unless the
> attacker knows that this is a cluster, and then finds out it is an LVS
> cluster. But if he doesnt, our system becomes more useful. If he does, his
> attack is worth only half as he needs to send an double the packets i.e. the
> SYN and then the ACK, which makes it harder as he needs double the number of
> machines to launch an attack of similar impact.
The masq box is easy to fool. You just need to send dummy packets
with ACK bit. Build a pool with 1,000,000 addresses and you can start
to generate SYN and then ACK packets to the victim.
> > And we enter established state where the discussed priority is raised
> > and we assume this connection is valid while it is not. So, there is
> > a good reason the number of incoming packets to be involved in the
> > checks.
> Any suggestions on how to change the priority based on number of incoming
> packets.
There is a counter for the incoming packets, it is used from
the drop_entry strategy. You have to look at todrop_entry(): ms->in_pkts
> I almost forgot to mention another advantage of the proposed system, apart
> from protection from DOS it could be used to prioritize connections from
> particular subnets.
>
>
> Say Company A has an LVS cluster and it wants its staregic partner company B
> to be given priority access to the cluster, we could tell the cluster
> directory to give say priorty 0 to connections coming from the subnet company
> B has. This could also be used to give them access to say a dedicated real
> server/ real servers for their purpose. For example yahoo, has something
> similar where, a company's employees can access that company's specfic
> internal information via my.yahoo.com.
Hitting a real server with different name is easy. If there is
only one site name something like this may work:
ip rule add prio 100 from B table 100
ip route add table 100 A via dedicated_real_server dev eth0
IMO, it is difficult to guard the real servers. You can rate
limit the incoming SYNs or just to rely on the real server's own
defense.
There are some smarter attacks that even don't reach the director.
You can't do anything. The effect is the same: the link before your
router/LVS box is full with invalid requests and the LVS even does not
know about this. Of course, the LVS receives only the valid requests
but the service is very slow.
> Thanks again.
>
> Anush.
Hm, it is interesting to see what the other people think about
the proposed strategy.
Regards
--
Julian Anastasov <ja@xxxxxx>
|
