|
Hi julian,
Fisrt of all excuse me for the very long delay :) I am just back to work
and to LVS.
> I'm reading your pdf, first some questions:
>
>- in some setups we must consider how we can send the outgoing traffic
>usually according to the source addresses. I.e. when you get WANs from
>one ISP, how many subnets you receive. Same subnet or different subnet
>for each WAN?
Well, my first thought was just about using VRRP to add VIP redundance.
I need this functionnality for my job so I just wanted advices and point
of view to start implementation and to give my work back to the community.
As we said in different anteriors post, the only way to use VRRP on a
director
(LVS or other) owning multiple NICs is to run a VRRP instance per NICs
since
VRRP can be view as a routing protocol & synchronization between VRRP
instances
must be handled using part of VRRP protocol (synchronization using high
level
priority advert).
Reading this mail I can see that you see very ahead :) which is very nice !
So I can try to give my point of view on certain part.
First of all concerning WAN subnets : In a very common utilization, we are
working in the same subnet. But for ISP or operator utilization, I agree
with
you we can receive many subnets. But considering this last case, can we map
those
subnets and routings directly into a "global gateway router" to be focused
only onto
the redundance functionnality ?
>- when we have different public nets for each WAN can we send traffic
>with source addresses from WAN1 through WAN2? If not, we don't have
>many options for balancing the lines: we can balance only NAT-ed
>hosts (see my example from yesterday on the mailing list). No place
>for VRRP here. May be only if we want the internal host to use
>another path to the NAT box on eth failure. In this posting you can
>see that only by using routing on the NAT box one can use many WANs
>for the NAT-ed traffic by using multipath (yes, you can saturate
>them :)). Then what happens: the NAT box dies and our two WANs fail.
>So, we have to use another gateway (NAT box) with other two WANs.
>We all agree that the multipath routing plays nice role for the
>NAT-ed hosts - you can use many WANs, no matter they provide you
>with different public nets or with same nets. You hardcode them in
>the default routes for the NAT-ed network and you usually put source
>routing for the different nets. If you want to send traffic with
>public IPs you can only use the available (not failed) WANs for
>these IPs. In our case, when one WAN fails nobody can replace it.
>
> If only one WAN fails and you are lucky the driver to change
>the route DEAD flag, than the multipath can stop to create new routes
>to this WAN while it is down. So, the NAT-ed host notice only the
>broken existing routes through this WAN but any new connections can
>be routed through the available devices. OK, we still serve the NAT-ed
>hosts. But wait, there are non-NAT-ed hosts, i.e. other public
>hosts (this is the reason you to get /27 subnet for example, not one IP).
>Such hosts can usually use the different WANs (indirectly) as default
>gateways. Then they will select source address from a non-failed
>public net. For example:
>
>
> WAN1 WAN2 WAN3
> \ / | INT3
> GW1 GW2 |
>--------+---------------+---------------+----------------LAN
>
>
>
>GW1 - gateway for WAN1 and WAN2
> - WAN1 serves publicnet 10.1.0.0/24 which is placed on eth
> - WAN2 serves publicnet 10.2.0.0/24 which is placed on eth
> - have public address from 10.3.0.0/24 via 10.3.0.1 on eth
>
>GW2 - gateway for WAN3
> - WAN3 serves publicnet 10.3.0.0/24 which is placed on eth
> - have public address from 10.1.0.0/24 via 10.1.0.1 on eth
> - have public address from 10.2.0.0/24 via 10.2.0.1 on eth
>
>INT3 - internal NAT-ed box that uses all WANs
> - needs two private networks to use source routing to the
> two gateways
>
>
> The NAT-ed clients don't care which WAN is used and they can
>use any public net (they don't know about it). If one WAN fails, the
>others are used. The main questions in this setup:
>
>- how GW2 and INT3 can notice that WAN1 is failed?
>
> One of the solutions is to put 10.1.0.1 (all use this IP
>as gateway for packets from 10.1.0.0/24 to world) on the WAN1 device.
>Yes, not at the usual eth device. In Linux we can put local IP on any
>device but now we have a reason to put it on WAN1. If the net driver
>changes IFF_UP GW2 and INT3 will notice that the ARP probes for
>10.1.0.1 are not replied. It is possible to check this IP more often
>and not to rely on the ARP timers. GW2 and INT3 can now mark the path
>to their gateway as broken. This is already implemented in all end hosts.
>So, even with static routes the internal hosts can accommodate to the
>routing changes. The kernel marks them and does not use them.
>
> Both the GW1 and GW2 have 3 default routes, each of them
>can originate traffic to the world when there is at least one WAN
>working. The admin can choose the order. Of course, the traffic
>with specific source can fail if there is no available WAN for these
>public sources.
>
> It seems VRRP can't help here. There are no IPs that move
>from one host to another because you can't send one traffic through
>another device. If you put a master and backup device between the
>internal hosts and the real gateways you can then use VRRP to provide
>single gateway IP for the NAT-ed hosts but you introduce SPOF. The
>question is whether we need such device. See below.
>
> What we can notice here is that "we assume" the lines are
>balanced but this highly depends on the traffic kind. Multipath
>works better when there are more flows aggregated in this path but
>we rely on the chaos.
>
> The real problems come when we use these lines for servers,
>i.e. not for NAT-ed clients. In such case the remote clients select
>the public IP they connect to, so the admins have to differentiate
>the traffic by DNS names and services. It is hard to proportionally
>load the links in such case.
>
> So, for the multipath balancing. What we know for multipath,
>bonding, teql and LVS:
>
>- multipath forwards traffic by scheduling paths to different
>nexthops. Cons: many connections from one path use only one nexthop
>
>- bonding and teql can reorder packets. Cons: requires/assumes support
>on both ends. Any ARP problems with bonding and when using hubs?
>
>- LVS can route traffic for next hops
>
>- LVS forwards only traffic belonging to virtual service or related
>to it
>
> So, can we use LVS as intermediate router to provide one
>gateway IP to the internal hosts and to balance the outgoing traffic
>through different links or gateways? It is already known that LVS
>can be used in Direct Routing setups where the real servers are
>the transparent proxy servers. This works by delivering locally
>the outgoing web traffic and balancing it through the real servers.
>Now the questions is: can we do it for most of the used traffic:
>
>- TCP/UDP (virtual service with port 0?)
>
>- the related ICMP traffic already follows the TCP/UDP traffic
>
>- what remains is the outgoing ICMP
>
>- Stupid WLC scheduler or complex tools that tune WRR
>
>Cons: we waste memory for keeping the info for each connection - this
>is the way LVS is working
>
>Pros: per-"connection"
>
> The result (if such balancing can work) is that a LVS box
>can be used to balance traffic from internal hosts through many
>gateways per-connection. One can put VRRP too. The benefits can
>be in providing higher layer metrics about the balancing: how much
>the lines are loaded with traffic, etc. We can at least try to
>balance better than multipath. Of course, proper routing should
>be used to deliver all outgoing traffic locally, for example, the
>order of ip rules can be:
>
>- prio 0: local routes (already hardcoded)
>
>- prio X: link routes (by destination)
>
>- prio Y: gatewayed routes (by source)
>
>- prio Z: everything from device XXX and IPs YYY deliver locally
>and feed LVS with it. LVS should forward without change any traffic
>that can't be balanced
>
>- last prio: our default routes
>
>Some pros:
>
>- if you can't use multipath for non-gatewayed routes, you can do
>it with LVS-DR and the proposed feature
>
>- you can tune your cluster software to stop using one gateway (WAN) if
>the web page of your ISP does not work :)
>
>
>Example setup:
>
>
> GW1 GW2 LVSIN1 LVSOUT1 INT1 INT2
> | | | | | |
> --------+-------+-------+-------+---------------+-------+---
>
>LVSIN1: for out->in traffic, using INT1 and INT2 as real servers
>
>LVSOUT1: for in->out traffic, using GW1 and GW2 as real servers
>
>The situation:
>
>- the requests come according to the client action
>
>- the replies are balanced in the best way: they are sent where possible
>and we hope the links are proportionally loaded. If we have one publicnet
>per WAN, then we don't need LVS to balance server-oriented traffic - there
>are no options to send one packet to more than one place.
>
>
> Another view:
>
> ,--in1---> LVSIN1 --------> INT 1, same for INT 2
> GW1 / <-out1--- LVSOUT1 <--------'
> GW2 <---out2----'
>
>[I really can't draw]
>
> The SNAT function, if used, can be performed on the GW hosts
>or on the LVS box (LVSIN1==LVSOUT1)
>
>So, gurus, what problems you see in trying to balance outgoing traffic
>by using LVS? Pros, Cons, what is possible and what is not? Possible
>good and bad setups?
>
> We assume the out->in traffic reaches the internal hosts
>directly from the real servers (the real/intermediate gateways) - DR
>method. Of course, may be we can SNAT the traffic too. Then the
>replies will return.
>
>For now, I see that we must check all ICMP packets and to try to
>forward them after creating connections structure (same as the MASQ
>code). May be I'm missing something that breaks such idea (I'm already
>sleeping).
===============
All your points are very interresting... considering outgoing traffic
loadbalancing,
we are out of the VRRP scope... we must consider routings setups &
scheduling.
for security setups SNAT can be use but DR can be much more efficient
decreasing the director
load.
The best point of the setup you describe is to be able of selecting
outgoing interface using
LVS keys... To answer to this I really need to setup a small architecture
to try it :)
will schedule work on this :)
Regards,
Alexandre
|
