LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] Outgoing TCP from real servers using VIP as the source i

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] Outgoing TCP from real servers using VIP as the source in DR balancing mode
Cc: Julian Anastasov <ja@xxxxxx>
From: Dmitry Akindinov <dimak@xxxxxxxxxxx>
Date: Fri, 06 Jan 2012 14:02:05 +0400
Hello,

On 2012-01-06 12:23, Julian Anastasov wrote:
>
>       Hello,
>
> On Fri, 6 Jan 2012, Dmitry Akindinov wrote:
>
>> Hello,
>>
>> We have met the following problem with "reverse balancing". There is a
>> set of servers (let's say 10), and a load-balancer (for which we would
>> like to use Linux with the ipvs module). The balancer works in the
>> Direct Response (DR) mode, so all packets coming to the "standard" ports
>> (such as 25, 80, 110, 143) are redirected to running "real-severs" by
>> changing the packet destination MAC, but leaving the IP packet itself
>> intact.
>>
>> Each real server has a lo:x virtual interface with the VIP address
>> configured. This interface does not answer any arp request, so all
>> packets with VIP as IP destination hit the load balancer first.
>> Each real server has its own "gray" address 10.10.10.x - used for
>> pinging and used to retrieve a MAC to redirect the incoming packets to.
>>
>> It is the standard DR setup, repeated here just for the clarity.
>>
>> Now, these servers have to make outgoing TCP connections, too. And the
>> application requirement is that these outgoing connections are made
>> using the same VIP as the source address.
>>
>> The main idea is to implement the "Direct Client Request" - something as
>> the DR method, but inverted.
>>
>> a) each server has its own unique port range assigned (16000-16999 for
>> the server #0, 17000-17999 for the server #1, etc.), and all outgoing
>> connections are made using an available port from that range.
>>
>> b) TCP response packets will go to the VIP address, to the port that was
>> specified as the source port on the initiating server.
>> These packets will hit the load balancer first, and we need it to relay
>> them - WITHOUT modification - to the proper real server, in the same way
>> it does when it implements the DR method for incoming connections. It
>> must  redirect all packets coming to VIP address, port=16000-16999 to
>> the server #0, port 17000-17999 to the server #1, etc.
>
>       What about using 10 match rules in mangle/PREROUTING
> to mark every port range. Add 10 fwmark-based virtual servers
> containing single real server again in DR mode. There is a
> problem if this download traffic overloads the single
> IPVS box.

We tried exactly that on a CentOS 6 install. It LOOKS like IPVS alreays 
builds a 'session' to relay tcp traffic, even if there is only 1 "real" 
server in the "virtual server set". And it LOOKS like it builds sessions 
only when it receives the initial SYN packet. In our case, no SYN packet 
goes via the IPVS server, the first packet for a new connection is SYN 
ACK. It looks like IPVS does not build a session for it, and drops it 
and all other packets relayed to it due to lack of session.

IF IPVS could work in a stateless mode, dispatching connection using any 
algorithm based on packet source/destination (we do not care which one, 
as we would have only one real server for each of these virtual servers) 
- then it would work.

Alternatively, if a Virtual server configured with just one real server 
would redirect ALL packets to it, without building sessions - that would 
work, too.

>> c) in order to implement this, we tried to play with the iptables TPROXY
>> module, but to no avail: the server RIP opens an outgoing TCP connection
>> to some server X, port RPORT and we want to redirect all packets coming
>> FROM RPORT to our server RIP, w/o modifying the IP packet:
>>
>> *mangle
>> :PREROUTING ACCEPT [1969:219531]
>> :INPUT ACCEPT [1777:151627]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [882:697245]
>> :POSTROUTING ACCEPT [882:697245]
>> -A PREROUTING -p tcp -m tcp --sport RPORT -j TPROXY --on-port 0 --on-ip
>> RIP --tproxy-mark 0x0/0x0
>> COMMIT
>>
>> No packet is being relayed, and it seems like TPROXY works for
>> redirecting traffic to an internal socket only.
>>
>> Is there any way to redirect a packet without modification to a
>> different server (substituting the taget MAC),
>> using just the iptables module? Or is there a way to manage LVS director
>> connection tables for this purpose: directing the tcp connection
>> response packets to the proper real server?
>
>       Another option is to use ip rules with fwmark
> matching to forward the traffic. New kernels support the
> option to override the priority 0 for table local.

How that can be configured?

> You can change the priority and to put your fwmark
> rules before the rule for table local. Add 10 routing
> tables containing single route to every real server.
> Not sure what happens with related ICMP in this case.
> That is why the virtual servers is a better option.

Thank you very much!

-- 
Best regards,
Dmitry Akindinov

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>