LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] vip-to-vip connection between two sets of realservers wi

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] vip-to-vip connection between two sets of realservers with one director?
From: "Fred Clift" <fclift@xxxxxxxxx>
Date: Sun, 02 May 2010 21:38:36 -0600
Gary Smith wrote:
>> All,
>>
>>
>> Lets say hypothetically I have a director with two vips.  The vips
>> represent different services, different areas of responsibility, etc.
(snipped explanation of two realservers communicating via their VIPs)
> Use different subnets for the different classes of real servers.
>
> Ex:
>
> Data rail: 10.0.1.0/24
> Web rail: 10.0.2.0/24
>
> Data server:
>       * IP 10.0.1.2/24
>
> Web server:
>       * IP 10.0.2.2/24
>
> Director:  
>       * IP 10.0.1.1/24
>       * IP 10.0.2.1/24
>       * VIP 10.0.3.10/24 Data
>       * VIP 10.0.3.11/24 Web (or the public IP if ipvs is the firewall as 
> well)
>
> ipvsadm -A t 10.0.3.10:3306 -s wlc
> ipvsadm -a t 10.0.3.10:3306 -r 10.0.1.2:3306 -m -we 100
> ipvsadm -A t 10.0.3.11:80  -s wlc
> ipvsadm -a t 10.0.3.11:80 -r 10.0.2.2:80 -m -we 100
>
>
> no need to nat/snat at this point.
>
>
>   
>> So, does my question make sense?  I would like realservers for one vip
>> to make connections to the vip of another virtual server on the same
>> director.  Anyone know how?
>>     






Ok - so I've had a look at this solution, and tried something like 
this.  It may be that the solution could be made to work but it depends 
on having a different internal 'routing' vip per set of servers.  For 
one or three sets this would probably be ok.

I'm trying to develop a commercial offering based on lvs in which 
different customers are assigned one or more realservers behind the 
director.  I can't predict ahead of time what customers will buy, and I 
can't renumber on a whim (though to some extent the private IPs behind 
the NAT are somewhat hidden...).  The realservers are actually virtual 
servers on fairly beefy hardware  and I can end up with many many 
(virtual) realservers and many 10s of vips.

It turns out the performance is acceptable for such a solution according 
to the load tests we've done, but this vip-to-vip communication is 
somewhat of a deal-breaker for us.  I've got a few ugly workarounds, but 
nothing I'd like to put real services on.  Since I dont have direct 
control over the customer's use of the services, and the customers may 
want to communicate with each other and may not even be aware that 
whoever they're sending email to, etc are on another one of our 
realservers they rightly expect that they should just be able to reach 
the IP of someone else on the internet...

The problem appears to be outbound LVS routing.  It doesn't follow the 
normal iptables paths as far as I can tell.

Let me see if I can give you some more detail:

I have two real servers A and B.  A has a VIP of 200.0.0.2 and a private 
IP of 10.0.0.2,  B has 200.0.0.3 and 10.0.0.3.  The director has a 
private vip of 10.0.0.1 used for outbound routing.

If A tries to connect to B, packets arrive at the directory with a 
source IP of 10.0.0.2 and a dest IP of 200.0.0.3.  LVS changes that 
destination to 10.0.0.3.  At this point using symmetric dnat/snat rules, 
then iptables fires the snat/postrouting rules and changes the source 
IP, but LVS doesn't dot this.  The packet is then dropped back out on 
the wire on the private interface (vlan in this case...).  B gets a 
packet addressed to 10.0.0.3 from 10.0.0.2 (which should be 200.0.0.2), 
responds, and A sees the response hit the wire because its on the same 
lan, the director does nothing with the packet because it thinks it 
doesn't need to.  

At any rate, I've tried many many configurations and iptables rulesets 
to try and get this to work right.  One thing that might affect this is 
that I'm using one physical lan and a couple of vlans for this 
configuration.  If I used two physical interfaces (one for public, one 
for private) perhaps this would work.

In short, do you have any other suggestions I might try?

Fred


This email message is intended for the use of the person to whom it has been 
sent, and may contain information that is confidential or legally protected. If 
you are not the intended recipient or have received this message in error, you 
are not authorized to copy, distribute, or otherwise use this message or its 
attachments. Please notify the sender immediately by return e-mail and 
permanently delete this message and any attachments. Verio, Inc. makes no 
warranty that this email is error or virus free.  Thank you.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>