|
I guess I was under a different impression for OPS. It does exactly
what I want it to do which is completely load balance perfectly over my
realservers. Its entirely possible in my application that all the
requests from a client could come from the same CIP:CPORT. My problem
with the timeout is that the lowest I could get the timeout was 1
second. In the telecom world for me, this is an eternity.
Going a step further maybe it is possible to use LVS-DR instead of
LVS-NAT. This would negate the need for the director to keep up with
anything as the packet would be going directly out from the realserver.
I've tried this with the SNAT rule as I've previously reported and had
problems SNATing to port 53.
My setup is 2 bonds: 1 with 2 vlans, 1 with 1 vlan
Bond0.200 (public)
Bond0.202 (private)
Bond1.201 (public and vlan DNS traffic is used on)
So I send my DNS query to my VIP on my directors. It gets routed to a
realserver which I've attached the vip to bond1.201:0. According to
others I've talked to I shouldn't need an iptables rule but I still
don't see the packet out with the source ip address of the VIP. I see
the packet with the source IP of the actual realserver. Its possible it
is a routing issue though so I plan on digging deeper on that today.
Should I need an iptables rule at all for LVS-DR?
Thanks for your insight.
Mike
-----Original Message-----
From: Julian Anastasov [mailto:ja@xxxxxx]
Sent: Tuesday, April 17, 2007 5:12 PM
To: Rudd, Michael
Cc: LinuxVirtualServer.org users mailing list.
Subject: RE: SNAT / Masquerading problems using LVS-NAT
Hello,
On Thu, 12 Apr 2007, Rudd, Michael wrote:
> After some more digging it appears this is related to the OPS or One
> Packet Scheduling feature. With the OPS feature turned off, the source
> IP address is correctly SNATed to my VIP. With the OPS feature on and
> working correctly(which we need for our UDP service), the source IP
> address isn't correctly SNATed.
OPS is implemented for setups where there is no reply for the
original packet. DNS and RADIUS have reply, so they need something
different which is not done by OPS:
- every original packet should pass scheduling step
- reply (or replies) should go back properly, the hash connection should
keep the needed information (VIP, VPORT)
So, what you need is something different. OPS now works in this
way:
- schedule original packet but don't hash the connection
- when reply packet is received it can not find connection and the reply
packet is treated as non-IPVS packet (real server receives ICMP or
packets goes further). That is why such replies don't have proper
VIP:VPORT if passed to output device.
> Is anybody aware of the code for this? I assume its related to not
> looking up the connection in the hash table anymore with OPS thus not
> SNATing. Maybe an iptables rule coudl fix this possibly?
You can use rule but may be you will not get the right VPORT
every time. Not sure why you need OPS. It was created when someone
needed to generate many requests from same CIP:CPORT with the assumption
that there are no replies. Only when many connections come from same
CIP:CPORT in the configured UDP time period the connection reusing does
not allow scheduling to be used for every packet. That is why OPS was
needed to schedule many packets (coming before expiration) from same
CIP:CPORT->VIP:VPORT to different real servers.
May be what you need from OPS is impossible: when OPS is not
used if reply is delayed IPVS will wait until the configured UDP timeout
is expired but this value can be different from the timeout your clients
is using. Difference in miliseconds can be fatal. What can happen is
that a different request from same CPORT will go to the same real server
as long as the UDP timeout is not expired. There can be different
situations:
- clients can retransmit on some timeout (DNS, RADIUS)
- nobody is instructed how many requests should be passed (and the same
number of replies if such application mode is used) before removing NAT
connection explicitly before expiration to allow next request to be
scheduled to different real server.
So, the main problem is that it is not easy to balance single
CIP to many real servers if there are replies that can be delayed or
when requests can be retransmitted. There is no way IPVS to know when to
forget one connection to allow scheduling for the next packet from same
CIP:CPORT. So, if the client expects replies then OPS should not be
used. Instead, short UDP timeout should be used and one should be ready
single CIP:CPORT to be scheduled to same real server even if many
distinct (from application point of view) requests are sent from same
socket.
> Update has anyone seen this? I am basically seeing using LVS-NAT that
> the return packets are not being SNATed with the LVS directors VIP but
> have a source IP address of the realservers. I saw this work in the
> 2.4 kernel but havent been able to make it work in 2.6. Any clues?
>
> Thanks
> Mike
Regards
--
Julian Anastasov <ja@xxxxxx>
|
