Subject: [lvs-users] CentOS 4.7 (2.6.9-based) -- LVS-NAT return packets leaving via wrong interface

From: Charles Duffy <[email-masked]>
Date: Wed, 11 Mar 2009 19:14:51 -0500

Howdy!

I have a two-interface configuration on my director, where each 
interface is on a different subnet -- an internal interface with the 
realservers, and an external one with the VIPs. Using LVS-NAT, SYN 
packets are correctly routed by the director to an appropriate 
realserver and ACKs are appropriately routed back to the director from 
the realclient (via the default gateway) -- but when the director emits 
the demasqueraded ACK to be sent to the client, it does so on the 
internal interface rather than the external one, and the router between 
the two (which I don't control) is disinclined to forward it.

I've tried to work around this using source routing, as follows:

# ip rule show
0:      from all lookup local
32764:  from <INTERNAL_NET> lookup int
32765:  from <EXTERNAL_NET> lookup ext
32766:  from all lookup main
32767:  from all lookup default
# ip route show table ext
<EXTERNAL_NET> dev eth1  scope link
default via <EXTERNAL_GW> dev eth1
# ip route show table int
<INTERNAL_NET> dev eth0  scope link
default via <INTERNAL_GW> dev eth0

As the demasqueraded packets have a source address on <EXTERNAL_NET>, I 
would expect them to leave on eth1 via <EXTERNAL_GW>. However, this does 
not happen -- the demasqueraded packet attempts to leave via eth0, and 
thus never reaches its destination.

Any hints?

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - [email-masked]
Send requests to [email-masked]
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

---

From: Joseph Mack NA3T <[email-masked]>
Date: Wed, 11 Mar 2009 18:27:08 -0700 (PDT)

On Wed, 11 Mar 2009, Charles Duffy wrote:

> internal interface rather than the external one,

This is supposed to work.

Things to look for would be

o you have an after market enhanced version of LVS. Use a 
standard kernel not a centos kernel

o you have iptables rules running.

> I've tried to work around this using source routing, as follows:

this is not the solution

Joe
-- 
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

---

From: Graeme Fowler <[email-masked]>
Date: Thu, 12 Mar 2009 10:57:27 +0000

> default via <INTERNAL_GW> dev eth0

Simple fix: have a single default route. You should only very rarely
have two defaults. If you make sure your box has a single default route
via EXTERNAL_GW then your problem will resolve itself.

Networking 101 :)

Graeme

---

From: Charles Duffy <[email-masked]>
Date: Thu, 12 Mar 2009 10:24:54 -0500

> standard kernel not a centos kernel

This was it; moving to a freshly compiled 2.6.28.7 resolved the issue.

I'll probably be looking into whether RHEL5 exhibits this issue at some 
point, and submitting a ticket to Red Hat if so.

---

From: Charles Duffy <[email-masked]>
Date: Thu, 12 Mar 2009 12:20:37 -0500

> via EXTERNAL_GW then your problem will resolve itself.

Except that my box isn't allowed to talk to anything via EXTERNAL_GW; 
packets routed out through it, except those coming from the VIPs, are 
silently dropped. I'm only allowed to talk to the outside world when 
going through <INTERNAL_GW>.

Finding out why, or getting this changed, would mean putting in a work 
order with the firewall guys and waiting a few weeks.

---

From: Charles Duffy <[email-masked]>
Date: Wed, 25 Mar 2009 16:57:53 -0500

> This was it; moving to a freshly compiled 2.6.28.7 resolved the issue.

Actually, this was incorrect: Moving to a freshly compiled 2.6.28.7 
resolved the issue *only when combined with the source routing 
configuration previously described*.

Without source routing used to shepherd demasqueraded packets to eth1 
(with a different, appropriate gateway), they still were leaving eth0 
and being discarded by the local router.