On Mon, Jun 20, 2005 at 05:10:27PM +1000, Dave Whitla wrote:
Having read through the list archives looking for a solution to my
problem, it seems as though variations of this request have been posted
a few times over the last few years without success. However the
posters expressed their problem in rather unclear terms. I will attempt
to describe it more clearly.
I am trying to load balance to two "real" servers which are actually
listening on virtual IPs on the load-balancing host. Why would I want to
do this?
To build a test environment for a web application which usually runs on
an IPVS cluster. The purpose of the test environment is to test for
database/cache contention issues before we deploy to our production
cluster. The catch is I must make the test environment (lvs director + 2
x application server instances) run on one physical host (each
developer's development machine).
The man page for ipvsadm makes specific mention of forwarding to real
servers which are in fact running on local interfaces stating that the
load balancing "forward" mechanism specified for a virtual service is
completely ignored if the kernel detects that a real server's IP is
actually a local interface. The "Local Node" page describes a
configuration in which I could load balance between a "real" server
running on the director's virtual service IP and a real server running
on another host. This does not solve my problem however as I must bind
each instance of my application to a different IP address on the same
physical box.
You may be thinking "Why not run the two instances on different ports on
the same IP (the virtual service IP)?". Sadly the application is not a
simple web-site, and source code and deployment container dependencies
on certain port numbers exist. eg RMI-IIOP listeners.
Does anyone know of some config or kernel hack, patch or whatever which
might make my ipvs present forwarded packets to the virtual interfaces
as though they had appeared on the wire so that my forward directives
are not ignored and the packets are not simply presented to the TCP
stack for the virtual service IP? I guess this is like NAT to local
destination addresses (as opposed to NAT of locally originated
connections which is supported in the kernel).
Hi Dave,
this is a pretty interesting problem that crops up all the time.
I have often wondered how hard it would be to make nat work locally
(not that Tun and Route don't/can't support portmaping anyway).
The patch below for 2.6.12 is a hack that allows nat to work locally by:
1. Not marking local real-servers as local
2. Passing nat-mangled packets down to local processes
instead of pussing them out onto the network
3. Reversing nat in the LOCAL_IN chain
Please note that this completely breaks NAT for non-Local hosts.
It could be cleaned up and made so it doesn't do that. But I
thought I'd throw it onto the list before putting any more time
into it.
------------------------------------------------------------------------
diff --git a/net/ipv4/ipvs/ip_vs_core.c b/net/ipv4/ipvs/ip_vs_core.c
--- a/net/ipv4/ipvs/ip_vs_core.c
+++ b/net/ipv4/ipvs/ip_vs_core.c
@@ -1080,7 +1080,7 @@ static struct nf_hook_ops ip_vs_out_ops
.hook = ip_vs_out,
.owner = THIS_MODULE,
.pf = PF_INET,
- .hooknum = NF_IP_FORWARD,
+ .hooknum = NF_IP_LOCAL_OUT,
.priority = 100,
};
diff --git a/net/ipv4/ipvs/ip_vs_ctl.c b/net/ipv4/ipvs/ip_vs_ctl.c
--- a/net/ipv4/ipvs/ip_vs_ctl.c
+++ b/net/ipv4/ipvs/ip_vs_ctl.c
@@ -670,12 +670,6 @@ __ip_vs_update_dest(struct ip_vs_service
atomic_set(&dest->weight, udest->weight);
conn_flags = udest->conn_flags | IP_VS_CONN_F_INACTIVE;
- /* check if local node and update the flags */
- if (inet_addr_type(udest->addr) == RTN_LOCAL) {
- conn_flags = (conn_flags & ~IP_VS_CONN_F_FWD_MASK)
- | IP_VS_CONN_F_LOCALNODE;
- }
-
/* set the IP_VS_CONN_F_NOOUTPUT flag if not masquerading/NAT */
if ((conn_flags & IP_VS_CONN_F_FWD_MASK) != 0) {
conn_flags |= IP_VS_CONN_F_NOOUTPUT;
diff --git a/net/ipv4/ipvs/ip_vs_xmit.c b/net/ipv4/ipvs/ip_vs_xmit.c
--- a/net/ipv4/ipvs/ip_vs_xmit.c
+++ b/net/ipv4/ipvs/ip_vs_xmit.c
@@ -260,10 +260,6 @@ ip_vs_nat_xmit(struct sk_buff *skb, stru
if (skb_cow(skb, rt->u.dst.dev->hard_header_len))
goto tx_error_put;
- /* drop old route */
- dst_release(skb->dst);
- skb->dst = &rt->u.dst;
-
/* mangle the packet */
if (pp->dnat_handler && !pp->dnat_handler(&skb, pp, cp))
goto tx_error;
@@ -279,10 +275,8 @@ ip_vs_nat_xmit(struct sk_buff *skb, stru
/* Another hack: avoid icmp_send in ip_fragment */
skb->local_df = 1;
- IP_VS_XMIT(skb, rt);
-
LeaveFunction(10);
- return NF_STOLEN;
+ return NF_ACCEPT;
tx_error_icmp:
dst_link_failure(skb);
