Re: how to support transparent cache cluster in ipvs?

To: " users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: how to support transparent cache cluster in ipvs?
Cc: Jinhua Luo <home_king@xxxxxxx>
From: Horms <horms@xxxxxxxxxxxx>
Date: Tue, 28 Nov 2006 15:19:30 +0900

this patch seems pretty nice to me, and it seems that it should work
quite well. Have you tested it? If so, could you provide a signed-off-by
line, as described in section 5 of
so that I can submit it to netdev for inclusion in the kernel.

I have reformated the patch a bit, it is below. Feel free
to rework the comment if you like.


[IPVS] transparent proxying

Patch from home_king <home_king@xxxxxxx> to allow a web cluseter using
transparent proxying. It works by simply grabing packets that have the
fwmark set and have not already been processed by ipvs (ip_vs_out) and
throwing them into ip_vs_in.


Normally LVS packets are processed by ip_vs_in fron on the INPUT chain,
and packets that are processed in this way never show up on the FORWARD
chain, so they won't hit this rule.

This patch seems like a good precursor to moving LVS permanantly to
the FORWARD chain. As I'm struggling to think how it could break things.

Reformated to use tabs for indentation (instead of 4 spaces)
Reformated to be < 80 columns wide

Cc: Jinhua Luo <home_king@xxxxxxx>
Signed-off-by: Simon Horman <horms@xxxxxxxxxxxx>

diff --git a/net/ipv4/ipvs/ip_vs_core.c b/net/ipv4/ipvs/ip_vs_core.c
index 1445bb4..5038386 100644
--- a/net/ipv4/ipvs/ip_vs_core.c
+++ b/net/ipv4/ipvs/ip_vs_core.c
@@ -23,7 +23,9 @@
  * Changes:
  *     Paul `Rusty' Russell            properly handle non-linear skbs
  *     Harald Welte                    don't use nfcache
- *
+ *     Jinhua Luo                      redirect packets with fwmark on 
+ *                                     NF_IP_FORWARD chain to ip_vs_in(), 
+ *                                     mainly for transparent cache cluster
 #include <linux/module.h>
@@ -1070,6 +1072,17 @@ ip_vs_forward_icmp(unsigned int hooknum,
        return ip_vs_in_icmp(pskb, &r, hooknum);
+static unsigned int
+ip_vs_forward_with_fwmark(unsigned int hooknum, struct sk_buff **pskb,
+                         const struct net_device *in,
+                         const struct net_device *out,
+                         int (*okfn)(struct sk_buff *))
+       if ((*pskb)->ipvs_property || ! (*pskb)->nfmark)
+               return NF_ACCEPT;
+       return ip_vs_in(hooknum, pskb, in, out, okfn);
 /* After packet filtering, forward packet through VS/DR, VS/TUN,
    or VS/NAT(change destination), so that filtering rules can be
@@ -1160,9 +1173,17 @@ static int __init ip_vs_init(void)
                goto cleanup_postroutingops;
+       ret = nf_register_hook(&ip_vs_forward_with_fwmark_ops);
+       if (ret < 0) {
+               IP_VS_ERR("can't register forward_with_fwmark hook.\n");
+               goto cleanup_forwardicmpops;
+       }
        IP_VS_INFO("ipvs loaded.\n");
        return ret;
+  cleanup_forwardicmpops:
+       nf_unregister_hook(&ip_vs_forward_icmp_ops);
@@ -1182,6 +1203,7 @@ static int __init ip_vs_init(void)
 static void __exit ip_vs_cleanup(void)
+       nf_unregister_hook(&ip_vs_forward_with_fwmark_ops);

Here, I redirect the packets with fwmark to ip_vs_in() on the FORWARD 
chain, and ip_vs_in() will handle the packets which are marked by 
iptables to indicate transparent cache virtual service, but ignore other 
packets (let them continue to flow on the FORWARD chain).

Now you can use ipvs to deploy TP at ease:
@ ipvs director
# sysctl -w net.ipv4.ip_forward=1
# iptables -t mangle -A FORWARD -p tcp -s <internal network> --dport 80 
-j MARK --set-mark 1
# ipvsadm -A -f 1 -s lblcr
# ipvsadm -a -f 1 -r RS1
# ipvsadm -a -f 1 -r RS2

@ RS
# iptables -t mangle -A PREROUTING -p tcp -s <internal network> --dport 
80 -j REDIRECT --to-ports 3128
# cat >> /etc/squid/squid.conf << EOF
  httpd_accel_host virtual
  httpd_accel_port 80
  httpd_accel_with_proxy on
  httpd_accel_uses_host_header on
# /etc/init.d/squid start

Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline


<Prev in Thread] Current Thread [Next in Thread>