how to support transparent cache cluster in ipvs?

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: how to support transparent cache cluster in ipvs?
From: home_king <home_king@xxxxxxx>
Date: Fri, 24 Nov 2006 11:47:03 +0800
Transparent cache, taking transparent web cluster for example, seems like below topology or deployment:

Given client machines make ipvs director default gateway, http requests toward internet web sites from the clients forward through the ipvs director. Obviously, the director must be enable to capture those packets which are involved in some virtual service defined in the ipvs rule table, though they exist in FORWARD chain which go beyond the power control of ipvs, because the only outside-inside function entrance is ip_vs_in() which only hooks in INPUT chain, right? As long as ip_vs_in() handles these packets, ipvs can finally forward (VS/DR) them to the backend real servers. The real servers use the REDIRECT iptable rule to let local squid adopt them, and at last, client machines get the correct http responses.

LVS HOWTO provides two ways to meet our need, but I found they don't work.

1. define a route in local route table (in "17.3. How you use TP" chapter)
I test this way in Redhat AS4.0_update2 which uses 2.6.x kernel, but nothing magic happens, :-( Maybe 2.6.x kernel does not support this trick. Even this can work, but how can you capture all packets to the whole internet web sites in your local route table? :-)

2. REDIRECT iptable rule
Let's see how REDIRECT rule work in 2.6.x kernel first.

Given such a rule configured on ipvs director:
# iptables -t mangle -A PREROUTING -p tcp -s <internal network> --dport 80 -j REDIRECT --to-ports 3128

When a packet [cip:cport -> website_ip:80] comes in, netfilter redirects it from PREROUTING chain to INPUT chain, and it becomes [cip:cport -> localhost_ip:3128], and then it's handled by squid (with transparent proxy settings). Squid gets the webpage from cache or from internet, and send back the webpage to the client. When the response packet [localhost_ip:3128 -> cip:cport] flows at the end of POSTROUTING chain, netfilter recovers its origin, then the packet becomes [website_ip:80 -> cip:cport] and go toward the client machine on the wire.

We can see that, netfilter MASQUERADEs the packet both on PREROUTING and POSTROUTING chains. However, it is useless to ipvs director.

First, the packets which ip_vs_in() handles own the unique localhost ip as the destination ip, and that make those schedule algorithms like lblc, lblcr, dh (dedicated to cache cluster) useless!

Second, ipvs registers ip_vs_post_routing(), which hooks on POSTROUTING chain, with a priority of NF_IP_PRI_NAT_SRC-1, which means it is called by netfilter core before NAT hooks like REDIRECT. This function returns NF_STOP for those packets which ipvs_property is set, and REDIRECT has no chance to MASQUERADE those packets. Finally, client receives the response, but it dost not match the request on the orign ip:port!

I think out a solution for the support of TP in ipvs natively, and make a patch:

--- linux-2.6.16/net/ipv4/ipvs/ip_vs_core.c 2006-03-20 13:53:29.000000000 +0800 +++ 2006-11-21 18:28:35.000000000 +0800
@@ -23,6 +23,8 @@
 * Changes:
 *    Paul `Rusty' Russell        properly handle non-linear skbs
 *    Harald Welte            don't use nfcache
+ * Jinhua Luo redirect packets with fwmark on NF_IP_FORWARD chain + * to ip_vs_in(), mainly for transparent cache cluster

@@ -1060,6 +1062,16 @@
    return ip_vs_in_icmp(pskb, &r, hooknum);

+static unsigned int
+ip_vs_forward_with_fwmark(unsigned int hooknum, struct sk_buff **pskb,
+           const struct net_device *in, const struct net_device *out,
+           int (*okfn)(struct sk_buff *))
+    if ((*pskb)->ipvs_property || ! (*pskb)->nfmark)
+            return NF_ACCEPT;
+    return ip_vs_in(hooknum, pskb, in, out, okfn);

/* After packet filtering, forward packet through VS/DR, VS/TUN,
   or VS/NAT(change destination), so that filtering rules can be
@@ -1072,6 +1084,14 @@
    .priority       = 100,

+static struct nf_hook_ops ip_vs_forward_with_fwmark_ops = {
+    .hook        = ip_vs_forward_with_fwmark,
+    .owner        = THIS_MODULE,
+    .pf        = PF_INET,
+    .hooknum        = NF_IP_FORWARD,
+    .priority       = 101,
/* After packet filtering, change source only for VS/NAT */
static struct nf_hook_ops ip_vs_out_ops = {
    .hook        = ip_vs_out,
@@ -1150,9 +1170,17 @@
        goto cleanup_postroutingops;

+    ret = nf_register_hook(&ip_vs_forward_with_fwmark_ops);
+    if (ret < 0) {
+        IP_VS_ERR("can't register forward_with_fwmark hook.\n");
+        goto cleanup_forwardicmpops;
+    }
    IP_VS_INFO("ipvs loaded.\n");
    return ret;

+  cleanup_forwardicmpops:
+    nf_unregister_hook(&ip_vs_forward_icmp_ops);
@@ -1172,6 +1200,7 @@

static void __exit ip_vs_cleanup(void)
+    nf_unregister_hook(&ip_vs_forward_with_fwmark_ops);

Here, I redirect the packets with fwmark to ip_vs_in() on the FORWARD chain, and ip_vs_in() will handle the packets which are marked by iptables to indicate transparent cache virtual service, but ignore other packets (let them continue to flow on the FORWARD chain).

Now you can use ipvs to deploy TP at ease:
@ ipvs director
# sysctl -w net.ipv4.ip_forward=1
# iptables -t mangle -A FORWARD -p tcp -s <internal network> --dport 80 -j MARK --set-mark 1
# ipvsadm -A -f 1 -s lblcr
# ipvsadm -a -f 1 -r RS1
# ipvsadm -a -f 1 -r RS2

@ RS
# iptables -t mangle -A PREROUTING -p tcp -s <internal network> --dport 80 -j REDIRECT --to-ports 3128
# cat >> /etc/squid/squid.conf << EOF
 httpd_accel_host virtual
 httpd_accel_port 80
 httpd_accel_with_proxy on
 httpd_accel_uses_host_header on
# /etc/init.d/squid start

<Prev in Thread] Current Thread [Next in Thread>