|
Transparent cache, taking transparent web cluster for example, seems like below
topology or deployment:
Given client machines make ipvs director default gateway, http requests toward
internet web sites from the clients forward through the ipvs director.
Obviously, the director must be enable to capture those packets which are
involved in some virtual service defined in the ipvs rule table, though they
exist in FORWARD chain which go beyond the power control of ipvs, because the
only outside-inside function entrance is ip_vs_in() which only hooks in INPUT
chain, right? As long as ip_vs_in() handles these packets, ipvs can finally
forward (VS/DR) them to the backend real servers. The real servers use the
REDIRECT iptable rule to let local squid adopt them, and at last, client
machines get the correct http responses.
LVS HOWTO provides two ways to meet our need, but I found they don't work.
1. define a route in local route table (in "17.3. How you use TP" chapter)
I test this way in Redhat AS4.0_update2 which uses 2.6.x kernel, but nothing
magic happens, :-( Maybe 2.6.x kernel does not support this trick.
Even this can work, but how can you capture all packets to the whole internet
web sites in your local route table? :-)
2. REDIRECT iptable rule
Let's see how REDIRECT rule work in 2.6.x kernel first.
Given such a rule configured on ipvs director:
# iptables -t mangle -A PREROUTING -p tcp -s <internal network> --dport 80 -j
REDIRECT --to-ports 3128
When a packet [cip:cport -> website_ip:80] comes in, netfilter redirects it
from PREROUTING chain to INPUT chain, and it becomes [cip:cport ->
localhost_ip:3128], and then it's handled by squid (with transparent proxy
settings). Squid gets the webpage from cache or from internet, and send back
the webpage to the client. When the response packet [localhost_ip:3128 ->
cip:cport] flows at the end of POSTROUTING chain, netfilter recovers its
origin, then the packet becomes [website_ip:80 -> cip:cport] and go toward the
client machine on the wire.
We can see that, netfilter MASQUERADEs the packet both on PREROUTING and
POSTROUTING chains. However, it is useless to ipvs director.
First, the packets which ip_vs_in() handles own the unique localhost ip as the
destination ip, and that make those schedule algorithms like lblc, lblcr, dh
(dedicated to cache cluster) useless!
Second, ipvs registers ip_vs_post_routing(), which hooks on POSTROUTING chain,
with a priority of NF_IP_PRI_NAT_SRC-1, which means it is called by netfilter
core before NAT hooks like REDIRECT. This function returns NF_STOP for those
packets which ipvs_property is set, and REDIRECT has no chance to MASQUERADE
those packets. Finally, client receives the response, but it dost not match the
request on the orign ip:port!
I think out a solution for the support of TP in ipvs natively, and make a patch:
--- linux-2.6.16/net/ipv4/ipvs/ip_vs_core.c 2006-03-20 13:53:29.000000000
+0800
+++ linux-2.6.16.new/net/ipv4/ipvs/ip_vs_core.c 2006-11-21
18:28:35.000000000 +0800
@@ -23,6 +23,8 @@
* Changes:
* Paul `Rusty' Russell properly handle non-linear skbs
* Harald Welte don't use nfcache
+ * Jinhua Luo redirect packets with fwmark on
NF_IP_FORWARD chain
+ * to ip_vs_in(), mainly for transparent
cache cluster
*
*/
@@ -1060,6 +1062,16 @@
return ip_vs_in_icmp(pskb, &r, hooknum);
}
+static unsigned int
+ip_vs_forward_with_fwmark(unsigned int hooknum, struct sk_buff **pskb,
+ const struct net_device *in, const struct net_device *out,
+ int (*okfn)(struct sk_buff *))
+{
+ if ((*pskb)->ipvs_property || ! (*pskb)->nfmark)
+ return NF_ACCEPT;
+
+ return ip_vs_in(hooknum, pskb, in, out, okfn);
+}
/* After packet filtering, forward packet through VS/DR, VS/TUN,
or VS/NAT(change destination), so that filtering rules can be
@@ -1072,6 +1084,14 @@
.priority = 100,
};
+static struct nf_hook_ops ip_vs_forward_with_fwmark_ops = {
+ .hook = ip_vs_forward_with_fwmark,
+ .owner = THIS_MODULE,
+ .pf = PF_INET,
+ .hooknum = NF_IP_FORWARD,
+ .priority = 101,
+};
+
/* After packet filtering, change source only for VS/NAT */
static struct nf_hook_ops ip_vs_out_ops = {
.hook = ip_vs_out,
@@ -1150,9 +1170,17 @@
goto cleanup_postroutingops;
}
+ ret = nf_register_hook(&ip_vs_forward_with_fwmark_ops);
+ if (ret < 0) {
+ IP_VS_ERR("can't register forward_with_fwmark hook.\n");
+ goto cleanup_forwardicmpops;
+ }
+
IP_VS_INFO("ipvs loaded.\n");
return ret;
+ cleanup_forwardicmpops:
+ nf_unregister_hook(&ip_vs_forward_icmp_ops);
cleanup_postroutingops:
nf_unregister_hook(&ip_vs_post_routing_ops);
cleanup_outops:
@@ -1172,6 +1200,7 @@
static void __exit ip_vs_cleanup(void)
{
+ nf_unregister_hook(&ip_vs_forward_with_fwmark_ops);
nf_unregister_hook(&ip_vs_forward_icmp_ops);
nf_unregister_hook(&ip_vs_post_routing_ops);
nf_unregister_hook(&ip_vs_out_ops);
Here, I redirect the packets with fwmark to ip_vs_in() on the FORWARD chain,
and ip_vs_in() will handle the packets which are marked by iptables to indicate
transparent cache virtual service, but ignore other packets (let them continue
to flow on the FORWARD chain).
Now you can use ipvs to deploy TP at ease:
@ ipvs director
# sysctl -w net.ipv4.ip_forward=1
# iptables -t mangle -A FORWARD -p tcp -s <internal network> --dport 80 -j MARK
--set-mark 1
# ipvsadm -A -f 1 -s lblcr
# ipvsadm -a -f 1 -r RS1
# ipvsadm -a -f 1 -r RS2
@ RS
# iptables -t mangle -A PREROUTING -p tcp -s <internal network> --dport 80 -j
REDIRECT --to-ports 3128
# cat >> /etc/squid/squid.conf << EOF
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
# /etc/init.d/squid start
|
