|
Transparent cache, taking transparent web cluster for example, seems
like below topology or deployment:
Given client machines make ipvs director default gateway, http requests
toward internet web sites from the clients forward through the ipvs
director. Obviously, the director must be enable to capture those
packets which are involved in some virtual service defined in the ipvs
rule table, though they exist in FORWARD chain which go beyond the power
control of ipvs, because the only outside-inside function entrance is
ip_vs_in() which only hooks in INPUT chain, right? As long as ip_vs_in()
handles these packets, ipvs can finally forward (VS/DR) them to the
backend real servers. The real servers use the REDIRECT iptable rule to
let local squid adopt them, and at last, client machines get the correct
http responses.
LVS HOWTO provides two ways to meet our need, but I found they don't work.
1. define a route in local route table (in "17.3. How you use TP" chapter)
I test this way in Redhat AS4.0_update2 which uses 2.6.x kernel, but
nothing magic happens, :-( Maybe 2.6.x kernel does not support this trick.
Even this can work, but how can you capture all packets to the whole
internet web sites in your local route table? :-)
2. REDIRECT iptable rule
Let's see how REDIRECT rule work in 2.6.x kernel first.
Given such a rule configured on ipvs director:
# iptables -t mangle -A PREROUTING -p tcp -s <internal network> --dport
80 -j REDIRECT --to-ports 3128
When a packet [cip:cport -> website_ip:80] comes in, netfilter redirects
it from PREROUTING chain to INPUT chain, and it becomes [cip:cport ->
localhost_ip:3128], and then it's handled by squid (with transparent
proxy settings). Squid gets the webpage from cache or from internet, and
send back the webpage to the client. When the response packet
[localhost_ip:3128 -> cip:cport] flows at the end of POSTROUTING chain,
netfilter recovers its origin, then the packet becomes [website_ip:80 ->
cip:cport] and go toward the client machine on the wire.
We can see that, netfilter MASQUERADEs the packet both on PREROUTING and
POSTROUTING chains. However, it is useless to ipvs director.
First, the packets which ip_vs_in() handles own the unique localhost ip
as the destination ip, and that make those schedule algorithms like
lblc, lblcr, dh (dedicated to cache cluster) useless!
Second, ipvs registers ip_vs_post_routing(), which hooks on POSTROUTING
chain, with a priority of NF_IP_PRI_NAT_SRC-1, which means it is called
by netfilter core before NAT hooks like REDIRECT. This function returns
NF_STOP for those packets which ipvs_property is set, and REDIRECT has
no chance to MASQUERADE those packets. Finally, client receives the
response, but it dost not match the request on the orign ip:port!
I think out a solution for the support of TP in ipvs natively, and make
a patch:
--- linux-2.6.16/net/ipv4/ipvs/ip_vs_core.c 2006-03-20
13:53:29.000000000 +0800
+++ linux-2.6.16.new/net/ipv4/ipvs/ip_vs_core.c 2006-11-21
18:28:35.000000000 +0800
@@ -23,6 +23,8 @@
* Changes:
* Paul `Rusty' Russell properly handle non-linear skbs
* Harald Welte don't use nfcache
+ * Jinhua Luo redirect packets with fwmark on
NF_IP_FORWARD chain
+ * to ip_vs_in(), mainly for
transparent cache cluster
*
*/
@@ -1060,6 +1062,16 @@
return ip_vs_in_icmp(pskb, &r, hooknum);
}
+static unsigned int
+ip_vs_forward_with_fwmark(unsigned int hooknum, struct sk_buff **pskb,
+ const struct net_device *in, const struct net_device *out,
+ int (*okfn)(struct sk_buff *))
+{
+ if ((*pskb)->ipvs_property || ! (*pskb)->nfmark)
+ return NF_ACCEPT;
+
+ return ip_vs_in(hooknum, pskb, in, out, okfn);
+}
/* After packet filtering, forward packet through VS/DR, VS/TUN,
or VS/NAT(change destination), so that filtering rules can be
@@ -1072,6 +1084,14 @@
.priority = 100,
};
+static struct nf_hook_ops ip_vs_forward_with_fwmark_ops = {
+ .hook = ip_vs_forward_with_fwmark,
+ .owner = THIS_MODULE,
+ .pf = PF_INET,
+ .hooknum = NF_IP_FORWARD,
+ .priority = 101,
+};
+
/* After packet filtering, change source only for VS/NAT */
static struct nf_hook_ops ip_vs_out_ops = {
.hook = ip_vs_out,
@@ -1150,9 +1170,17 @@
goto cleanup_postroutingops;
}
+ ret = nf_register_hook(&ip_vs_forward_with_fwmark_ops);
+ if (ret < 0) {
+ IP_VS_ERR("can't register forward_with_fwmark hook.\n");
+ goto cleanup_forwardicmpops;
+ }
+
IP_VS_INFO("ipvs loaded.\n");
return ret;
+ cleanup_forwardicmpops:
+ nf_unregister_hook(&ip_vs_forward_icmp_ops);
cleanup_postroutingops:
nf_unregister_hook(&ip_vs_post_routing_ops);
cleanup_outops:
@@ -1172,6 +1200,7 @@
static void __exit ip_vs_cleanup(void)
{
+ nf_unregister_hook(&ip_vs_forward_with_fwmark_ops);
nf_unregister_hook(&ip_vs_forward_icmp_ops);
nf_unregister_hook(&ip_vs_post_routing_ops);
nf_unregister_hook(&ip_vs_out_ops);
Here, I redirect the packets with fwmark to ip_vs_in() on the FORWARD
chain, and ip_vs_in() will handle the packets which are marked by
iptables to indicate transparent cache virtual service, but ignore other
packets (let them continue to flow on the FORWARD chain).
Now you can use ipvs to deploy TP at ease:
@ ipvs director
# sysctl -w net.ipv4.ip_forward=1
# iptables -t mangle -A FORWARD -p tcp -s <internal network> --dport 80
-j MARK --set-mark 1
# ipvsadm -A -f 1 -s lblcr
# ipvsadm -a -f 1 -r RS1
# ipvsadm -a -f 1 -r RS2
@ RS
# iptables -t mangle -A PREROUTING -p tcp -s <internal network> --dport
80 -j REDIRECT --to-ports 3128
# cat >> /etc/squid/squid.conf << EOF
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
# /etc/init.d/squid start
|
