|
So I am trying to implement Lars' Method according to the HOWTO @
http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-DR.html
and am having some problems. I think I am missing something simple but I can't
seem to figure it out. What I am trying to do here is not have another public
IP
on the VIP network bound on the realserver since public IPs are so precious.
Here is my config:
Director:
2 interfaces
- eth0 is an administrative interface.
This interface is trunked to the admin VLAN.
It has a DIP of 10.241.70.19
- eth1 is the production interface
This interface is trunked to the production VLAN
The production VLAN consists of 2 networks:
128.227.74/24 and 192.168.74/24
The VIP is 128.227.74.138 and is bound to this interface
This interface also has an IP bound to 192.168.74.138
The default gateway for this box is 10.241.70.1 out the eth0 interface. This
causes some issues with ICMP from machines not on these networks therefore I
have a src route using the iproute2 tools to add a rule that makes sure that
requests to 128.227.74/24 use the eth1 interface and use the default gateway on
that network which is 128.227.74.126 ( don't ask why. It has to do with HSRP).
Here are the commands to do that:
ip route add default via 128.227.74.126 dev eth1 table osg
ip route add 128.227.74.0/24 dev eth1 table osg
ip rule add from 128.227.74.0/24 table osg
This makes sure that packets that this machine makes that have a src address in
the 128.227.74.0/24 network go back out the eth1 interface otherwise if they
use
the other interface they will get dropped by the gateway since they are not
part
of this network.
My routing table looks like this:
tlvs01 ~ # netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
128.227.74.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.241.70.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.74.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 10.241.70.1 0.0.0.0 UG 0 0 0 eth0
and the list of ips bound to the box looks like this:
tlvs01 ~ # ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether e2:03:e1:8e:52:5d brd ff:ff:ff:ff:ff:ff
inet 10.241.70.19/24 brd 10.241.70.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether e2:03:e1:8e:52:5e brd ff:ff:ff:ff:ff:ff
inet 128.227.74.138/24 brd 128.227.74.255 scope global eth1
inet 192.168.74.138/24 brd 255.255.255.0 scope global eth1
What I want to do is have requests for port 80 to VIP be redirected to
192.168.74.81. So here is my ipvsadm table:
tlvs01 ~ # ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=32768)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 128.227.74.138:80 wrr
-> 192.168.74.81:80 Route 1 0 0
Now on to the realserver. It has two interfaces as well:
- eth0 is an administrative interface.
This interface is trunked to the same admin VLAN.
It has an IP of 10.241.70.8
- eth1 is the production interface
This interface is trunked to the same production VLAN as the director
This interface has an IP bound to 192.168.74.81
The realserver also has the VIP bound to lo using the following command:
ip addr add 128.227.74.138 dev lo
Again on this machine I have a similar problem as the director in terms of
packet insertion on the network. I therefore have the same iproute2 rule. Here
it is:
ip route add default via 128.227.74.126 dev eth1 table osg
ip route add 128.227.74.0/24 dev eth1 table osg
ip rule add from 128.227.74.0/24 table osg
To be conscience, here is the routing table and ips bound to the realserver:
misc07 ~ # netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.241.70.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.74.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 10.241.70.1 0.0.0.0 UG 0 0 0 eth0
misc07 ~ # ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet 128.227.74.138/32 scope global lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether e2:03:e1:8e:52:4b brd ff:ff:ff:ff:ff:ff
inet 10.241.70.8/24 brd 10.241.70.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether e2:03:e1:8e:52:83 brd ff:ff:ff:ff:ff:ff
inet 192.168.74.81/24 brd 192.168.74.255 scope global eth1
4: tunl0: <NOARP> mtu 1480 qdisc noop
link/ipip 0.0.0.0 brd 0.0.0.0
On both the director and the realserver I have the following sysctl.conf:
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.eth0.arp_filter = 1
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.lo.arp_filter = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
With this setup, I can ping the 192.168.74/24 IPs in both directions ie. from
the director I can ping 192.168.74.81 and on the realserver I can ping
192.168.74.138.
So now that I have detailed my setup, here is my problem:
When a machine not on these networks tries to access 128.227.74.138:80
1) the director gets the packet on eth1
2) the director replaces the destination mac address of the packet with the mac
address of the realserver and injects it back into the network
3) the realserver gets it on its eth1 interface
and this is where I have a problem. At this point, the realserver seems to not
know how to reply back to the client since I am seeing that the realserver is
generating ICMP host unreachable on the lo interface. Here is the tcpdump
output. Note that the IP of the machine making the request is 128.227.212.87.
misc07 ~ # tcpdump -s 0 -vv -Z nobody -i lo -n
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
09:55:35.201415 IP (tos 0xc0, ttl 64, id 5683, offset 0, flags [none], proto
ICMP (1), length 88) 128.227.74.138 > 128.227.74.138: ICMP host 128.227.212.87
unreachable, length 68
IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length
60) 128.227.74.138.80 > 128.227.212.87.45432: S, cksum 0x0668 (correct),
1327391410:1327391410(0) ack 3340736982 win 5792 <mss 1460,sackOK,timestamp
39702124 872235241,nop,wscale 7>
09:55:35.201422 IP (tos 0xc0, ttl 64, id 5684, offset 0, flags [none], proto
ICMP (1), length 88) 128.227.74.138 > 128.227.74.138: ICMP host 128.227.212.87
unreachable, length 68
IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length
60) 128.227.74.138.80 > 128.227.212.87.45432: S, cksum 0x037a (correct),
1327391410:1327391410(0) ack 3340736982 win 5792 <mss 1460,sackOK,timestamp
39702874 872235241,nop,wscale 7>
I am so very confused why the machine would be trying to contact the client
using loopback interface.
Am I trying to do something here that can not be done?
Any help would be appreciated. Thanks.
--
Eli Ben-Shoshan ebs@xxxxxxx
|
