|
Hello,
We have a LVS NAT+TUN setup like this:
(ALL ips are fake)
DC=Data Center, different geographic locations
VIP=Virtual IP
DIP=Director IP
RIP=Real Server IP
DC#1
VIP: 43.54.12.3 (bound to lo) RIP#1
DIP: 212.212.212.87(eth0) 212.212.212.24 (eth0)
_____ _____
[_____] ---------------------- [_____]
LVS director | Real server #1
|
|
| router gateway 212.212.212.1
.----------------.
|
Internet
|
|
|
|
DC#2 | RIP#2
|____ 67.23.24.25 (eth0)
[_____] Real server #2
The idea is to load balance HTTP requests for VIP (43.54.12.3) to two different
Real Servers #1 and #2. Currently, we have initiated a GRE IP tunnel from
DIP to RIP#2 and to RIP#1 (yes, even though on the same network) with private
IP addressing. So actually, we forward connections to 10.* and 192.168.* class
IPs.
192.168.1.1 LVS director tunnel A IP
192.168.2.1 Real Server #1 tunnel A IP
10.0.1.1 LVS director #1 tunnel B IP
10.0.2.1 Real Server #2 tunnel B IP
We are using 2.4.18 kernels patched with Julian's hidden arp patch.
This is all working very well, other than that there are some serious ARP
issues in the DC#1 network, which appear somewhat randomly. There appears
to be peak times when the ARP traffic is high, but traffic does balance to
normal, ARP is not continuous. Still 6 - 40 Mbps ARP is too much..
Real Server #1 uses 212.212.212.1 as its default
gateway, and I use iproutes to setup the tunnel and the routing tables
for private network routes:
(this example is for Real Server #1)
ip tunnel add ${DEV} mode gre remote ${REMOTEIP} local ${LOCALIP} ttl 255
ip link set ${DEV} up
ip addr add ${PRIVATEIP} dev ${DEV}
ip route add ${PRIVATENET} dev ${DEV}
echo 1 > /proc/sys/net/ipv4/conf/all/hidden
echo 1 > /proc/sys/net/ipv4/conf/${DEV}/hidden
ip rule add from 192.168.2.1/24 table 202 prio 220
ip route add 0/0 src 192.168.2.1 via 192.168.1.1 table 202
I repeat: VIP is bound to LVS director's lo interface, and all
tunnel interfaces are set hidden. We also have rp_filter=0.
If we forget ARP issues, everything is working well.. but we can't do that.
We're not using the normal LVS scenario where each Real Server
has its tunneled interface set up with the VIP, we have
to use NAT and DIP as gateway since the Data center network routers/firewalls
will not
(ever) allow sending packets with addresses not in data center provider's
network address classes
to outside.
We have also tried without the default route to 212.212.212.1, no luck, heavy
arping
still involved!
What could be causing extra high ARP traffic?
|
