|
Greetings,
Where I work, I set up an LVS load balancer under Linux 2.6.10 around
3 months ago. At the time I built it out originally, there were only
three NICs and only two networks to work with. For sanity sake, I will
refer to networks hereafter as ExtNet1 and IntNet1. NIC 1 was
configured as the .254 on IntNet1, while NIC 2 was configured as the
.254 on ExtNet1. A few days later I started adding aliases to NIC2,
filling up the entire class C subnet. Up to this point everything was
working great. I was load balancing around 250 sites with 3 real
servers each.
Recently, we decided to retire our old 1:1 NAT system for our older
legacy sites. The goal was to let the LVS load balancer take over the
two class C subnets that were being managed by the old 1:1 NAT server.
Friday evening, I converted all of the NAT rules that we used on the old
system to LVS rules on the new system and added two more NICs, NIC 3 and
NIC 4, to support the two class C subnets from the old system. I will
call these subnets ExtNet2 and ExtNet3.
At this point, a simplified drawing of our system looks like this;
----------------------
| ISP Router |
----------------------
|
|
----------------------
| Firewall |
----------------------
|
|
----------------------
| | |
| | |
| | |
----------------------
| Firewall |
----------------------
|
|
|
---------------------------------------
| | |
| | |
| | |
| | |
---------------------- ----------------------
----------------------
| SVR | | SVR | | SVR
|
---------------------- ----------------------
----------------------
Requests to any of the sites on ExtNet1 are coming in and getting
load balanced perfectly. Requests to any of the sites on ExtNet2 or
ExtNet3, however, are failing. With tcpdump, I see the syn packet
coming in to the correct NIC device, for example NIC 3 (ExtNet2),
however, the syn packet never makes it to NIC 1 (IntNet1). Thinking
that it might have something to do with a limit on the number of NICs, I
shut down NIC 3 and NIC 4 and created aliases on NIC 2 to support the
two class Cs for ExtNet2 and ExtNet3. Fired up tcpdump and ExtNet1
still works correctly, but, ExtNet2 and ExtNet3 continue to fail. The
SYN packets are getting to the device, but never being passed through to
NIC 1.
Next, I turned on iptables logging, starting at the INPUT chain.
The logs clearly show that traffic destined for ExtNet1 is coming in and
being processed, but I see no indication of any packets destined for
ExtNet2 or ExtNet3. At this point, I turned logging on in mangle
PREROUTING and mangle INPUT, with mangle PREROUTING, I see the SYN
packets, but they never get to mangle INPUT.
At this point, I am at a complete loss for how to proceed, so I come
here begging for help. My iptables configuration is as follows;
--- Cut --- Paste --- Cut --- Paste --- Cut --- Paste ---
#!/bin/bash
#
#
export PATH=/bin:/sbin:/usr/bin:/usr/sbin
# === enable stuff ===
modprobe ip_tables
modprobe ip_conntrack
modprobe iptable_nat
modprobe ip_nat_ftp
modprobe ip_conntrack_ftp
# === turn on forwarding ===
echo "1" > /proc/sys/net/ipv4/ip_forward
# === clear all rules ==-
iptables -t nat --flush
iptables --flush
# === Base Policies
=========================================================
echo Installing base policies ...
# === Set Default Policies ===
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
# === Accept Input From Any Internal Network ===
iptables -A INPUT -i lo -p all -j ACCEPT
iptables -A INPUT -i eth0 -p all -j ACCEPT
iptables -A INPUT -i eth1 -p all -j ACCEPT
# === Accept Output To Any Internal Network ===
iptables -A OUTPUT -o lo -p all -j ACCEPT
iptables -A OUTPUT -o eth0 -p all -j ACCEPT
iptables -A OUTPUT -o eth1 -p all -j ACCEPT
# === Accept Forward From Any Internal Network ===
iptables -A FORWARD -i eth0 -j ACCEPT
iptables -A FORWARD -i eth1 -j ACCEPT
# === NAT ===
#iptables -t nat -A POSTROUTING -s 10.0.0.0/8 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth3 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth4 -j MASQUERADE
# === Disallow INVALID Incoming Or Forwarded Packets From Internet ===
#iptables -A INPUT -m state --state INVALID -j REJECT
iptables -A INPUT -i eth4 -m state --state INVALID -j REJECT
iptables -A INPUT -i eth3 -m state --state INVALID -j REJECT
iptables -A INPUT -i eth2 -m state --state INVALID -j REJECT
#iptables -A FORWARD -m state --state INVALID -j REJECT
iptables -A FORWARD -i eth4 -m state --state INVALID -j REJECT
iptables -A FORWARD -i eth3 -m state --state INVALID -j REJECT
iptables -A FORWARD -i eth2 -m state --state INVALID -j REJECT
--- Cut --- Paste --- Cut --- Paste --- Cut --- Paste ---
And for each site, I have the following;
--- Cut --- Paste --- Cut --- Paste --- Cut --- Paste ---
/sbin/iptables -A INPUT -p tcp -s 0.0.0.0/0 -d ${PUBIP} --dport 80 -j
ACCEPT
/sbin/iptables -A INPUT -p udp -s 0.0.0.0/0 -d ${PUBIP} --dport 80 -j
ACCEPT
/sbin/iptables -A INPUT -p tcp -s 0.0.0.0/0 -d ${PUBIP} --dport 443 -j
ACCEPT
/sbin/iptables -A INPUT -p udp -s 0.0.0.0/0 -d ${PUBIP} --dport 443 -j
ACCEPT
--- Cut --- Paste --- Cut --- Paste --- Cut --- Paste ---
${PUBIP} is replaced by the actual IP of the VIP
From there, I create the following LVS rules;
--- Cut --- Paste --- Cut --- Paste --- Cut --- Paste ---
/sbin/ipvsadm -A -t ${PUBIP}:80 -p 600 -s lc
/sbin/ipvsadm -A -t ${PUBIP}:443 -p 600 -s lc
/sbin/ipvsadm -a -t ${PUBIP}:80 -r ${RIP1}:${HTTP} -m -w 1
/sbin/ipvsadm -a -t ${PUBIP}:443 -r ${RIP1}:${HTTPS} -m -w 1
/sbin/ipvsadm -a -t ${PUBIP}:80 -r ${RIP2}:${HTTP} -m -w 1
/sbin/ipvsadm -a -t ${PUBIP}:443 -r ${RIP2}:${HTTPS} -m -w 1
/sbin/ipvsadm -a -t ${PUBIP}:80 -r ${RIP3}:${HTTP} -m -w 1
/sbin/ipvsadm -a -t ${PUBIP}:443 -r ${RIP3}:${HTTPS} -m -w 1
--- Cut --- Paste --- Cut --- Paste --- Cut --- Paste ---
|
