|
Ok - I finally solved this, and I'd like to offer it up as a
suggestion for another resolution to the LVS-NAT RealServers as
Clients notes. While this does route all traffic back through the
director, it is an entirely generic solution so it could work for as
many VIPs and RIPs as necessary. As a fringe benefit and result of
going round-about via your default gateway through the director, all
connections follow the same load balancing you've established for the
system. This might be important if you have a lot of these cases. This
setup also provides the added benefit that realservers can communicate
to anywhere for outbound connections the normal network gateway.
My work is largely inspired by Jonathan and Karen's blog post here:
http://developingrapids.blogspot.com/2006/05/conning-mark-multiwan-connections.html
I'm describing redhat/centos based configs, slight variations may be
required on other systems that I'm not familiar with.
The assumption is that you have 2 internal VLANs, one for "management"
traffic and one for LVS traffic. I'll call those:
InternalNet 192.168.1.x
LVSNet 192.168.2.x
The LVS director is 192.168.2.100
On realservers:
- Use your normal default gateway for your network, NOT the LVS
Director, eg: 192.168.1.1
- Use mangle table prerouting firewall marks to mark any LVS traffic
to fwmark 2
- Use mangle table prerouting firewall marks to mark any non-LVS
traffic to fwmark 1 (may be redundant)
- Use ip rules to direct any traffic received with fwmark 2 via the
LVSNet table (192.168.2.100 gateway)
- Use ip rules to direct any traffic received on 192.168.2.x via the
LVSNet table (192.168.2.100 gateway - may be redundant)
- Use ip rules to direct any traffic received with fwmark 1 via the
InternalNet table (192.168.1.1 gateway - may be redundant to the
default route)
- Use ip rules to direct any traffic received on 192.168.1.x via the
InternalNet table (192.168.1.1 gateway - may be redundant)
Configuration:
===========
1) Add two tables to your /etc/iproute2/rt_tables
200 InternalNet
201 LVSNet
2) Set your default gateway to 192.168.1.1 (/etc/sysconfig/network)
GATEWAY=192.168.1.1
3) In your iptables mangle table (for real server 192.168.1.112), add
the following. In different realservers, replace the IP address
192.168.1.112 with that of the current real server:
-A PREROUTING -j CONNMARK --restore-mark
-A PREROUTING -m mark --mark 0x1 -j ACCEPT
-A PREROUTING -d 192.168.1.112 -j MARK --set-mark 0x1
-A PREROUTING -m mark --mark 0x2 -j ACCEPT
-A PREROUTING -d 192.168.2.112 -j MARK --set-mark 0x2
-A PREROUTING -j CONNMARK --save-mark
4) Setup IP routes and rules (for each real server you change the IP
address in the rules below to that of the real server IPs)
routes: /etc/sysconfig/network-scripts/route-eth1:
default table InternalNet via 192.168.1.1
default table LVSNet via 192.168.2.100
rules: /etc/sysconfig/network-scripts/rule-eth1
from 192.168.1.112 table InternalNet
from 192.168.2.112 table LVSNet
fwmark 1 lookup InternalNet
fwmark 2 lookup LVSNet
------ DONE ----
Net result is that LVS-NAT based inbound connections route back out
through the LVS Director appropriately, even though your default
gateway is on 192.168.1.1. All other real server requests route
through your normal LAN default gateway. If that outbound request is
to one of your LVS VIPs (real server as client), the connection goes
via your normal LAN to the front end of the LVS director, and the
reply comes back the same path.
---------------------
Verify configuration:
You end up with:
[root@rack112 ~]# route
eth1 Link encap:Ethernet
inet addr:192.168.1.112 Bcast:192.168.1.255 Mask:255.255.255.0
eth1:1 Link encap:Ethernet
inet addr:192.168.2.112 Bcast:192.168.2.255 Mask:255.255.255.0
[root@rack112 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth1
[root@rack112 ~]# ip rule show
0: from all lookup 255
32759: from all fwmark 0x2 lookup LVSNet
32760: from all fwmark 0x1 lookup InternalNet
32761: from 192.168.2.112 lookup LVSNet
32762: from 192.168.1.112 lookup InternalNet
32763: from all fwmark 0x2 lookup main
32766: from all lookup main
32767: from all lookup default
Best,
James
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
