|
On Wednesday, August 6, 2003, at 10:44 AM, Chris wrote:
-> What is the output of "ipvsadm" in your desired configuration (using
-> the x.y.237.234 VIP)?
IP Virtual Server version 1.0.9 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 wrr persistent 300
-> localhost:http Local 1 0 0
The configuration may have changed slightly - isp is fiddling.
Basically, why does it not work when the VIP is outside the subnet of
the DIP?
for example,
RIP 192.168.1.1/255.255.255.0
VIP 192.168.1.2/255.255.255.0
Works fine.
RIP 192.168.1.1/255.255.255.0
VIP 192.168.2.2/255.255.255.0
Does not work.
How does the upstream router know that it should be sending packets
going to 192.168.2.2 to the same ethernet segment as 192.168.1.1?
Example A (one that works) When Router A (the upstream router) gets a
packet going to (192.168.1.2) it knows that it can reach 192.168.1.x
via its Ethernet interface which is directly connected. Router A ARPs
for 192.168.1.2 and the LVS server answers. Router A now has the MAC
address in its ARP table and builds an Ethernet frame with the LVS MAC
address as its destination. Router A gives the Ethernet frame to
Switch A which sends the packet to the LVS
Example B (one that doesn't work). When Router A (the upstream router)
gets a packet going to (192.168.2.2) it looks at its connected
interfaces. Router A doesn't see a directly connected 192.168.2.x
network so it looks in its routing table. Router A doesn't see a route
tell it where to send 192.168.2.x network traffic so it sends it to its
default route (its upstream router) creating a routing loop.
Solution:
Add a route in your upstream router telling it to send 192.168.2.x
packets to 192.168.1.1 (RIP)
(on cisco 'ip route 192.168.2.0 255.255.255.0 192.168.1.1)
Example C: (Example B + static route). When Router A (the upstream
router) gets a packet going to (192.168.2.2) it looks in its route
table and routes the packet to (192.168.1.1). 192.168.1.1 is a
connected address so it does the ARP lookup. Router A builds an
Ethernet frame for the packet with the MAC address of 192.168.1.1 and
gives it to the switch. The switch delivers it to 192.168.1.1 (LVS
server). The LVS server accepts the packet (it has its MAC address on
it) and the IP layer acccepts the IP as local because it has a
configured interface with that IP.
Here is what I do.. Much more complex example, works great for me and
is very flexible. (IPs have been changed to protect the stupid (i.e.
me))
LVS1 RIP = 192.168.1.10
LVS2 RIP = 192.168.1.11
LVS VIP = 192.168.1.20
LVS SIP = 159.250.250.0/24
** NEW TERM** SIP = Service IP. The IP that is used to provide the
Cluster services (i.e. the working IP)
LVS servers manage VIP and LVS Table using Keepalived and VRRP.
Upstream routers have a route for 159.250.250.0/24 going to LVS VIP.
This can be done with static routes. It goes to the VIP so the LVS
failover works properly.
In my setup SIP is a /24 (256 IP's). SIP.1 = 159.250.250.1, SIP/24 =
159.250.250.0/24
I have fwmark rules that tag DEST IP = SIP.1:25, SIP.1:110 & SIP.1:143
with fwmark 0x01
I have fwmark rules that tag DEST IP = SIP/24:80 & SIP/24:443 with
fwmark 0x02
fwmark 0x01 is LVS:NATed to a cluster of four mail cluster servers
(qmail)
fwmark 0x02 is LVS:DRed to a cluster of web servers (apache)
** Below configures Linux Networking to treat the above marked packets
as local so LVS will see them **
** Without this your LVS server will route the packets and LVS will
have no effect **
ip rule add prio 100 fwmark 1 table 100
ip rule add prio 100 fwmark 2 table 100
ip route add local 0/0 dev lo table 100
The cool thing about this setup is that packets going to the SIP that
are not tagged (i.e. not for services I provide) are not considered
local by the kernel and are routed back out to the upstream router
(creating a routing loop). You cannot traceroute/ping my SIP addresses.
My next step would be to create a blackhole box and set that as the
default route for those packets so they go into the bit bucket instead
of the route loop.
SMTP,IMAP,POP3 are all considered one service and are load balanced
(wlc) using the combined connection count.
Same thing for HTTP, HTTPS
I can easily pick more services out of my SIP pool, assign them a
fwmark and load balance them to any servers I wish.
mail real servers only have their internal IP (192.168.15.x)
web servers have an internal IP (192.168.15.x) and aliases for the SIP
pool because they are direct routed the destination IP doesn't change.
256 IP's = 256 SSL certificates
| 