|
On Tue, 18 Jul 2000, tc lewis wrote:
>
> this isn't working for me / i'm missing something.
> here's the setup again, in detail:
my configure script makes this a lot easier
(it's on the
Documentation page).
Here's the conf file for what you've set up
#lvs_dr.conf (C) Joseph Mack mack@xxxxxxxxxxx
LVS_TYPE=VS_DR
INITIAL_STATE=on
VIP=eth0:132 lvs 255.255.255.255 lvs
DIRECTOR_INSIDEIP=eth2 director-inside 192.168.1.0 255.255.255.0 192.168.1.255
DIRECTOR_DEFAULT_GW=router
SERVICE=t telnet rr realserver1 realserver2
SERVER_VIP_DEVICE=TP
SERVER_NET_DEVICE=eth1
SERVER_DEFAULT_GW=router
#----------end lvs_dr.conf------------------------------------
where your /etc/hosts (or DNS) is the following
64.208.49.1 router
64.208.49.132 lvs
192.168.100.130 director-inside
192.168.100.99 realserver1
192.168.100.98 realserver2
My configure script fails when it's run (although it's a bizarre failure
and not one I'd anticipated). The problem is that your VIP is not in the
realserver network. Since you're using TP for the VIP on the realserver
network, I didn't have a device in the 64.x.x.x network on the realserver
and couldn't route to the outside world from the realservers.
I haven't played with realservers for VS-DR having private network IPs.
Make sure this isn't going to be a problem for your local routing.
I'd put the VIP and the realservers into the same network for a start.
Since I'm about to dive out the door to Ottawa, I won't be able to do a
2nd round on this question till Sunday or so.
Good luck.
Joe
>
> [client]
> |
> [router] 64.208.49.1
> |
> [director] 64.208.49.130 (DIP) on eth1
> 64.208.49.132 (VIP) on eth1:0
> 192.168.100.130 on eth2
> |
> |
> [real server 1] 192.168.100.99 on eth1
> gw 64.208.49.1
> [real server 2] 192.168.100.98 on eth1
> gw 64.208.49.1
>
>
> both interfaces of the director and real servers are connected to the same
> switch--there's no physical segmentation there. i'm using lvs-dr and
> horms' ipchains method on the real servers to avoid the arp problem.
> here's how i do it:
>
> on the director:
> /sbin/ifconfig eth1:0 64.208.49.132 netmask 255.255.255.0 up
> /sbin/route add -host 64.208.49.132 dev eth1
> /bin/echo 1 > /proc/sys/net/ipv4/ip_forward
> /usr/sbin/ipvsadm -A -t 64.208.49.132:80 -s lc
> /usr/sbin/ipvsadm -a -t 64.208.49.132:80 -r 192.168.100.99:80 -g
> /usr/sbin/ipvsadm -a -t 64.208.49.132:80 -r 192.168.100.98:80 -g
>
> on the real servers:
> /sbin/route add -net 64.208.49.0 netmask 255.255.255.0 dev eth1
> /sbin/route add -net 0.0.0.0 gw 64.208.49.1
> /bin/echo 1 > /proc/sys/net/ipv4/ip_forward
> /sbin/ipchains -A input -d 64.208.49.132 80 -p tcp -j REDIRECT 4080
>
>
> eth0 is not used on any of these machines.
>
> i have another machine there that will eventually be used as a failover
> director, but right now it's just chilling out. it's 64.208.49.131 on
> eth1 and 192.168.100.131 on eth2. if i use this machine as the client,
> everything works fine. http responses are received. life is wonderful.
> if i use a client outside of the local network, things break.
>
> for example, if i try from 208.219.36.67, tcpdump shows stuff like:
>
> director:
> 12:18:36.700977 eth1 B arp who-has 64.208.49.1 tell 192.168.100.99
> 12:18:36.700985 eth2 B arp who-has 64.208.49.1 tell 192.168.100.99
> 12:18:37.574251 eth1 < 208.219.36.67.62374 > 64.208.49.132.www: S
> 98091686:98091686(0) win 32120 <mss 1460,sackOK,timestamp 161199526
> 0,nop,wscale 0> (DF)
> 12:18:37.574282 eth2 > 208.219.36.67.62374 > 64.208.49.132.www: S
> 98091686:98091686(0) win 32120 <mss 1460,sackOK,timestamp 161199526
> 0,nop,wscale 0> (DF)
> 12:18:37.649761 eth1 < 192.168.100.98 > 64.208.49.132: icmp: host
> 208.219.36.67 unreachable [tos 0xc0]
> 12:18:37.649831 eth2 > 192.168.100.98 > 64.208.49.132: icmp: host
> 208.219.36.67 unreachable [tos 0xc0]
>
>
> real server:
> 12:17:52.999514 eth1 B arp who-has 64.208.49.1 tell 192.168.100.98
> 12:17:52.999521 eth2 B arp who-has 64.208.49.1 tell 192.168.100.98
> 12:17:53.550585 eth1 > 192.168.100.99 > 64.208.49.132: icmp: host
> 208.219.36.67 unreachable [tos 0xc0]
> 12:17:53.550598 eth1 > 192.168.100.99 > 64.208.49.132: icmp: host
> 208.219.36.67 unreachable [tos 0xc0]
>
>
> other real server:
> 12:17:55.251106 eth1 B arp who-has 64.208.49.1 tell 192.168.100.99
> 12:17:55.251114 eth2 B arp who-has 64.208.49.1 tell 192.168.100.99
> 12:17:56.124482 eth1 < 208.219.36.67.62374 > 64.208.49.132.www: S
> 98091686:98091686(0) win 32120 <mss 1460,sackOK,timestamp 161199526
> 0,nop,wscale 0> (DF)
> 12:17:56.199872 eth1 > 192.168.100.98 > 64.208.49.132: icmp: host
> 208.219.36.67 unreachable [tos 0xc0]
>
>
> assumingly because the real servers can't contact the client.
>
> route -n on real server:
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use Iface
> 192.168.100.99 0.0.0.0 255.255.255.255 UH 0 0 0 eth1
> 192.168.200.99 0.0.0.0 255.255.255.255 UH 0 0 0 eth2
> 192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
> 64.208.49.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
> 192.168.200.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
> 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
> 0.0.0.0 64.208.49.1 0.0.0.0 UG 0 0 0 eth1
>
>
> the other real server's is similar.
> (that 192.168.200.0/24 net is another backend private net for db
> interaction, pay no attention to it).
>
> i was assuming that this would be ok. the outgoing packets from the real
> servers should have a source ip of the VIP, which the router would see and
> forward to the outside world appropriately, correct? apparently this
> isn't the case. what am i missing here? the router is not setup to know
> about 192.168.100.0/24, but i didn't think it would have to be. does it?
> what am i missing?
>
> advice appreciated!
>
> -tcl.
>
>
>
> ---------- Forwarded message ----------
> Date: Tue, 11 Jul 2000 14:17:18 -0400 (EDT)
> From: tc lewis <tcl@xxxxxxxxx>
> To: Ian S. McLeod <ian@xxxxxxxxxxx>
> Cc: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Subject: Re: will this work (direct routing)?
>
>
> ok, cool. gotcha on the other traffic being dropped thing. i could
> always just throw a separate box in there entirely to masquerade the real
> servers. hmm but then everything would be forwarded through that box,
> which is a needless extra hop for web traffic, so yeah that leads back to
> what you were saying about source-based forwarding. that's no big deal
> for me at this point. the real servers shouldn't need to get outside of
> the internal network except for lvs-forwarded traffic (http requests).
> nevertheless, thanks for the heads up on that in case i go down that road
> later.
>
> in rh6.2, "ip" (/sbin/ip) and related tools are in the "iproute" package.
>
> -tcl.
>
>
> On Tue, 11 Jul 2000, Ian S. McLeod wrote:
>
> > This should work. However, attempts to connect directly to the outside
> > internet from the Real Servers will most likely fail. Why? Because they
> > will forward packets to the gateway with a source address inside of a
> > private IP range (192.168) which the router will drop.
> >
> > As best I can tell, the only way to solve this problem is to have the LVS
> > servers double as masquerading gateways and use source based routing on
> > the Real Servers such that:
> >
> > Packets with a source address of the VIP go directly to the "real"
> > gateway, achieving the performance benefits of DR.
> >
> > Packets with a source address inside of 192.168 are routed to the
> > masquerading gateway on the LVS boxes.
> >
> >
> > When I last investigated this the only way to do source based routing on
> > Linux was with the "ip" command (which I can't find in any recent
> > distributions). Anyone know where it went?
> >
> > -Ian
> >
> > On Tue, 11 Jul 2000, tc lewis wrote:
> >
> > >
> > > here's what i'm thinking i can do:
> > >
> > > 200.200.200.1 = router
> > > (whatever, some publically-accessable ip range...)
> > > 200.200.200.11 = lvs balancer 1.
> > > 200.200.200.12 = lvs balancer 2.
> > > route 192.168.100.0/255.255.255.0 added to both balancers (not sure if
> > > this is even necessary)
> > > 192.168.100.101 = real server 1.
> > > 192.168.100.102 = real server 2.
> > > route 200.200.200.0/255.255.255.0 added to both real servers.
> > > gateway on real servers = 200.200.200.1
> > >
> > > 2 balancers that fail over via heartbeat/ultramonkey.
> > >
> > > i'd like to do balancing on port 80 with the direct routing method. i'll
> > > probably use ipchains on the real servers to solve the arp problem as i'll
> > > probably be redirecting port 80 to some non-priviledged port on the real
> > > server anyway (8080, whatever). the machines listed above will not be
> > > physically segmented--they'll all be on the same vlan of a foundry
> > > workgroup network switch.
> > >
> > > will this work? if they're on the same physical segment like this then
> > > the balancers should be able to redirect traffic properly via direct
> > > routing, and the real servers can then send back out to the real world
> > > with that 200.200.200.0 route through the .1 gateway.
> > >
> > > am i correct or am i missing something here?
> > >
> > > sorry, it's been a while since i've done much with lvs, so i just wanted a
> > > quick confirmation. thanks!
> > >
> > > -tcl.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
>
>
>
--
Joseph Mack mack@xxxxxxxxxxx
|
