|
we've been having an ongoing discussion on load balancing here over the
past few days, my last composition being this. there are 3 issues in the
middle which i've run into and i've heard other people run into, so maybe
the points and solutions will be helpful to someone on here. if not, oh
well, hit d.
i hope i'm not violating any sort of nda by forwarding this, laugh. the
text is almost all mine so it's not like i'm giving away some sort of
secret company info. heh.
if my explanations of things in here are wrong or can be commented on,
please feel free as i'd love to have all relevant info and not be
incorrect.
-tcl.
---------- Forwarded message ----------
Date: Wed, 1 Nov 2000 14:42:11 -0800 (PST)
From: tc lewis <tcl@xxxxxxxxxxxxxxx>
Reply-To: webmanglers@xxxxxxxxxxx
To: webmanglers@xxxxxxxxxxx
Cc: Jonathan Stark <stark@xxxxxxxxxxx>
Subject: Re: [webmanglers] Load balancer scalability
Resent-Date: Wed, 1 Nov 2000 14:43:45 -0800 (PST)
Resent-From: tc lewis <tcl@xxxxxxxxxxxxxxx>
Resent-To: tcl@xxxxxxxxx
Resent-Subject: Re: [webmanglers] Load balancer scalability
On Wed, 1 Nov 2000, Jonathan Stark wrote:
> Really great discussion guys...
>
> Quoting tc lewis (tcl@xxxxxxxxxxxxxxx):
> > the difference that you're still missing is that lvs' dr _doesn't rewrite
> > the packet _at all__. it just forwards it on a lower network level (via
> > hardware address). so: rewriting a packet in any way (even a very small
> > manner) and forwarding it vs just forwarding it. the latter seems
> > obviously more efficient.
>
> TC, can you go through the DR process so everyone's on the same page?
> Explain the localhost thing, and how the return packets never
> touch the LVS box?
>
> It's really cool, and it sounds like not everbody knows about it...
sure. the way i understand it, at least with lvs' direct routing, is
this:
a packet comes in to a router. the router sends it along to the balancer.
the balancer checks its ipvs (managed with ipvsadm or some other tool)
table to see where the packet should be directed to, and does any
necessary calculations in the process in terms of round robin or least
connection or weighted values and if persistence is on. the packet is
then pushed along to the proper real server via the real server's network
interface's hardware address. the real server then, assuming proper
configuration, accepts the packet, processes it however necessary (web
request? etc) and sends its response (if any) back to the requesting
client ip. the real server's gateway (next hop) is that router in step 1.
the outgoing packets do _not_ go back through the balancer.
lvs' nat method, in contrast, works by the packet going through the router
to the balancer, the balancer does similar procesing of determining which
real server it should go to, then rewrites the packet's destination
ip/port (source ip/port also?), and sends it to the real server. the real
server processes it, and sends the response back. the real server's
gateway is the _balancer_. so the balancer then gets the response packet,
rewrites it again to put the proper source/destination information back in
it, and the balancer then sends it along to the router and back to the
client.
here are some typical reasons i've heard as to why lvs' dr won't fly, and
the workarounds:
. in order for the real server to send directly back to the client, the
real server needs a route to get there. this is usually accomplished by
giving the machine a real public ip and route. this is a waste of ip
space and an extra missed layer of security. the workaround is to give
the real servers internal unroutable ips, and set their gateway to the
router's address, and add the router's route the the real server.
unfortunately, this isn't enough for proper arp requests and responses to
be sent between the router and real server. the solution is to add a
static arp entry in the real server's arp table of the router's ip and mac
address. this solution was presented to me by Ian S. McLeod
<ian@xxxxxxxxxxx> on the lvs list, thread entitled "direct routing/gateway
issues." (started by tc lewis <tcl@xxxxxxxxx>). this will allow the real
servers to not use public ip addresses, altho the interfaces used for the
responses still has to be on the same _physical_ network as the router.
. "but i don't want to run my webserver on port 80 on the real servers."
direct routing, since it doesn't rewrite the packet, cannot change the
destination port of a packet, unlike nat/pat. so normal web traffic will
hit the real servers on port 80. the solution to this, assuming linux
real servers, is to use an ipchains redirect to the proper port (something
alone the lines of /sbin/ipchains -A input -d [public ip of the webserver]
80 -p tcp -j REDIRECT [local port of the webserver]). this is also one
method ("horms' method") of avoiding the "arp problem" of lvs' direct
routing and tunneling methods which is discussed at
http://www.linuxvirtualserver.org/arp.html. i won't get into the rest of
that "arp problem" unless desired.
. "but i want my real servers to be able to send requests out to the real
world so i can ftp from them and stuff". utilizing nat, the real servers
can get back out to the real world since nat (masquerading) is already
there. with dr, nat isn't there. the balancer, or a separate machine,
can do masquerading for the real servers for non-request traffic, but you
don't want your real servers using that masquerading machine as its
gateway, or its responses will go through there as well, ruining part of
the dr efficiency. the solution is to use policy routing on the real
servers to tell them to use the masquerading machine as its gateway in
some circumstances and to use the normal router as its gateway for the
request-response (web, whatever) traffic. i believe that goes something
along the lines of:
ip rule add prio 100 from [private network address]/[private network netmask]
table 100
ip route add table 100 0/0 via [private ip address for masq machine] dev eth0
the masq machine also needs to then listen on its private ip address, and
needs to be told to not issue icmp redirects so the real server doesn't
skip the masq machine, since it's on the same physical network as the
router. something like:
ifconfig eth# [private ip] netmask [netmask]
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
and of course to do masquerading:
ipchains -A forward -s [private net]/[netmask] -j MASQ
the ip policy routing and icmp redirect issues were responded to by Julian
Anastasov <ja@xxxxxx> on the lvs list, subject "doing both NAT and DR, I
need help.", a thread which was started by idealab's own Jeremy Hansen
<jeremy@xxxxxxxxxxxx>.
- - - - -
that's about it. it sounds like those problems can be a hassle, but it's
not really. all of that research has already been done, so it's just a
matter of running the commands. all of the above assumes use of linux
2.2 for the real servers and balancer, but the theories behind them apply
to use of any systems or appliances.
lvs' direct routing method is discussed more at
http://www.linuxvirtualserver.org/VS-DRouting.html and other documents
found at http://www.linuxvirtualserver.org/Documents.html, and of course
on the mailing list.
-tcl.
---------------------------------------------------------------------
To unsubscribe, e-mail: webmanglers-unsubscribe@xxxxxxxxxxx
For additional commands, e-mail: webmanglers-help@xxxxxxxxxxx
|
