Subject: [lvs-users] Trouble getting first LVS setup working

From: "Chris St. Pierre" <[email-masked]>
Date: Fri, 20 Mar 2009 14:37:09 -0500 (CDT)

I'm playing with LVS to try and get a working test setup going, and
I'm running into some problems.  I've got three VMs -- two "real"
servers, one LVS router.  Everything appears to start up correctly,
'ipvsadm -L' looks like I think it should, but the routing from the
VIP to the real servers just plain isn't happening.  Network
connectivity between the router and real servers works fine, but I
can't even connect to the VIP on port 80.

Following Redhat's documentation, I've tried both the iptables
PREROUTING and the arptables methods, but neither has worked.

I'm sure I've made a silly mistake, but I can't for the life of me
figure it out.  And LVS's logging is so quiet that I'm not getting
much; all appears to be working according to the logs.

Thanks for your help!

##### lvs.cf #####

service = lvs
primary = 10.9.3.3
heartbeat = 1 
heartbeat_port = 1050
keepalive = 6 
deadtime = 18 
reservation_conflict_action = preempt
debug_level = NONE

network = direct

virtual server1 {
         address = 10.9.3.6 eth0:1
         vip_nmask = 255.255.0.0
         fwmark = 100
         active = 1
         load_monitor = rup
         timeout = 5
         reentry = 10
         port = http
         send = "GET / HTTP/1.0\r\n\r\n"
         expect = "HTTP"
         protocol = tcp

         server Real1 {
                 address = 10.9.3.1
                 active = 1
                 weight = 1
         }

         server Real2 {
                 address = 10.9.3.2
                 active = 1
                 weight = 1
         }
}

Chris St. Pierre
Unix Systems Administrator
Nebraska Wesleyan University

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - [email-masked]
Send requests to [email-masked]
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

---

From: Joseph Mack NA3T <[email-masked]>
Date: Fri, 20 Mar 2009 18:05:38 -0700 (PDT)

On Fri, 20 Mar 2009, Chris St. Pierre wrote:

> servers, one LVS router.
.
.
> Following Redhat's documentation,

Redhat never comes up on this mailing list. I didn't know 
they knew anything about LVS.

Have you read the HOWTO about the (limited number of) 
problems that we know about with VMs? Can you find a simpler 
setup (separate nodes) for your first attempt?

Joe

-- 
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

---

From: "Chris St. Pierre" <[email-masked]>
Date: Fri, 20 Mar 2009 20:33:25 -0500 (CDT)

> they knew anything about LVS.

Yup:

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.2/html/Virtual_Server_Administration/index.html

I won't lie and say it's particularly thorough documentation -- most
of it covers using the configuration web GUI -- but it's readable, and
I figured it'd be a good place to start as I'm running RHEL.

> problems that we know about with VMs?

I have now, and none of them seem to be my problem.  Many are little
bugs or resource allocation issues, but they all appeared to be
problems that exhibited themselves once things were at least
marginally working.  In my case, I can't even get the dang thing
working at all.

> Can you find a simpler setup (separate nodes) for your first
> attempt?

I'd love to, but I have this problem with my email address: it ends in
.edu. :) Getting a single test box took me about four years, so if
you're willing to put this thread on hold until 2013, then I should be
able to get a bigger test environment. :)

Chris St. Pierre

---

From: Graeme Fowler <[email-masked]>
Date: Sat, 21 Mar 2009 09:21:07 +0000

Hi Chris

> I figured it'd be a good place to start as I'm running RHEL.

It's as good a place as any, but it doesn't really explain the basics
very well - it's a sort of "click here, do this, all will be well"
approach which doesn't give you the full grounding about the "why" (or
even the "how" in some cases!).

> marginally working.  In my case, I can't even get the dang thing

I would normally ask for all sort of debug info here, but as you already
stated this is a test platform I'm going to ask you to completely clear
the RH-driven configs and start from a clean slate. BTW, I feel it's
your VIP netmask causing the problem, but we'll come to that later...

What you'll need to do is as follows:

# /sbin/ip address add 10.9.3.6/32 dev eth0
# /sbin/ipvsadm -A -t 10.9.3.6:80 -s rr
# /sbin/ipvsadm -a -t 10.9.3.6:80 -r 10.9.3.1:80 -w 100 -m
# /sbin/ipvsadm -a -t 10.9.3.6:80 -r 10.9.3.2:80 -w 100 -m

That then gets you a very basic virtual server on 10.9.3.6:80/tcp which
has realservers on port 80/tcp on addresses 10.9.3.1 and 10.9.3.2, using
the "masquerade" method (LVS-NAT). The prerequisite for this is that the
realservers have a default gateway address pointing to the director, ie.
10.9.3.6.

Once you have this set up, you can try to access the VIP from a client
which is *not on the same LAN as the VIP or the realservers*. This is a
key point - the return traffic MUST go back via the director in LVS-NAT
or the SYN/AYN-ACK/ACK handshake will fail.

Please try this and let us know how you get on.

By the way, if your VIP netmask is too wide on the director then the
director will believe that a Very Large Network is local to it or
immediately adjacent. That can stop normal traffic flow dead, right
there.

Graeme

---

From: "Chris St. Pierre" <[email-masked]>
Date: Mon, 23 Mar 2009 08:50:17 -0500 (CDT)

> Please try this and let us know how you get on.

Okay, I did that, and the result is -- improved?  I think?

I can see the HTTP request in the logs of one of the real servers.  So
the request gets that far.  But the response apparently never reaches
the client; curl just sits waiting until it finally times out.  I
_was_ testing from a different subnet.  I can ping the client from the
real servers, so at least some of the networking magic appears to be
working fine.

Chris St. Pierre

---

From: Joseph Mack NA3T <[email-masked]>
Date: Mon, 23 Mar 2009 08:52:19 -0700 (PDT)

On Mon, 23 Mar 2009, Chris St. Pierre wrote:

> I can ping the client from the real servers,

you shouldn't be able to do this (at least on a well setup 
machine). The only connection between the client and the 
realservers is through the LVS-NAT. A ping packet shouldn't 
be able to get to the realservers from the client.

Is the only route to the client from the realservers through 
the director?

Joe

---

From: "Chris St. Pierre" <[email-masked]>
Date: Mon, 23 Mar 2009 11:11:00 -0500 (CDT)

> Is the only route to the client from the realservers through

No, all I configured wasa route from the real servers to the client
going through the director.

I'm not sure how to force that behavior, either, without creating a
VLAN with no default route to the outside world.  When pinging the
real servers from the client, our routers are quite happy to route the
packets directly to the real servers.  I don't control the network
here, so that's not an option -- and, incidentally, as I don't intend
to use LVS-NAT in production, I think it'd be kind of
counterproductive to jump through those kinds of hoops to get this
working.

That said, I'm not sure why packet routing from the client to the real
servers should matter; in the test I was doing, I was requesting the
VIP from the client; the director passed the request on to a real
server; and the real server routes its replies through the director.
At no time during that test should the client _try_ to communicate
directly with the real server, so I'm not sure why that matters.

Chris St. Pierre

---

From: Graeme Fowler <[email-masked]>
Date: Mon, 23 Mar 2009 17:34:01 +0000

> Okay, I did that, and the result is -- improved?  I think?

Righto, so you're seeing something good(ish). Although you then go on to
say in response to Joe's message that you're not intending to use NAT in
production, so it would be productive now to switch to the method you
are intending to use which I assume will be -DR.

In that case, please clear the director's config (ipvsadm -C) and switch
it to -DR mode instead:

On the director:
# /sbin/ip address add 10.9.3.6/32 dev eth0
# /sbin/ipvsadm -A -t 10.9.3.6:80 -s rr
# /sbin/ipvsadm -a -t 10.9.3.6:80 -r 10.9.3.1:80 -w 100 -g
# /sbin/ipvsadm -a -t 10.9.3.6:80 -r 10.9.3.2:80 -w 100 -g

On the realserver(s):
# /sbin/ip address add 10.9.3.6/32 dev lo

Add the following to /etc/sysctl.conf:
# LVS switches
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

Then do
# sysctl -p

Then see what you get from your clients. Note that you should remove the
route you put in for NAT usage from the realservers.

What that lot does is puts the VIP on the director's live interface and
sets up the ipvs table, then you put the VIP on the realservers'
loopback adapters and ensure that the realservers don't send ARP
responses for that address.

Hopefully you'll then see a fully-working LVS.

Graeme

---

From: "Chris St. Pierre" <[email-masked]>
Date: Mon, 23 Mar 2009 12:53:15 -0500 (CDT)

> Hopefully you'll then see a fully-working LVS.

$beers_i_owe{'graeme'}++;

Awesome!  It works!

With a working config in hand, I looked into getting it working with
pulse.

I compared the output of 'ipvsadm -L' after entering the literal
commands you gave me, with the output having started pulse with my
afore-pasted config file; there's a fairly substantive difference, but
I'm not sure how _important_ it actually is.  Namely:

Graeme's working config:

Prot LocalAddress:Port Scheduler Flags
TCP  cluster-test6.NebrWesleyan.e rr
[...snip...]

My non-working config:

Prot LocalAddress:Port Scheduler Flags
FWM  100 wlc

That pretty well explains why it wasn't working: my config had
'fwmark=100' in the config, left over from the sample config I
cannibalized, although I wasn't using (and wasn't planning on using)
firewall marks.  I removed that line, and suddenly a) 'ipvsadm -L'
outputs matched; and b) the balancer worked with the pulse service.

Woot!  Thanks for everyone's help on this!

Chris St. Pierre

---

From: Sebastian Vieira <[email-masked]>
Date: Tue, 24 Mar 2009 09:37:14 +0100

> net.ipv4.conf.all.arp_announce = 2

I always asuumed that you need to set the sysctl values first and only then
bring up the loopback adapter, else it would do an arp broadcast.. Are there
situations where you don't need to?

Sebastian.

---

From: Graeme Fowler <[email-masked]>
Date: Tue, 24 Mar 2009 09:58:05 +0000

> situations where you don't need to?

Well, test environments I guess - but good point, if you don't want any
ARP traffic emanating at all, it's good practive to apply the sysctls
first. I only did it that way because I'd written the VS setup bit of
the email before I cribbed a handy startup script :)

Graeme