Re: [lvs-users] LVS + Xen + NAT

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: Re: [lvs-users] LVS + Xen + NAT
From: "David Dyer-Bennet" <dd-b@xxxxxxxx>
Date: Wed, 17 Sep 2008 14:51:44 -0500 (CDT)
On Wed, September 17, 2008 13:08, Josh Mullis wrote:
> Here is my output from iptables-save:

Looks sane so far as I can tell (remember that I'm still struggling with
my own first setup, though).

You don't need the "secondary" addresses for the LVS nodes (that's the
terminology from piranha-gui; one of the problems here is that there's
lots of conflicting terminology being used in different tools around all
this stuff).

The LVS node needs two real (soporate lan, or internet) IPs and two
virtual (private network for the cluster) IPs; one each outside and
inside.  The virtual IPs will be moved between the two LVS nodes on
failover, whereas the real IPs will stay with the hardware they're
assigned to.

The private virtual IP must be configured in each realserver as the
gateway node.

My impression is that getting that wrong is both the easiest and most
common way of getting packets to go in but not come out; but that
impression may be specific to redhat/centos setups using piranha-gui,
which you haven't mentioned using (but it's what I'm using, hence it
influences what I know).

Here's my ipvsadm output:

sh-3.2# /sbin/ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  prcvmod01.employer.local     wlc
  ->              Masq    8      0          0
  ->              Masq    8      0          0

172.17. is the private internal lan for the cluster.  .4 and .3 there are
realservers, separate physical systems in my case (they'll have Xen and
multiple virtual servers on them soon). is the virtual IP for the LVS director; it gets applied to
the inside IP port of the LVS node, and would be moved to the active LVS
if a failover happened (I haven't gotten to redundant LVS and failover

prcvmod01.employer.local is the public (corporate lan) virtual IP for the
service (it's  The LVS system itself has its own corporate
lan IP of

The iptables look like:

sh-3.2# /sbin/iptables-save
# Generated by iptables-save v1.3.5 on Wed Sep 17 14:32:03 2008
:INPUT ACCEPT [7375193:1143767059]
:FORWARD ACCEPT [396083:75791540]
:OUTPUT ACCEPT [6115668:423106080]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d -o virbr0 -m state --state
-A FORWARD -s -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m physdev  --physdev-in vif5.0 -j ACCEPT
# Completed on Wed Sep 17 14:32:03 2008
# Generated by iptables-save v1.3.5 on Wed Sep 17 14:32:03 2008
:PREROUTING ACCEPT [1498599:131914689]
:POSTROUTING ACCEPT [398187:28611428]
:OUTPUT ACCEPT [409849:29413401]
# Completed on Wed Sep 17 14:32:03 2008

The most obvious difference is just that you're running your virtual
server on the same box, and I'm not.  I started out trying to run it all
on one box, and gave up.  I think I now understand the things I gave up
on, and I think it should work in just one box with my current config --
but I haven't tried it yet.  I'm going to get it working in the simple
case of multiple boxes first, and *then* add in the complexity of some
realservers on the same hardware as the LVS director.
David Dyer-Bennet, dd-b@xxxxxxxx;

<Prev in Thread] Current Thread [Next in Thread>