LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] Standby node in HA connects to its lo:0's instead of the

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] Standby node in HA connects to its lo:0's instead of the lived server's VIP (CentOS 5)
From: "Steven Truong" <midair77@xxxxxxxxx>
Date: Fri, 21 Mar 2008 17:33:02 -0700
On Fri, Mar 21, 2008 at 5:11 PM, Steven Truong <midair77@xxxxxxxxx> wrote:
> Dear all.  I tried to implement HA with 2 CentOS 5 servers (OpenLDAP)
>  using LVS (Ultramonkey).  At this point, I have a weird problem that
>  when I was in the hot _standby_ real server and tried to ssh to the
>  VIP using the VIP address, I actually ended up in the same server
>  instead of the lived _real_ server.
>
>  This caused problems because my OpenLdap slave server needs to be able
>  to connect to the master server (lived server) to replicate but right
>  now the slave server (hot _standby_) keeps connecting to itself.  The
>  whole things got complicated because of SSL/TLS certificates.  This
>  _standby_ server needs to connect to the VIP address that the master
>  server uses to connect with start_tls thingy.
>
>  I have set up /etc/hosts, arptables, /etc/sysctl.conf,
>  /etc/sysconfig/network-scripts/ifcfg-lo and I can not think of a way
>  to do anything else but removing the lo:0.  My slave server was able
>  to replicate and connect to the master server ASAP I removed VIP -
>  lo:0 and restarted lo.
>
>  Hosts on my LAN sshed to the VIP and got in the server (master)
>  correctly when both servers are up.
>
>  Here are the contents of these files:
>
>  #/etc/hosts
>  127.0.0.1       localhost.localdomain   localhost
>
>  #VIP
>  192.168.10.15   red.mynetwork.com    red
>  #REAL servers
>  192.168.0.16     blue.mynetwork.com  blue
>  192.168.0.14     green.mynetwork.com  green
>
>  #/etc/sysctl.conf
>  net.ipv4.ip_forward = 1
>  net.ipv4.conf.default.rp_filter = 1
>  net.ipv4.conf.default.accept_source_route = 0
>  kernel.sysrq = 0
>  net.ipv4.conf.eth0.arp_ignore = 1
>  net.ipv4.conf.eth0.arp_announce = 2
>  net.ipv4.conf.all.arp_ignore = 1
>  net.ipv4.conf.all.arp_announce = 2
>  net.ipv4.vs.expire_quiescent_template=1
>
>  #Centos's kernel seems not to have these
>  #net.ipv4.conf.all.hidden = 1
>  #net.ipv4.conf.lo.hidden = 1
>
>  #/etc/sysconfig/arptables (on green)
>  *filter
>  :IN ACCEPT [37:1036]
>  :OUT ACCEPT [7:196]
>  :FORWARD ACCEPT [0:0]
>  [0:0] -A IN -d 192.168.0.15 -j DROP
>  [0:0] -A OUT -s 192.168.0.15 -o eth0 -j mangle --mangle-ip-s 192.168.0.14
>  COMMIT
>
>  #/etc/sysconfig/network-scripts/ifcfg-lo
>  DEVICE=lo
>  IPADDR=127.0.0.1
>  NETMASK=255.0.0.0
>  NETWORK=127.0.0.0
>  BROADCAST=127.255.255.255
>  ONBOOT=yes
>  NAME=loopback
>
>  DEVICE=lo:0
>  IPADDR=192.168.0.15
>  NETMASK=255.255.255.255
>  NETWORK=192.168.0.0
>  BROADCAST=192.168.0.255
>  ONBOOT=yes
>  NAME=loopback
>
>  #/etc/sysconfig/network-scripts/ifcfg-eth0 (on green)
>  DEVICE=eth0
>  BOOTPROTO=none
>  HWADDR=00:0C:29:4A:2A:93
>  ONBOOT=yes
>  NETMASK=255.255.255.0
>  IPADDR=192.168.0.14
>  GATEWAY=192.168.0.1
>  TYPE=Ethernet
>  USERCTL=no
>  IPV6INIT=no
>  PEERDNS=yes
>
>  #/etc/ha.d/ha.cf
>  debugfile /var/log/ha-debug
>  logfile /var/log/ha-log
>  logfacility     local0
>  mcast eth0 225.0.0.1 694 1 0
>  auto_failback on
>  node    blue.mynetwork.com
>  node    green.mynetwork.com
>  ping 192.168.0.1
>  respawn hacluster /usr/lib64/heartbeat/ipfail
>  apiauth ipfail gid=haclient uid=hacluster
>
>  # /etc/ha.d/haresources
>  blue.mynetwork.com \
>          ldirectord::ldirectord.cf \
>          LVSSyncDaemonSwap::master \
>          IPaddr2::192.168.0.15/24/eth0/192.168.0.255
>
>   #/etc/ha.d/ldirectord.cf
>  checktimeout=10
>  checkinterval=60
>  autoreload=yes
>  logfile="/var/log/ldirectord.log"
>  emailalert="mee@xxxxxxxxxxxxx"
>  quiescent=no
>  virtual=192.168.0.15:389
>         real=192.168.0.16:389 gate
>         real=192.168.0.14:389 gate
>         fallback=127.0.0.1:389
>         service=ldap
>         scheduler="rr"
>         protocol=tcp
>         checktype=negotiate
>         checkport=389
>         login="cn=mee,dc=mynetwork,dc=com"
>         passwd="onepassword"
>         request="uid=bogus,dc=mynetwork,dc=com"
>         receive="uid=bogus,dc=mynetwork,dc=com"
>
>  on green server:
>   ipvsadm -L -n
>  IP Virtual Server version 1.2.1 (size=4096)
>  Prot LocalAddress:Port Scheduler Flags
>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
>
>   ip addr sh
>  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>     inet 192.168.0.15/32 brd 192.168.0.255 scope global lo:0
>     inet6 ::1/128 scope host
>        valid_lft forever preferred_lft forever
>  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 
> 1000
>     link/ether 00:0c:29:4a:2a:93 brd ff:ff:ff:ff:ff:ff
>     inet 192.168.0.14/24 brd 192.168.0.255 scope global eth0
>     inet6 fe80::20c:29ff:fe4a:2a93/64 scope link tentative
>        valid_lft forever preferred_lft forever
>  3: sit0: <NOARP> mtu 1480 qdisc noop
>     link/sit 0.0.0.0 brd 0.0.0.0
>
>  on blue server
>
>  ipvsadm -L -n
>  IP Virtual Server version 1.2.1 (size=4096)
>  Prot LocalAddress:Port Scheduler Flags
>   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
>  TCP  192.168.0.15:389 rr
>   -> 192.168.0.14:389            Route   1      0          0
>   -> 192.168.0.16:389            Local   1      0          0
>
>   ip addr sh
>  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>     inet6 ::1/128 scope host
>        valid_lft forever preferred_lft forever
>  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 
> 1000
>     link/ether 00:0c:29:7c:1f:66 brd ff:ff:ff:ff:ff:ff
>     inet 192.168.0.16/24 brd 192.168.0.255 scope global eth0
>     inet 192.168.0.15/24 brd 192.168.0.255 scope global secondary eth0
>     inet6 fe80::20c:29ff:fe7c:1f66/64 scope link tentative
>        valid_lft forever preferred_lft forever
>  3: sit0: <NOARP> mtu 1480 qdisc noop
>     link/sit 0.0.0.0 brd 0.0.0.0
>
>  chkconfig --list | grep 3:on
>  acpid           0:off   1:off   2:off   3:on    4:on    5:on    6:off
>  anacron         0:off   1:off   2:on    3:on    4:on    5:on    6:off
>  arptables_jf    0:off   1:off   2:on    3:on    4:on    5:on    6:off
>  atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
>  autofs          0:off   1:off   2:off   3:on    4:on    5:on    6:off
>  cpuspeed        0:off   1:on    2:on    3:on    4:on    5:on    6:off
>  crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off
>  firstboot       0:off   1:off   2:off   3:on    4:off   5:on    6:off
>  haldaemon       0:off   1:off   2:off   3:on    4:on    5:on    6:off
>  heartbeat       0:off   1:off   2:on    3:on    4:on    5:on    6:off
>  iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off
>  ldap            0:off   1:off   2:off   3:on    4:off   5:on    6:off
>  network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
>  ntpd            0:off   1:off   2:off   3:on    4:off   5:on    6:off
>  readahead_early 0:off   1:off   2:on    3:on    4:on    5:on    6:off
>  sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
>  syslog          0:off   1:off   2:on    3:on    4:on    5:on    6:off
>  vmware-tools    0:off   1:off   2:on    3:on    4:off   5:on    6:off
>
>  Please point me to the right direction as I run out of things to fix
>  this to work.
>
>  Thank you very much.
>

Oops.  Prior to set up arptables, as soon as I removed lo:0, my slave
(standby)  server was able to replicate or ssh to VIP address, but
with arptables this is no longer true.  Anyway, I still have the
problem without arptables.....

#/etc/sysconfig/arptables (on green)
*filter
:IN ACCEPT [37:1036]
:OUT ACCEPT [7:196]
:FORWARD ACCEPT [0:0]
[0:0] -A IN -d 192.168.0.15 -j DROP
[0:0] -A OUT -s 192.168.0.15 -o eth0 -j mangle --mangle-ip-s 192.168.0.14
COMMIT

Ouch...


<Prev in Thread] Current Thread [Next in Thread>