[lvs-users] Connecting to a VIP from a realserver

[Vesa-Matti J Kari <vmkari@xxxxxxxxxxxxxx>]

Hello,

The OS is CentOS release 5.5 with

  ipvsadm-1.24-10
  piranha-0.8.4-16
  kernel-2.6.18-194.11.3.el5

My lvs.cf configuration file is at the end of this
message. Below is an ASCII depiction of the nodes:

#######################################################

LVS ROUTERS (have SMTP VIP and MILTER VIP)
  +--------+  +--------+
  | lvs-01 |  | lvs-02 |
  +--------+  +--------+



SMTP AND MESSAGE SUBMISSION REAL SERVERS
  +----------+  +----------+
  | smtp-rs1 |  | smtp-rs2 |
  +----------+  +----------+

               MILTER REAL SERVERS
  +-----------+ +----------+ +----------+ +----------+
  |milters-rs1| |milter-rs2| |milter-rs3| |milter-rs4|
  +-----------+ +----------+ +----------+ +----------+

#######################################################

I would like to have the smtp-rs1 and smtp-rs1 use
the MILTER VIP for load balancing and also redundancy
in the case of milter-rs realserver failures.

Now, unfortunately problems arise when one of the
realservers smtp-rs1 or smtp-rs2 tries to connect to
the MILTER VIP.

For example I have tried simply:

  telnet [MILTER VIP] 7777

from the smtp-rs1 realserver, with bad consequences:

First, the connection hangs. What is worse, the
currently active LVS router loses track of its round
robin scheduling, leaving only one connection active.
However, the LVS logs do not indicate that anything went
wrong.

The problem does exist and can be verified by observing
the logs on real servers, e.g. only one realserver of the
milter-rs machines receives connections via the LVS router.
Strangely enough, despite the reality, ipvsadm shows
that all realservers are online and receiving connections.

When the inactive LVS router is rebooted, the active
LVS router gets back to its senses and starts using
round robin again.



>From the MINI-HOW at:

  http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html

I found the following:

##### quotation starts #####

2.2. Gotchas: you need an outside client (the director and realservers
can't access the virtual service)

To set up and test/run LVS, you need a minimum of 3
machines: client, director, realserver(s).

>From the outside, the LVS functions as one machine. The client
cannot be one of the machines in the LVS (the director, or
realserver). You need an outside client. If you try to access an LVS
controlled service (eg http, smtp, telnet) from any of the machines
in the LVS; access from the director will hang, access from a
realserver will connect to the service locally, bypassing the LVS.

##### quotation ends #####


My questions: Have I run into one of the "gotchas" here by
trying to access MILTER VIP from the smtp-rs realservers?

If yes, does anyone know what would be the recommended method
of connecting to MILTER VIP from smtp-rs realservers?

Thanks in advance for any information.


Below is the configuration file:

#######################################################


serial_no = 41
primary = 128.214.148.196
primary_private = 10.0.0.1
service = lvs
backup_active = 1
backup = 128.214.148.197
backup_private = 10.0.0.2
heartbeat = 1
heartbeat_port = 539
keepalive = 6
deadtime = 18
network = direct
debug_level = 7
monitor_links = 1
syncdaemon = 0
virtual SMTP {
     active = 1
     address = 128.214.148.198 eth0:1
     vip_nmask = 255.255.255.240
     port = 25
     send = "GET / HTTP/1.0rnrn"
     expect = "OK"
     use_regex = 0
     send_program = "/usr/local/sbin/smtp-nanny.py %h 25"
     load_monitor = none
     scheduler = rr
     protocol = tcp
     timeout = 11
     reentry = 15
     quiesce_server = 1
     server real_server_1 {
         address = 128.214.148.199
         active = 1
         weight = 1
     }
     server real_server_2 {
         address = 128.214.148.200
         active = 1
         weight = 1
     }
}
virtual SUBMISSION {
     active = 1
     address = 128.214.148.198 eth0:1
     vip_nmask = 255.255.255.240
     port = 587
     send = "GET / HTTP/1.0rnrn"
     expect = "OK"
     use_regex = 0
     send_program = "/usr/local/sbin/smtp-nanny.py %h 587"
     load_monitor = none
     scheduler = rr
     protocol = tcp
     timeout = 11
     reentry = 15
     quiesce_server = 1
     server real_server_1 {
         address = 128.214.148.199
         active = 1
         weight = 1
     }
     server real_server_2 {
         address = 128.214.148.200
         active = 1
         weight = 1
     }
}
virtual MILTER {
     active = 1
     address = 128.214.148.201 eth0:2
     vip_nmask = 255.255.255.240
     port = 7777
     send = "GET / HTTP/1.0rnrn"
     expect = "OK"
     use_regex = 0
     send_program = "/usr/local/sbin/milter-nanny %h 7777"
     load_monitor = none
     scheduler = rr
     protocol = tcp
     timeout = 11
     reentry = 15
     quiesce_server = 1
     server real_server_1 {
         address = 128.214.148.202
         active = 1
         weight = 1
     }
     server real_server_2 {
         address = 128.214.148.203
         active = 1
         weight = 1
     }
     server real_server_3 {
         address = 128.214.148.204
         active = 1
         weight = 1
     }
     server real_server_4 {
         address = 128.214.148.205
         active = 1
         weight = 1
     }
}

#######################################################

vmk
-- 
************************************************************************
               Tietotekniikkaosasto / Helsingin yliopisto
                 IT department / University of Helsinki
************************************************************************


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users