Connections not being balanced to multiple servers

[Tom <bigendian+lvs@xxxxxxxxx>]

I've been testing a new installation of LVS + keepalived and am having some
trouble.  Things work to a degree, but I notice that connections to the VIP
always stick with a single real server instead of being distributed across
all nodes in the real server list.  When a health check to the actively
utilized real server fails then another server will be utilized as if the
VIP is doing some sort of failover.  I've tried the rr, wrr, lc, wlc, and nq
scheduling algorithms without observing any change in behavior.  I can force
the use of a particular server by increasing its weight, but this does
nothing to distribute the load.

I'm running a fresh install of Opensuse 10.1.  I used the prebuilt
keepalived and ipvsadm packages and found that keepalived did not support
LVS as it would seem to do VRRP but never set up any virtual servers visible
with "ipvsadm -l".  To remedy this, I downloaded the src packages for
ipvsadm and keepalived and simple rebuilt the latter with a combination of
"rpmbuild -bc ipfwadm.spec" and "rpmbuild -ba keepalived.spec".  This gave
me a fresh keepalived rpm which I installed and got the results reported
above.  Not sure if any of this matters.

Here's some connection data gathered after loading up the VIP with
connection from the apache benchmarking tool "ab2 -n 10000 -c 100
http://10.0.0.118/"; which sends 10,000 GETs for "/" to the VIP in batches of
1,000 in parallel.

# ipvsadm -l -n --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes
OutBytes
 -> RemoteAddress:Port
TCP  10.0.0.118:80 <http://10.0.0.118/>                   10100    56678
56554  3884911  6916037
 -> 192.168.1.124:80 <http://192.168.1.124/>                10100
56678    56554  3884911  6916037
 -> 192.168.1.121:80 <http://192.168.1.121/>                    0
0        0        0        0


Here's my configuration:

global_defs {
       notification_email {
               root
       }
       notification_email_from tomm@xxxxxxxxxxx

       smtp_server localhost
       smtp_connect_timeout 30

       lvs_id LVS_MAIN
}

vrrp_sync_group VG1 {
       group {
               VI_1
       }
}

vrrp_instance VI_1 {
       state MASTER
       interface eth0
       virtual_router_id 51
       priority 100
       advert_int 1
       smtp_alert
       authentication {
               auth_type PASS
               auth_pass 1111
       }

       ! There should be an address here for each virtual server
       virtual_ipaddress {
               10.0.0.118
       }
}

virtual_server 10.0.0.118 80 {

       ! Seconds between checks
       delay_loop 6

       ! load balancing algorythm
       lb_algo nq

       ! Packet forwarding method.  NAT requires all packets associated
with
       ! a connection to be routed through the load balancer.
       lb_kind NAT

       ! Connections are tracked for this number of seconds before being
reset.
       persistence_timeout 50

       protocol TCP

       ! The sorry server handles requests when no real servers are
available
!       sorry_server 192.168.1.130 80

       real_server 192.168.1.121 80 {
               weight 1
               TCP_CHECK {
                       connect_port 80
                       connect_timeout 3
               }
       }

       real_server 192.168.1.124 80 {
               weight 1
               TCP_CHECK {
                       connect_port 80
                       connect_timeout 3
               }
       }
}



Any help is much appreciated!