Strange LVS behavior -- any thoughts?

["Tony Requist" <tony.requist@xxxxxxxxxxxxx>]

I have a LVS-DR setup using keepalived balancing HTTP and SMTP connections
without persistence.  This was working fine and seemingly load balancing
fine, though with little traffic.  I have a custom compiled 2.6.7 kernel.

Now I am seeing a couple of odd things:

1) All requests are sent to one server, even though "ipvsadm" shows all the
servers available.  If I reboot or restart keepalived, another server is
seemingly picked at random to get all the requests.  I am sending requests
from several different clients and seeing this.

2) "ipvsadm" and "ipvsadm -Ln --stats" show nothing (all zeros) even when
requests are clearly being handled.

# ipvsadm
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  www.airset.net:smtp rr
  -> web5.airset.net:http         Route   1      0          0
  -> web4.airset.net:http         Route   1      0          0
  -> web3.airset.net:http         Route   1      0          0
  -> web2.airset.net:http         Route   1      0          0
  -> web1.airset.net:http         Route   1      0          0
TCP  www.airset.net:http rr
  -> web5.airset.net:http         Route   1      0          0
  -> web4.airset.net:http         Route   1      0          0
  -> web3.airset.net:http         Route   1      0          0
  -> web2.airset.net:http         Route   1      0          0
  -> web1.airset.net:http         Route   1      0          0

# ipvsadm -L --stats
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes
OutBytes
  -> RemoteAddress:Port
TCP  www.airset.net:smtp                 0        0        0        0
0
  -> web5.airset.net:http                0        0        0        0
0
  -> web4.airset.net:http                0        0        0        0
0
  -> web3.airset.net:http                0        0        0        0
0
  -> web2.airset.net:http                0        0        0        0
0
  -> web1.airset.net:http                0        0        0        0
0
TCP  www.airset.net:http                 0        0        0        0
0
  -> web5.airset.net:http                0        0        0        0
0
  -> web4.airset.net:http                0        0        0        0
0
  -> web3.airset.net:http                0        0        0        0
0
  -> web2.airset.net:http                0        0        0        0
0
  -> web1.airset.net:http                0        0        0        0
0

There were small numbers shown (that didn't change as more requests were
handled), then I did a "ipvsadm -Z" and they have been at zero ever since.

Thanks in advance for any suggestions.

And a quick "thank you" to everybody involved.  LVS and keepalived are great
tools (as are iproute2, iptables, ...)  So great that I am such I have done
something pretty stupid to get the behavior above :)

- tony