ktcpvs configuration problems

["David A. Sinck" <lvs-lists@xxxxxxxxxxxxxx>]

My current goal here is to produce a load balancing LVS for some
number of real machines, complete with 'application server
stickiness'.  Eventually, with failover director support, but not
right now.

Behaviour observed:
* some "good" results
* some corrupted results
* ktcpvs unilaterally ignores cookies
* http requests from the director to 'realservers' work


What I've done:

Redhat 7.3 core OS install
kernel 2.4.18
 + most stock redhat 7.3 patches (2.4.18-3)  (ie NOT ipvs 0.9.7 :-)
 + ipvs 1.0.3
 + ipvadm 1.20
 + ktcpvs 0.0.9
simple perl script to report/set cookies
ktcpvs setup to notice cookies and change server balancing

Current network setup

+- client -+
|10.10.10.2|  
+----------+
     |
+---------+
| switch  |
+---------+
     |
+---- director -------------+
| eth0: 10.10.10.3 (switch) |
| eth1: 192.168.0.4 (hub)   |
+---------------------------+
   |
+-----+
| hub |
+-----+
   |
+-- realserver ------+
|eth0   192.168.0.20 |
|eth0:1 192.168.0.21 |
|eth0:2 192.168.0.22 |
+--------------------+

The realservers default gateway is set to 192.168.0.4 ('director') .
The director default gateway is set to 10.10.10.2 ('client').  I don't
think the director default gateway is important, but I'm scraping the
bottom of the bucket for clues and I'd just as soon avoid more
splinters under the fingernails.

Diagnostic elements:

If I query from the director to any of the IPs being listened to by
the realserver, the script accurately notices the change in
SERVER_ADDR from .20 to .21 to .22, so my impression is that the
script works.

If I query from the client to the director VIP (10.10.10.3) I *always*
wind up on .20, despite it *looking* like it should be configured to
notice the cookie that says 'server21' or 'server22'.

Configuration files are inlined below.


Sample 'good' result:
------------
HTTP_COOKIE => cyanocrylate=server22
SERVER_ADDR => 192.168.0.20
saw old cookie as server22
[Sticky]
-----------

This is a cut and paste from a browser to
http://10.10.10.3/~sinck/sticky.cgi (as is the one below), and notices
that I've been there before (HTTP_COOKIE ... server22), yet notice
that the script is reporting that it's still bouncing off of .20, not
.22 like I think the ktcpvs config file specifies.


Sample corrupted result:
---------------
30 HTTP_COOKIE => cyanocrylate=server22
27 SERVER_ADDR => 192.168.0.20
1f saw old cookie as server22
26 [Sticky] 0 HTTP/1.1 200 OK Date: Sun, 30 Jun 2002 23:13:25 GMT Server: 
Apache Keep-Alive: timeout=15, max=98 Connection: Keep-Alive Transfer-Encoding: 
chunked Content-Type: text/html 30 HTTP_COOKIE => cyanocrylate=server22
27 SERVER_ADDR => 192.168.0.20
1f saw old cookie as server22
26 [Sticky] 0 
----------

Er, herm, that's new and different behaviour actually.  Previously it
would start with the "0 HTTP/1.1 200 OK" in the corrupted results and
give the rest.  Again, the cookie is set to 22, yet both (?)
connections are to .20 AND the results are corrupted.  Also, the
corrupted result is about 3x as long to return as the non corrupted
result. 

So, in sum, my questions are:
1) Why are my results sometimes corrupted?
2) What did I do wrong (if anything) in the ktcpvs config?
3) Did I miss a step someplace?
4) What are the unexpected numbers in the corrupted output possibly
   related to (30, 27, 26)?
5) Is there further diagnostic information I could provide?
6) Why doesn't it work?  :-)


Thanks,

David 





ktcpvs:
-----------------
/sbin/insmod ktcpvs.o
/sbin/insmod tcp_vs_wlc.o
/sbin/insmod tcp_vs_http.o
echo 12 > /proc/sys/net/ktcpvs/debug_level

userspace/tcpvsadm -f realserver
userspace/tcpvsadm --list -n
userspace/tcpvsadm --start -i http
-------------------

realserver
--------------
Virtual http {
    listen = 10.10.10.3:80
    scheduler = http
    startservers = 8
    maxclients = 256
    minspareservers = 6
    maxspareservers = 18
    server = 192.168.0.22:80 1
    server = 192.168.0.21:80 1
    server = 192.168.0.20:80 1
    rule = pattern "cyanocrylate.server22" use server 192.168.0.22:80
    rule = pattern "cyanocrylate.server21" use server 192.168.0.21:80
    rule = pattern ".*" use server 192.168.0.20:80
}
---------------


sticky.cgi
--------------
#!/usr/bin/perl

# quick script to look at the current value of the 
# cyanocrylate cookie and report it; if it isn't set
# set it to a random server spec that should match
# something in the ktcpvs script

use CGI qw/:standard/;

my $q = new CGI;

$frag = $ENV{SERVER_ADDR};
$frag =~ s/.*\.//;

$frag += int(rand(2)+1);
my $oldfh = select(STDOUT); $| = 1; select($oldfh);  #make STDOUT piping hot

my $oldcook = $q->cookie(-name => 'cyanocrylate');
$cook = $q->cookie(-name => 'cyanocrylate', -value => 'server'.$frag)
   unless ($oldcook);

if (!$oldcook)
  {
     print $q->header(-cookie=>$cook);    
  }
else
  {
    print $q->header();
  }

foreach (sort keys %ENV)
  {
    next unless /SERVER_ADDR|COOK/;
    print "<B>$_</B> => $ENV{$_}<BR>\n";
  }

print "set cookie to +$frag+<BR>\n" if (!$oldcook);
print "saw old cookie as $oldcook<BR>\n";
print "<A HREF=/~sinck/sticky.cgi>Sticky</A>\n";
------------------------