LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: Our LVS/DR backends freezes

To: Joseph Mack NA3T <jmack@xxxxxxxx>
Subject: Re: Our LVS/DR backends freezes
Cc: Horms <horms@xxxxxxxxxxxx>
Cc: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
From: Olle Ö?stlund <olle@xxxxxxxxxxx>
Date: Tue, 28 Nov 2006 17:22:50 +0100
> you will have persistence if you used the -p option to 
> ipvsadm when setting up the virtual service (see with 
> `ipvsadm` - I don't see it in the output below).

No, we are not using ipvsadm persistance.


> This seems an large number of InActConn for the number of 
> ActiveConn. Once you change the weight of the realserver to 
> 0, the InActConn should all go away in FIN_WAIT (about 
> 2mins). Does this happen?

Look for yourself. These are figures from the director when all
realservers are open (weight = 10)

IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.3.10:80 wrr
  -> 192.168.3.105:80             Route   10     755        33987
  -> 192.168.3.107:80             Route   10     842        35426
TCP  192.168.3.9:80 wrr
  -> 192.168.3.7:80               Route   10     42         2732
  -> 192.168.3.5:80               Route   10     51         2567
TCP  192.168.3.9:443 wrr
  -> 192.168.3.7:443              Route   10     0          4
  -> 192.168.3.5:443              Route   10     0          4
=====
TOTAL
=====
NONE            84 (max/mean expire time 59/29)
FIN_WAIT     34623 (max/mean expire time 119/30)
ESTABLISHED   1675 (max/mean expire time 900/160)
CLOSE        40143 (max/mean expire time 60/29)
SYN_RECV        49 (max/mean expire time 58/32)

=====================
PER BACKEND-INTERFACE
=====================
192.168.3.7    NONE             5 (max/mean expire time 59/39)
192.168.3.7    FIN_WAIT      1168 (max/mean expire time 113/30)
192.168.3.7    ESTABLISHED     43 (max/mean expire time 899/97)
192.168.3.7    CLOSE         1560 (max/mean expire time 59/28)
192.168.3.7    SYN_RECV         8 (max/mean expire time 58/33)
-----------------------
192.168.3.5    FIN_WAIT      1150 (max/mean expire time 107/30)
192.168.3.5    ESTABLISHED     53 (max/mean expire time 899/136)
192.168.3.5    CLOSE         1424 (max/mean expire time 59/29)
192.168.3.5    SYN_RECV         3 (max/mean expire time 51/27)
-----------------------
192.168.3.107  NONE            34 (max/mean expire time 59/31)
192.168.3.107  FIN_WAIT     16439 (max/mean expire time 119/30)
192.168.3.107  ESTABLISHED    832 (max/mean expire time 900/161)
192.168.3.107  CLOSE        19010 (max/mean expire time 59/29)
192.168.3.107  SYN_RECV        24 (max/mean expire time 58/33)
-----------------------
192.168.3.105  NONE            45 (max/mean expire time 58/26)
192.168.3.105  FIN_WAIT     15866 (max/mean expire time 119/30)
192.168.3.105  ESTABLISHED    747 (max/mean expire time 900/164)
192.168.3.105  CLOSE        18149 (max/mean expire time 60/29)
192.168.3.105  SYN_RECV        14 (max/mean expire time 57/31)
-----------------------

These are figures from the director when the realserver with interfaces
192.168.3.5 and 192.168.3.105 have been closed (weight = 0) for 5
minutes:

IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.3.10:80 wrr
  -> 192.168.3.105:80             Route   0      623        33687
  -> 192.168.3.107:80             Route   10     929        36252
TCP  192.168.3.9:80 wrr
  -> 192.168.3.7:80               Route   10     53         2746
  -> 192.168.3.5:80               Route   0      46         2549
TCP  192.168.3.9:443 wrr
  -> 192.168.3.7:443              Route   10     0          4
  -> 192.168.3.5:443              Route   0      0          4
=====
TOTAL
=====
NONE            84 (max/mean expire time 59/29)
FIN_WAIT     35008 (max/mean expire time 119/30)
ESTABLISHED   1654 (max/mean expire time 899/153)
CLOSE        40115 (max/mean expire time 59/29)
SYN_RECV        50 (max/mean expire time 59/27)

=====================
PER BACKEND-INTERFACE
=====================
192.168.3.7    NONE             5 (max/mean expire time 38/25)
192.168.3.7    FIN_WAIT      1183 (max/mean expire time 119/29)
192.168.3.7    ESTABLISHED     53 (max/mean expire time 898/266)
192.168.3.7    CLOSE         1562 (max/mean expire time 59/29)
192.168.3.7    SYN_RECV         8 (max/mean expire time 54/25)
-----------------------
192.168.3.5    FIN_WAIT      1129 (max/mean expire time 59/28)
192.168.3.5    ESTABLISHED     46 (max/mean expire time 322/38)
192.168.3.5    CLOSE         1421 (max/mean expire time 59/29)
192.168.3.5    SYN_RECV         3 (max/mean expire time 50/30)
-----------------------
192.168.3.107  NONE            34 (max/mean expire time 58/27)
192.168.3.107  FIN_WAIT     17162 (max/mean expire time 119/30)
192.168.3.107  ESTABLISHED    932 (max/mean expire time 899/231)
192.168.3.107  CLOSE        19007 (max/mean expire time 59/29)
192.168.3.107  SYN_RECV        25 (max/mean expire time 56/29)
-----------------------
192.168.3.105  NONE            45 (max/mean expire time 59/31)
192.168.3.105  FIN_WAIT     15534 (max/mean expire time 113/29)
192.168.3.105  ESTABLISHED    623 (max/mean expire time 600/36)
192.168.3.105  CLOSE        18125 (max/mean expire time 59/29)
192.168.3.105  SYN_RECV        14 (max/mean expire time 59/24)
-----------------------



> If the client/server holds the connection, then the 
> ActiveConn will not drop till they disconnect. If InActConn 
> is behaving properly, the above numbers show that the 
> connection time for the client is 2min*(722/32908) = 2 secs. 
> This seems short: the server (I thought) held the connection 
> open incase the client makes a subsequent request.

It should. Our Tomcats are setup to keep the client connections open for
20 seconds (I have checked and doublechecked).


<Prev in Thread] Current Thread [Next in Thread>