|
On Thu, 30 Nov 2006, Olle Ö~Vstlund wrote:
director and the realserver?
I now have analysed the data from the realserver close yesterday. The
connections at the realserver disappear within minutes, but at the
director they "just keep hanging on..."
Drop off rate at the director ("cat /proc/net/ip_vs_conn", only the connections
directed at the realserver is counted):
Before 41872 (925 ESTABLISHED)
I assume 41872 is total (FIN_WAIT + ....) and 925 is only
the ESTABLISHED connections
After 0 minutes: 41796 (931 ESTABLISHED)
After 4 minutes: 41437 (907 ESTABLISHED)
After 12 minutes: 41428 (787 ESTABLISHED)
After 16 minutes: 41386 (772 ESTABLISHED)
After 20 minutes: 41380 (773 ESTABLISHED)
After 23 minutes: 41385 (774 ESTABLISHED)
this is not right. It should be similar to the drop
off for the realserver you show below.
Why do you have ESTABLISHED connections after 23 mins?
Shouldn't they all have reconnected to another realserver?
So both the ESTABLISHED and FIN_WAIT counters are wrong
on the director?
Drop off rate at the realserver ("netstat -an", only LVS-handled
webserver-connections counted):
Before 167 (31 ESTABLISHED)
After 0 minutes: 93 (28 ESTABLISHED)
After 4 minutes: 0
After 12 minutes: 0
After 16 minutes: 0
After 20 minutes: 0
After 23 minutes: 0
this looks normal.
o does the drop off rate on either machine have anything to
do with FIN_WAIT time (about 2mins)
I'm trying to understand how ipvs is using the connection-table. It's
used to keep track of all connections I've understood. What I do not
understand is what is actually happening when a realserver is assigned a
weight of 1. I thought this would mean that there would be no new
connections (or new clients) dispatched to the realserver.
weight=0 means no new connections.
I wrote a
perlscript to check this out, and arrived at very surprising results.
The script is reading all connections from one of
the /proc/net/ip_vs_conn-snapshots I collected, and then it compares all
connections to the content of another snapshot. First I let a connection
be identified by Protocol + FromIP + FromPort + ToIp + ToPort + DestIp
(i e the six leading fields of a /proc/net/ip_vs_conn-record). The
script suggested that later snapshots contained connections which were
not found in earlier snapshots? I then figured that ipvs allow existing
clients (FromIp?) to open new connections, so perhaps Protocol + FromIP
+ FromPort + ToIp + ToPort + DestIp is not a good
"connection-identifier" for my purposes after all.
each new connection request will come from the next port up
in the list, ie if the last connection was from FromIP:3001,
the next will be from FromIP:3001.
Instead I tried using
FromIP as a sole identifier of a connection. I ran the script again and
it still suggested that "new connections" were popping after the
realserver had been closed.
not right.
I have manually verified that new FromIps
are actually popping up in the directors ipvs-table, directed at the
realserver whos weight is 0. This didn't make sense? Then I checd the
STATE of the connections popping up. It was generally CLOSE but I also
saw an ESTABLISHED. My conclusion is that the only implication of
assigning a realserver weight 0 is that SYN-request from a new client
will not be sent to the realserver, but all other types of IP-packets
will be sent there. Is this somewhere near to the truth?
mostly yes. weight=0 means that current connections are
maintained. However it's only with persistence that clients
will be able to make new connections to the same realserver
(if they connect within the persistence time) (and you don't
have persistence). Without persistence, a new connection
request (SYN) will be sent to a realserver with weight!=0.
In an apache/httpd/html situation, the client-server
connection is not neccessarily closed after the hits are
served to the client (the client, or the server can
explicitely shut them down, but usually will not). I assume
Tomcat is much the same. So the client will be able to get
more hits through an ESTABLISHED connection. The director,
using its own timeouts may not realise that the connection
has dropped and will send a new request (I think) to the
same realserver, although I expect the director should see
the SYN as a new connection request.
Are you using the latest kernel/ipvs?
Joe
--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!
| 