LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

Re: [lvs-users] initial connection delay

To: "LinuxVirtualServer.org users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: [lvs-users] initial connection delay
From: Rob <ipvsuser@xxxxxxxxxxxxxxxx>
Date: Thu, 26 Jul 2007 01:04:30 -0700
Gerry Reno wrote:
> Right now it's just Apache. The only thing I have to go on is the httpd 
> access log:
> 192.168.2.102 - - [24/Jul/2007:19:59:54 -0400] "GET /gw/ HTTP/1.1" 302 -
> 192.168.2.102 - - [24/Jul/2007:19:59:55 -0400] "GET 
> /gw/login.php?cd=10&forward=%2Findex.php HTTP/1.1" 200 4742
> 192.168.2.102 - - [24/Jul/2007:20:00:50 -0400] "POST 
> /gw/login.php?forward=%2Findex.php HTTP/1.1" 302 2
> 192.168.2.102 - - [24/Jul/2007:20:01:27 -0400] "GET /gw/index.php 
> HTTP/1.1" 302 2
> 
> I just tried to login as another user and it delayed again. And when I 
> try it directly to the real server I have the same effect so it must be 
> something with apache. ... After some investigation I think this is an 
> application problem. It is mainly just one application (the one I was 
> testing of course) that appears to exhibit this behavior.

This is why I always recommend (and try myself) to use thttpd and a single 
image and hand coded html page to do initial _and_ ongoing testing.

So for instance, I'll run thttpd permanently on port 8088 or something so that 
if the app is going slow then you can test the timing to the thttpd and see 
what the response time is vs the "real" application.

For initial testing, I use the acme labelmaker
http://acme.com/labelmaker/
to create a different gif for each real server with the same name, server.gif
And then a simple <html><head></head><body>server X</body></html>
html file named server.html or something.

That way you can see the effects of the scheduler (rr or whatever) because 
you'll see "server X" or "server Y" depending on which server you hit but most 
importantly you don't get hung up with testing a complex multilayer system all 
at once. I know that Joe uses telnet in the howtos, I find thttpd easier since 
I can use a browser and can use thttpd to test from outside the firewall, too 
whereas with telnet I can't. And then once it is basically working, I can use 
web testing tools to stress test it a bit to check how the LVS/ipvs director, 
firewall and other parts of the path hold up since generally thttpd will be 
able to server more request per second than any app server.

Later, I save a logo (or some other image) that is used in the live app to the 
thttpd doc folder, so that I have an apples-to-apples way to compare 
performance. One nice thing is that then you are testing the same full path 
all the way to up to the last step - app server vs thttpd. Another is that 
thttpd is small enough that it can be run (in a testing capacity) on any 
system without impact on memory or cpu.

So then, when someone says "it" is slow, you can quickly identify which part 
of "it" that is actually slow or not. This is vital if you have different 
groups responsible for different parts of your systems/networks since it is 
never their part that is slow and handy for when "nobody" changes "anything" 
and suddenly performance changes radically.

BTW, here is the curlrc file I use to give me a complete breakdown of times 
for each phase of the web transaction:

%<---------------
w = "\t%{http_code}\t%{time_total}\t%{time_namelookup}\t%{time_connect}
\t%{time_pretransfer}\t%{time_redirect}\t%{time_starttransfer}
\t%{size_download}\t%{size_header}\t%{speed_download}\t%{url_effective}\n"
%<---------------

Here is the curl command line I have in cron to monitor changes in web 
performance:

curl -s -S -K /root/.curlrc -o /dev/null http://192.168.2.240/gw/logo.gif >> 
./gwtiming.out 2>&1

curl -s -S -K /root/.curlrc -o /dev/null http://192.168.2.240:8088/gw/logo.gif 
 >> ./gwthttpdtiming.out 2>&1

If anyone knows how to get the same level of detail from wget, please let me 
know.

Sorry if this repeats info from past posts/rants...

> Rob, Bill,
> Got no hits on the tcpdump.
> On DNS: no DNS, everything is IP.
> 
> Gerry


<Prev in Thread] Current Thread [Next in Thread>