|
Hello,
On Sun, 11 Jun 2000, Vivek Raghunathan wrote:
> Hi there.
>
> I am a junior at the Indian Institute of Technology, Bombay and am working
> on this project where I am using LVS.
>
> The problem I am facing is this: how do I create a test environment in the
> lab where I would be able to simulate the real-world environment
> carefully. The setup I have been trying to test with is as follows: I have
> a LVS-DR setup with VIP and RIP's on the same network as the client. I
> have 3 web servers running Apache and configured to open a min of 20
> httpd, a max of 50 and 30 on startup. Also MaxClient = 500. These
> conditions I found increase processor + memory utilisation to almost 100%
> on a single machine.
>
> My machines are connected via a 10 MBPS Ethernet. With my LVS up and
> running, my performance drops (yes!) to about 75% of the value for a
> single machine. My guess is that the network is the bottlenect and that
> somehow the traffic in the network is gettting more bursty which is
> causing increased delays for a fraction of packets.
>
> The software I am using for testing is Apache Bench -- at a concurrency
> level of 100-500 and around 5000-50000 connections. Also, the maximum
> connection time reported by Apache Bench increases almost 3 fold when i
> run the lvs as opposed to a single web server. This makes me think that my
> suspicions about a fraction of requests taking too much time and pushing
> up the averages is true.
>
> My stats are as follows (figures are approx)
> to a single web server --> approx. 800 K/s throughput
> around 320 requests/sec
> to a lvs cluster of 3 machines ---> approx 600 K/s
> around 250 req/s
> Processing time increases by approx 70%. Connection time increases by
> almost 130%.
>
> Also, funny thing is that the kind of load I am generating is easily
> saturating the one web server, but hardly stresses the cluster which
> operates at around 80% processor idle. This again seems to suggest a
> bottleneck in the network(correct me if i am wrong). The machines I am
> using are all Linux Kernel 2.2.12-20 machines with apache 1.3.
> P-III/550MHZ + 128 MB RAM + Realtek 8029 (PCI) cards (pci-ne2k.c)
>
> COuld you please
> 1. suggest a better traffic generator than Apache Bench.. which reports
> too few diagnostics to be of much use.
The preferred test could be with network-only
oriented service. You need a server who don't waste CPU
in fork()s, disk operations, etc. Just plain network traffic.
> 2. suggest the kind of network configuration that would be best put to
> test the LVS-DR
Not on 10Mbps :)
> 3. give info as to the max kind of connections/sec or requests/sec that
> LVS can handle. (order of magnitude would be fine ... assuming I have
> access to reasonably fast processors .. say dual/quad processor boards,
> unlimited RAM blah blah. WOuld it be able to support 10000-50000
> connections/sec....
Better to flood with UDP packets. Some aspects of
the code can't be tested without spoofing the source address
and so creating large number of clients in the LVS table.
Testing with small number of TCP clients measures only
the CPU and network throughput. But there is connection table
management too. It requires memory and CPU. You have to
test it too because when the number of clients increases
the picture can be different.
But you need 100Mbits or even gigabits.
> 4. where, if there is one, would the bottleneck be and is there any way i
> can get around it, other than going ahead and developing a hardware
> implementation.
Everywhere :) CPU, Memory, Network. It depends on
their values. In the real world there is a delay from
the real servers too. If you test LVS you have to do it
with the ideal real server: only network traffic.
LVS don't distinguish between UDP and TCP
entries when managing them. You can create more clients in
the LVS eyes by using UDP. Starting TCP connections from
limited number of clients only allows the connections to
be reused. The result: there is a limit in the number of
clients when testing with TCP (except if you test in production
with many clients):
C * P = number of entries in the LVS connection table
C=Number of client hosts
P=Number of client ports (3976 by default, port=1024..4999)
So, if there is a UDP test suite, the test will be
more accurate. Of course, you can simulate the same
throughput for each UDP client as for TCP but you are not
limited for the number of client TCP connections.
So, VS/DR setup like this can help:
CLIENT
/ \
Director ---- Real Server
- Client generates UDP requests with spoofed src and port
- Client is def gw for the Real server (we need to grab
the replies for the spoofed packets in the CLIENT)
- The replies from the Real server to the spoofed addresses
are delivered locally in the client, you need to measure
the traffic. This can be done with policy routing, not with
transparent proxy, we need speed.
- You still can ignore the replies to save some CPU in the
CLIENT and only to measure the network throughput.
Just stop ip_forward. If you analyze the replies you have to use
only one source port. In other case you have to listen on all
ports in the clients. So, better to spoof only the source address
in the requests. But you still can listen for replies in 1024
ports in the client. But this can be a bottleneck if you try
to analyze them. We still test VS/DR. May be we even don't need
to generate replies from the real server? Only this traffic:
CLIENT -> Director -> Real server
> 5. add any further useful info.
I think, the above is enough :) May be the perfect
LVS test suite can include such features.
>
>
> This is the first time I am posting and I apologize if some of the above
> has been covered previously in this newsgroup. I went through some of the
> archives, but it is 7 in the morning now and I thought, what the hell
> anyway.
> sorry again if i caused any of you trouble.
>
>
> thanks.
>
> Vivek
Regards
--
Julian Anastasov <uli@xxxxxxxxxxxxxxxxxxxxxx>
|
