|
On Tue, 04 Jul 2006 17:10:31 +0900, Hideaki Kondo wrote:
>
> Hello,
>
> Thanks a lot for your reply and advice.
>
> On Mon, 3 Jul 2006 19:25:15 +0900 (JST)
> Horms <horms@xxxxxxxxxxxx> wrote:
>
>> In article <004001c69ddf$6c357b70$0200a8c0@PCGTR1B> you wrote:
>> > Hello,
>> >
>> > I almost found out the reason of the limit (about 28230)
>> > of ActConn + InActConn.
>> > The default of /proc/sys/net/ipv4/ip_local_port_range is "32768 61000".
>> > In short, 61000 - 32768 = 28232.
>> > The number of client of our test environment is one.
>>
>> As Ratz mention, this value does not affect LVS directly. However,
>> it might affect your clients. In particular if you only have one,
>> it may only be able to generate 61000 - 32768 connections.
>>
>> >
>> > The hash key of ip_vs_conn_tab (connection table) is based on
>> > protocol, s_addr(caddr), s_port(cport), d_addr(vaddr), and d_port(vport).
>> > So I think that the max limit of hash key produced by hash function
>> > is 28232(default) for one client to same virtual server.
>> > Therefore, I think the limit of ActConn + InActConn for every client
>> > at a high load exists and the number of hash key for ip_vs_conn_tab
>> > from same client to same virtual server (to a realserver) is full.
>>
>> Please remember that the hash key is just that, and that the hash
>> buckets can have multiple entries. Certainly if s_addr, d_addr and
>> d_port are constant, which I believe they are in your test, then the
>> number of keys will be bound by s_port (which is probably bound by the
>> limit discussed above). However for a more usual case, s_addr at least
>> varies and this gives a much larger bound of unique hash keys.
>
> I also think your idea is correct in a usual case.
>
>>
>> That said, the real bound on the hash keys is likely to be
>> CONFIG_IP_VS_TAB_BITS, which by default is 12, giving 2^12 = 2048
>> hash keys. But again, the buckets can have multiple entries.
>
> I had understood what you explained.
> I also had checked that IP_VS_CONN_TAB_SIZE of hash table is related to
> CONFIG_IP_VS_TAB_BITS.
> In short, IP_VS_CONN_TAB_SIZE is (1 << IP_VS_CONN_TAB_BITS) in ip_vs.h,
> namely IP_VS_CONN_TAB_SIZE is 2^13 = 4096.
> And it is used as an initial value by "ip_vs_conn_tab = vmalloc(
> IP_VS_CONN_TAB_SIZE*sizeof(struct list_head));" in ip_vs_conn.c .
> Then the hash entries of ip_vs_conn_tab can increase If necessary.
The intial vmalloc() just initialises the hash table. Each new
connection creates a new hash entry which is added to the hash
and later removed when the connection expires.
> However as far as a hash key of ip_vs_conn_tab is based on
> s_addr and s_port, I think the max limit of hash key for same client
> exists and the limit is related to ip_port_local_range of the client.
> But I think bringing the max limit isn't responsibility of the client.
> Because the client has only to reuse a port number of tcp connection
> without setting HTTP Keep Alive or using many many simultaneous
> connections.
I think that the number of ports that the client has available is
limited by ip_port_local_range. And I think that this is an important
limitation that is effecting your results.
However, I don't think it is a limitation that is effecting LVS.
As it can reuse hash-buckets if there is key-collision (which is
normal). Rather, I think that the limitation affects how many
connections the client can generate.
The important thing is, that a connection entry is looked up using the
following inputs, as you pointed out: caddr, cport, vaddr, vport
If caddr, vaddr and vport are fixed, which they are for your test,
then only cport is free. And cport is limited by ip_port_local_range.
But that limit applies to the client. The LVS limit on cport is
(2^16)-1 (not sure about the -1, because IPV4 ports are only 16 bits
long.
In any case, yes there is a limit. ip_port_local_range imposes a limit
on the client side. and IPV4 imposes annother (usually higher) limit
on the maximum number of simulatneous connections a client may have
open. That is, assuming the client only has one IP address.
--
Horms
H: http://www.vergenet.net/~horms/
W: http://www.valinux.co.jp/en/
|
