|
Hello,
I was too busy to reply.
Thank you very much for your detailed explanation.
I've understood what you explained.
I agree with you in that regard.
On Wed, 12 Jul 2006 19:38:52 +0900 (JST)
Horms <horms@xxxxxxxxxxxx> wrote:
> On Tue, 04 Jul 2006 17:10:31 +0900, Hideaki Kondo wrote:
> >
> > Hello,
> >
> > Thanks a lot for your reply and advice.
> >
> > On Mon, 3 Jul 2006 19:25:15 +0900 (JST)
> > Horms <horms@xxxxxxxxxxxx> wrote:
> >
> >> In article <004001c69ddf$6c357b70$0200a8c0@PCGTR1B> you wrote:
> >> > Hello,
> >> >
> >> > I almost found out the reason of the limit (about 28230)
> >> > of ActConn + InActConn.
> >> > The default of /proc/sys/net/ipv4/ip_local_port_range is "32768 61000".
> >> > In short, 61000 - 32768 = 28232.
> >> > The number of client of our test environment is one.
> >>
> >> As Ratz mention, this value does not affect LVS directly. However,
> >> it might affect your clients. In particular if you only have one,
> >> it may only be able to generate 61000 - 32768 connections.
> >>
> >> >
> >> > The hash key of ip_vs_conn_tab (connection table) is based on
> >> > protocol, s_addr(caddr), s_port(cport), d_addr(vaddr), and d_port(vport).
> >> > So I think that the max limit of hash key produced by hash function
> >> > is 28232(default) for one client to same virtual server.
> >> > Therefore, I think the limit of ActConn + InActConn for every client
> >> > at a high load exists and the number of hash key for ip_vs_conn_tab
> >> > from same client to same virtual server (to a realserver) is full.
> >>
> >> Please remember that the hash key is just that, and that the hash
> >> buckets can have multiple entries. Certainly if s_addr, d_addr and
> >> d_port are constant, which I believe they are in your test, then the
> >> number of keys will be bound by s_port (which is probably bound by the
> >> limit discussed above). However for a more usual case, s_addr at least
> >> varies and this gives a much larger bound of unique hash keys.
> >
> > I also think your idea is correct in a usual case.
> >
> >>
> >> That said, the real bound on the hash keys is likely to be
> >> CONFIG_IP_VS_TAB_BITS, which by default is 12, giving 2^12 = 2048
> >> hash keys. But again, the buckets can have multiple entries.
> >
> > I had understood what you explained.
> > I also had checked that IP_VS_CONN_TAB_SIZE of hash table is related to
> > CONFIG_IP_VS_TAB_BITS.
> > In short, IP_VS_CONN_TAB_SIZE is (1 << IP_VS_CONN_TAB_BITS) in ip_vs.h,
> > namely IP_VS_CONN_TAB_SIZE is 2^13 = 4096.
> > And it is used as an initial value by "ip_vs_conn_tab = vmalloc(
> > IP_VS_CONN_TAB_SIZE*sizeof(struct list_head));" in ip_vs_conn.c .
> > Then the hash entries of ip_vs_conn_tab can increase If necessary.
>
> The intial vmalloc() just initialises the hash table. Each new
> connection creates a new hash entry which is added to the hash
> and later removed when the connection expires.
>
> > However as far as a hash key of ip_vs_conn_tab is based on
> > s_addr and s_port, I think the max limit of hash key for same client
> > exists and the limit is related to ip_port_local_range of the client.
> > But I think bringing the max limit isn't responsibility of the client.
> > Because the client has only to reuse a port number of tcp connection
> > without setting HTTP Keep Alive or using many many simultaneous
> > connections.
>
> I think that the number of ports that the client has available is
> limited by ip_port_local_range. And I think that this is an important
> limitation that is effecting your results.
>
> However, I don't think it is a limitation that is effecting LVS.
> As it can reuse hash-buckets if there is key-collision (which is
> normal). Rather, I think that the limitation affects how many
> connections the client can generate.
>
> The important thing is, that a connection entry is looked up using the
> following inputs, as you pointed out: caddr, cport, vaddr, vport
>
> If caddr, vaddr and vport are fixed, which they are for your test,
> then only cport is free. And cport is limited by ip_port_local_range.
> But that limit applies to the client. The LVS limit on cport is
> (2^16)-1 (not sure about the -1, because IPV4 ports are only 16 bits
> long.
>
> In any case, yes there is a limit. ip_port_local_range imposes a limit
> on the client side. and IPV4 imposes annother (usually higher) limit
> on the maximum number of simulatneous connections a client may have
> open. That is, assuming the client only has one IP address.
>
> --
> Horms
> H: http://www.vergenet.net/~horms/
> W: http://www.valinux.co.jp/en/
Thanks a lot.
Best regards,
--
Hideaki Kondo
|
