|
Hello,
Thanks a lot for your reply.
On Sun, 02 Jul 2006 21:19:37 +0200
Roberto Nibali <ratz@xxxxxxxxxxxx> wrote:
> Hello,
>
> > The default of /proc/sys/net/ipv4/ip_local_port_range is "32768 61000".
> > In short, 61000 - 32768 = 28232.
>
> Which has nothing to do with IPVS, normally. ip_local_port_range is for
> local sockets. IPVS does not do sockets. But let's check the rest of the
> email ...
>
> > The number of client of our test environment is one.
>
> Ok.
>
> > The hash key of ip_vs_conn_tab (connection table) is based on
> > protocol, s_addr(caddr), s_port(cport), d_addr(vaddr), and d_port(vport).
>
> Correct.
After all, the hash key of ip_vs_conn_tab is based on
protocol, s_addr(caddr), s_port(cport) in ip_vs_conn_in_get()
and protocol, d_addr(caddr), d_port(cport) in ip_vs_conn_out_get().
In short, referring to ip_vs_conn.c,
I think the hash key isn't based on vaddr/vport but caddr/cport.
> I've corrected your statement a bit:
> > So I think that the max amount of hash values produced by hash function
> > is 28232(default) for one client to same virtual server.
>
> Yes, this makes sense.
>
> > Therefore, I
> > think the limit of ActConn + InActConn for every client at a high load
> > exists and the number of hash key for ip_vs_conn_tab
> > from same client to same virtual server (to a realserver) is full.
>
> I don't follow you here anymore, I'm sorry. Where does this relation
> between act + inact connections to the hash table come from in the code?
I'm sorry, i can't understand about your comment.
I think the code includes in ip_vs_conn.c etc.
As far as checking "ipvsadm" or "ipvsadm -Lc" or "wc -l /proc/net/conn",
the max amount of hash values produced for one client is related to
the total amount of ActConn + InActConn.
>
> > So I think that strange behavior at a high load was occured by
> > the above reason.
>
> I have to go back and read the whole thread but this month I'm unable to
> do so.
>
> > In short, the cause of the load balancing trouble at a high load is mainly
> > related to ip_vs_conn table managed by hash key based on the above elements
> > and the limit of port range of a client
>
> Interesting observation, although right now I don't see the connection
> between the hash table and the amount of active and inactive
> connections. If your observation is correct, would you be able to
> perform following test conduct for me and report back, please?
>
> set ip_local_port_range to 10000-10100 and repeat your tests.
I tried to test again after setting ip_local_port_range
to 32768-32868 instead of 10000-10100.
(In short, 32868 - 32768 = 100. I think this setting is
the same meaning as your intention.)
[root@monkey1 conf]# ipvsadm
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.144.100:http rr
-> 192.168.50.102:http Masq 1 0 50
-> 192.168.50.101:http Masq 1 0 50
^[[A[root@monkey1 conf]# ipvsadm
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.144.100:http rr
-> 192.168.50.102:http Masq 1 1 49
-> 192.168.50.101:http Masq 1 0 50
[root@monkey1 conf]# ipvsadm
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.144.100:http rr
-> 192.168.50.102:http Masq 1 0 50
-> 192.168.50.101:http Masq 1 0 50
The above situation still continues at a high load.
In short, the total amount of ActiveConn + InActConn is 100.
>
> according to your statement it the maximum connection channels should be
> topped at around 100.
>
> > But I think that this specification of ip_vs is no problem in real
> > environment.
>
> I believe that if such a deficiency exists, it will very well be a
> problem. We still deploy 2.2.x kernel based systems where the local port
> range was set from 1024 to 5000.
>
> More questions below.
>
> >> ------------------------------------------------------------------------
> >> IP Virtual Server version 1.2.0 (size=4096)
> >> Prot LocalAddress:Port Scheduler Flags
> >> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> >> TCP 192.168.0.101:http rr
> >> -> rs02:http Masq 1 0 1
> >> -> rs01:http Masq 1 1 28229
>
> How slow is this machine? Did you fiddle around with TCP-related
> settings on the client?
>
> >> ------------------------------------------------------------------------
>
> Do you have netfilter modules loaded? Please don't do performance tests
> with any netfilter code unless you want to test netfilter. What does
> your /proc/net/ip_conntrack say in this situation?
>
> Regards,
> Roberto Nibali, ratz
> --
> echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' |
> _______________________________________________
Thanks.
Best regards,
--
Hideaki Kondo
|
