|
Hello,
On Fri, 9 Sep 2022, Jiri Wiesner wrote:
> On Fri, Sep 09, 2022 at 01:21:05AM +0300, Julian Anastasov wrote:
> > It is interesting to know what value for
> > IPVS_EST_TICK_CHAINS to use, it is used for the
> > IPVS_EST_MAX_COUNT calculation. We should determine
> > it from tests once the loops are in final form.
> > Now the limit increased a little bit to 38400.
> > Tomorrow I'll check again the patches for possible
> > problems.
>
> I couldn't wait so I have run tests on various machines and used the
> sched_switch tracepoint to measure the time needed to process one chain. The
> table contains a median time for processing one chain, the maximum time
> measured, the median divided by the number of CPUs and the time needed to
> process one chain if there were 1024 CPUs of that type in a machine:
> > NR CPU Time(ms) Max(ms) Time/CPU(ms) 1024
> > CPUs(ms)
> > 48 Intel Xeon CPU E5-2670 v3, 2 nodes 1.220 1.343 0.025
> > 26.027
> > 64 Intel Xeon Gold 6326, 2 nodes 0.920 1.494 0.014
> > 14.720
> > 192 Intel Xeon Gold 6330H, 4 nodes 3.957 4.153 0.021
> > 21.104
> > 256 AMD EPYC 7713, 2 NUMA nodes 3.927 5.464 0.015
> > 15.708
> > 80 ARM Neoverse-N1, 1 NUMA node 1.833 2.502 0.023
> > 23.462
> > 128 ARM Kunpeng 920, 4 NUMA nodes 3.822 4.635 0.030
> > 30.576
> I have to admit I was hoping the current IPVS_EST_CHAIN_DEPTH would work on
> machines with more than 1024 CPUs. If the max time values are used the time
> needed to process one chain on a 1024 CPU machine gets even closer to 40 ms,
> which it must not reach lest the estimates become inaccurate. I also have
> profiling data so I intend to look at the disassembly of
> ip_vs_estimation_kthread() to see which instructions take the most time. I
> will take a look at the v2 of the code on Monday.
v2 uses find_next_bit in for_each_set_bit which has
cost. But we should not be surprised, if 268ms are for 50000
estimators on 104 CPUs (I guess this is also the number of
possible CPUs we actually use), one estimator reads from
104 CPUs for 5.36 microsecs, we can conclude for 1024 CPUs
the following:
Num Est 104 CPU 1024 CPU
========================================
1 5.36us 53us
4 21us 211us
16 86us 845us
The v2 algorithm allows IPVS_EST_CHAIN_DEPTH to
be changed to var which we can determine based on the CPU
count, more CPUs will need more threads and we have CPUs
for them:
kd->chain_depth = max(1800 / num_possible_cpus(), 2);
Goals:
- chain time: sub-100 usec cond_resched rate
- tick time: 10% of max 40ms
CPUs Depth est_max_count Chain Time Tick Time
=========================================================
4 450 1080000 93us 4453us
16 112 268800 92us 4433us
104 17 40800 91us 4374us
1024 2 4800 106us 5066us
4096 2 4800 422us 20265us
Summary:
- For 4096 CPUs we can start 208 kthreads for 1000000 ests,
crazy :)
- 4096 CPUs need to be fast to go below these 20ms or we
should use chain with 1 estimator for 2048+ CPUs
If we track somehow when a stats is updated,
may be we can skip estimators that are idle for
some time, this can save CPU cycles for estimating
unused dests.
Also, I'm investigating the idea to use
task_rlimit(current, RLIMIT_NPROC) as kthread limit when
first service is added and to save it into
ipvs->est_max_threads.
Another idea is ip_vs_estimation_kthread not to
change add_row but ip_vs_start_estimator to consider instead
est_row for the same purpose but only when kd->est_count
becomes large, say 2 * IPVS_EST_TICK_CHAINS * kd->chain_depth.
The idea is to fill 2 ticks completely when small number
of estimators are added and to prefer est_row when
we exceed this threshold and prefer to spread the
estimators to more ticks by honouring the 2-second
initial timer.
For example:
if (kd->est_count >= 2 * IPVS_EST_TICK_CHAINS *
kd->chain_depth)
crow = READ_ONCE(kd->est_row);
else
crow = READ_ONCE(kd->add_row);
crow--;
...
Regards
--
Julian Anastasov <ja@xxxxxx>
|
