Re: [RFC PATCH 0/4] Use kthreads for stats

To: Julian Anastasov <ja@xxxxxx>
Subject: Re: [RFC PATCH 0/4] Use kthreads for stats
Cc: Simon Horman <horms@xxxxxxxxxxxx>, lvs-devel@xxxxxxxxxxxxxxx, yunhong-cgl jiang <xintian1976@xxxxxxxxx>,
From: Jiri Wiesner <jwiesner@xxxxxxx>
Date: Thu, 8 Sep 2022 20:32:34 +0200
On Thu, Sep 08, 2022 at 05:35:21PM +0200, Jiri Wiesner wrote:
> There is an alternative design where you could increase kd->est_max_count for 
> all kthreads once all of the available kthreads have kd->est_max_count 
> estimators. Nevertheless, there would also have to be a limit to the value of 
> kd->est_max_count. Imagine the estimation during a single tick would take so 
> long that the gap variable in ip_vs_estimation_kthread() would become 
> negative. You would need to have circa 250,000 estimators per kthread. Since 
> you are already measuring the timeout you need for schedule_timeout() in 
> ip_vs_estimation_kthread(), it should be possible to set the 
> kd->est_max_count limit based on the maximum processing time per chain. It 
> could be half a IPVS_EST_TICK, for example.
> But it seems to me that the alternative design - increasing kd->est_max_count 
> - should have some support in what is used in production. Are there servers 
> with more than 983,040 estimators (which would be IPVS_EST_MAX_COUNT * 30 
> kthreads) or even one third of that?

I did some profiling (but could have just looked at top, actually) of a kthread 
with IPVS_EST_MAX_COUNT estimators for 100 seconds:
# Samples: 4K of event 'bus-cycles'
# Event count (approx.): 125024900
# Overhead        Period  Command          Shared Object      Symbol
# ........  ............  ...............  .................  
    76.44%      95570475  ipvs-e:0:0       [kernel.kallsyms]  [k] 
     8.75%      10935925  ipvs-e:0:0       [kernel.kallsyms]  [k] _find_next_bit
     3.18%       3978975  swapper          [kernel.kallsyms]  [k] intel_idle
     1.00%       1251250  ipvs-e:0:0       [kernel.kallsyms]  [k] 
     0.36%        450450  swapper          [kernel.kallsyms]  [k] _raw_spin_lock
     0.36%        450450  swapper          [kernel.kallsyms]  [k] 

The bus-cycles event on this particular machine makes 25,000,000 events per 
second. Based on the period in the profile, the CPU utilization for various 
functions is:
ip_vs_estimation_kthread: 95570475/100/25000000*100 = 3.82%
_find_next_bit: 10935925/100/25000000*100 = 0.44%
_raw_spin_lock_bh: 1251250/100/25000000*100 = 0.05%

The kthread could definitely utilize the CPU more, which is an argument for 
increasing kd->est_max_count.

Jiri Wiesner

<Prev in Thread] Current Thread [Next in Thread>