|
Hello guys,
I've been helping with debugging a high traffic web site that deploys
LVS_DR and is hitting IPVS limits :(. We've got to re-think our
statements about IPVS being as fast or close to routing speed. Also CPU
L1 cache is getting in our way, with Intel pushing those lousy
dual/quad-core CPUs. They run with 16KB shared i/d-cache cross CPU (avg
cache miss close to 1%!) and this is just not enough anymore to handle
the slab cache lookup for sites with a high number of packets per
second. I haven't been able to measure the DTLB invalidation rate but I
suspect it high enough. And it seems that the bucket size of the hash
table is also not so insignificant than anticipated and calculated
before (around 2001/02 before switching to Jenkins hash).
I'll share the details hopefully before I leave for my sabbatical.
Just one question: When did IPVS fatten to a 256 byte slab object size
for struct ip_vs_conn? This is fatal! (Joe will need to rewrite about
1432 pages of his Howto :)). Seriously: We need to put ip_vs_conn on a
diet. What about the idea of the comment at the end of the ip_vs_conn
structure definition in ../include/net/ip_vs.h:
/* Note: we can group the following members into a structure,
in order to save more space, and the following members are
only used in VS/NAT anyway */
struct ip_vs_app *app; /* bound ip_vs_app
object */
void *app_data; /* Application private
data */
struct ip_vs_seq in_seq; /* incoming seq. struct */
struct ip_vs_seq out_seq; /* outgoing seq. struct */
};
How big is struct ip_vs_conn actually? It's only a 256 byte object since
we must be slightely bigger than 128 bytes and do a HW_CACHE_ALIGN on
kmem_cache_create().
Please comment if you find time. Thanks and best regards,
Roberto Nibali, ratz
--
echo
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc
| 