Subject: [lvs-users] Help with diagnosing performance issue 60K connections graphs provided

From: Malcolm Turnbull <[email-masked]>
Date: Thu, 12 Mar 2009 09:18:02 +0000

Hi all,

(sorry last post was in html by mistake)

I've been scratching my head a bit with a performance issue on a live
system and was wondering if anyone could point me in the right
direction for resolving?

I have a load balancer using 2.4.32 stock kernel running LVS with HTTP
& HTTPS in DR mode with 10 real servers each.
The load balancer is a 2.4GHz Celeron with 512Mb RAM, broadcom gigabit nics.

free -m shows 300MB used + 200MB cached (no swap file)

When the active connections climb to 60,000 and throughput is about
100MB+ we get a sudden and massive drop of in throughput...
We get a corresponding drop in system CPU and increase in Kernel CPU...
I have a selection of graphs here including throughput and cpu usage etc.

http://www.loadbalancer.org/test/

No errors in syslog relating to NICs or netfilter etc.

The actual number of connections is actually about 9K or 11K/sec
(Direct routing exagerating them), I was looking at using ipvsadm
--set to reduce the TCP timeout and therefore connection table size
but I don't think that the memory is an issue.

Any other ideas?

NB. I've tested similar hardware to 800MB+ and 50K cons/sec (obviously
with small/large packet sizes though not live data)

Many thanks in advance for any insight :-).

--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - [email-masked]
Send requests to [email-masked]
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

---

From: Graeme Fowler <[email-masked]>
Date: Thu, 12 Mar 2009 11:04:02 +0000

> 100MB+ we get a sudden and massive drop of in throughput...

That looks to me very much like an interrupt-handling issue. What
architecture are your NICs using - PCI, PCIe etc etc? Are they on board
using a common bus?

I'd surmise that either they're on a common PCI bus which is saturated,
or they're sharing an IRQ and just can't go any faster. Can you separate
them (either in BIOS or driver)?

You might want to look back at CPU affinity which was discussed
recently, although you say this a Celery so that throws out SMP magic as
a fix!

Graeme

---

From: Malcolm Turnbull <[email-masked]>
Date: Thu, 12 Mar 2009 16:43:10 +0000

Graeme,

Turned out it was a mixture of conntrack on the real servers filling up,

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.performance.html#conntrack_filling_tables

and fairly old hardware on the load balancer (I didn't realise just how old)....

Thanks.

-- 
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.

---

From: Graeme Fowler <[email-masked]>
Date: Thu, 12 Mar 2009 16:56:06 +0000

> Turned out it was a mixture of conntrack on the real servers filling up,

Gah.

Good practice: don't put LVS traffic into conntrack *unless you have
to*. "man iptables", look for the "raw" table and NOTRACK target:

NOTRACK
This  target disables connection tracking for all packets matching that
rule.

A Very Useful Thing Indeed, he said (adopting a Pooh-type voice).

Graeme