Subject: [lvs-users] LVS performance on SMP

From: New User <[email-masked]>
Date: Mon, 2 Mar 2009 21:45:12 -0800 (PST)

Hi,

We are currently running dedicated LVS-NAT (with the exception of iptables):
 Intel Xeon 3060 @ 2.4GHz (2 cores)
 2G RAM
 2 Gb/s NIC (one external, one internal)
 LVS-NAT
 Linux 2.6.18 kernel

It appears we are running out of CPU (usage reached 160, with 40 idle, 2 cores) when we reach:
 ~10K CPS
 ~100K InPPS
 ~120K OutPPS
 ~1 million+ active and number of objects in the ip_vs_conn entry in /proc/slabinfo

We are considering upgrading the setup to the following:
 2 x Intel Xeon 5430 (2 CPUs, 4 cores each)
 8G RAM
 8 Gb/s NIC (4x1Gb in bonding for external, 4x1Gb in bonding for internal)
 LVS-NAT
 Linux 2.6.18 (possible newer if it offers performance increase)

The question is how well can the LVS code take advantages of the multiple CPU. There are some conflicting answers to this question on the FAQ and mailing list (in gernal, it helps with the system as a whole). In our actual usage experience, it appears SMP helps, since it did reach 160 CPU usage. However, I do not know if this is real or just an illusion. I do not know if LVS is actually doing real work that caused the usage to be above 100 or is it because of SMP that causing locks that resulted that. Also, in our limited testing of the new setup, we find the SMP helps because with 8 NICs, just to process the interrupts can overwhelm a single CPU (core).

Can anyone share their knowledge or experience for the following:
 1. Can LVS increase linearly in capacity when we go from the setup we currently have (2 cores with 2 NICs) to what we are considering (8 cores, with 8 NICs)
 2. Has anyone else used similiar setup that can share their experience? There was a posting to the mailing list in Dec. 2007 that indicated the performance limit is ~450K (don't know if it's the sum of InPPS and OutPPS) pps. There was no conclusive answer as to this was the limit of LVS or not
 3. Has anyone have a setup that can process 1+ million pps (LVS-NAT) on a single LVS machine (we already use multipel LVS with keepalived in a live-live configuration)?

Thanks

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - [email-masked]
Send requests to [email-masked]
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

---

From: [email-masked]
Date: Tue, 3 Mar 2009 02:07:36 -0500 (EST)

-----Original Message-----
From: "New User" <[email-masked]>
Sent: Tuesday, March 3, 2009 12:45am
To: [email-masked]
Subject: [lvs-users] LVS performance on SMP

==============================

I'd be willing to bet that your problem lies in "soft IRQ %"... basically, you're CPU is getting eaten up by NIC interrupts moreso than anything LVS/iptables related. 

I've yet to find a good solution to this; if there is one I'm very keen to hear it.

--
Jason Faulkner 
Linux Systems Engineer
Mailtrust, a division of Rackspace

---

From: "Kit Gerrits" <[email-masked]>
Date: Tue, 3 Mar 2009 09:59:14 +0100

Certain NICs handle more of the IP processing themselves, but require kernel
support for this.
(I've seen NICs that will handle most of iSCSI themselves)
I suggest loking up the Linux Kernel or Networking mailing lists.

I don't know if the system balances IRQ handling over multiple cores.
If so, upgrading to quadcore should linearly scale your performance.

Regards,

Kit 

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: dinsdag 3 maart 2009 8:08
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] LVS performance on SMP

-----Original Message-----
From: "New User" <[email-masked]>
Sent: Tuesday, March 3, 2009 12:45am
To: [email-masked]
Subject: [lvs-users] LVS performance on SMP

Can anyone share their knowledge or experience for the following:
 1. Can LVS increase linearly in capacity when we go from the setup we
currently have (2 cores with 2 NICs) to what we are considering (8 cores,
with 8 NICs)
 2. Has anyone else used similiar setup that can share their experience?
There was a posting to the mailing list in Dec. 2007 that indicated the
performance limit is ~450K (don't know if it's the sum of InPPS and
OutPPS) pps. There was no conclusive answer as to this was the limit of LVS
or not
 3. Has anyone have a setup that can process 1+ million pps (LVS-NAT) on a
single LVS machine (we already use multipel LVS with keepalived in a
live-live configuration)?

I'd be willing to bet that your problem lies in "soft IRQ %"... basically,
you're CPU is getting eaten up by NIC interrupts moreso than anything
LVS/iptables related. 

I've yet to find a good solution to this; if there is one I'm very keen to
hear it.

---

From: Graeme Fowler <[email-masked]>
Date: Tue, 03 Mar 2009 09:28:40 +0000

> If so, upgrading to quadcore should linearly scale your performance.

http://www.cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt

Although slightly dated now, the basics in there still apply. You can
make a given interrupt (or device driving interrupts) have an affinity
with a single processor core.

Generally speaking, interrupts are shared more or less equally across
all cores so more cores should scale performance well; however in some
circumstances it can be beneficial to assign a given interrupt driver to
a specific core so you then know where the load on that core is coming
from.

I should point out that the OP stated a "usage" (which I assume to mean
% CPU utilisation) of 160 - that's a good thing. There's no point having
a totally idle CPU!

When utilisation hit 160%, what was the 1/5/15 minute load average?

Graeme

---

From: Sashi Kant <[email-masked]>
Date: Tue, 3 Mar 2009 10:01:25 -0800

Your CPU usage seems very high for this traffic. Quite possibly due to  
different configuration you may be running.

It will be interesting to see if you haven't dropped incoming traffic  
at this level of PPS. Please let us know if that is the case.

We run 2 NICs in bonding mode (Active/Passive) for both incoming and  
outgoing traffic.

Our CPU level are very low ~5% (all of it is system) at 174K in and  
174K out PPS. We run ldirector & heartbeat in DR mode. Hardware is  
comparable to yours. We had an issue of machine dropping packets when  
we were using boradcom nics. We replaced them by intel cards and  
adjusted ring parameters using ethtool.

We do not do iptables on the system and that could be the one eating  
up CPU, you may want to take a look at these parameters. Does kern.log  
says kernel table full at such level of traffic ?

# Adjust iptables related timeout to reduce number of connection in  
kernel tables
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait=10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait=10

Good luck and share your findings here on the list.

-Sashi

On Mar 2, 2009, at 9:45 PM, New User wrote:

> The question is how well can the LVS code take advantages of the  
> multiple CPU. There are some conflicting answers to this question on  
> the FAQ and mailing list (in gernal, it helps with the system as a  
> whole). In our actual usage experience, it appears SMP helps, since  
> it did reach 160 CPU usage. However, I do not know if this is real  
> or just an illusion. I do not know if LVS is actually doing real  
> work that caused the usage to be above 100 or is it because of SMP  
> that causing locks that resulted that. Also, in our limited testing  
> of the new setup, we find the SMP helps because with 8 NICs, just to  
> process the interrupts can overwhelm a single CPU (core).
>
>  1. Can LVS increase linearly in capacity when we go from the setup  
> we currently have (2 cores with 2 NICs) to what we are considering  
> (8 cores, with 8 NICs)
>  2. Has anyone else used similiar setup that can share their  
> experience? There was a posting to the mailing list in Dec. 2007  
> that indicated the performance limit is ~450K (don't know if it's  
> the sum of InPPS and OutPPS) pps. There was no conclusive answer as  
> to this was the limit of LVS or not
>  3. Has anyone have a setup that can process 1+ million pps (LVS- 
> NAT) on a single LVS machine (we already use multipel LVS with  
> keepalived in a live-live configuration)?
>
> Thanks
>
>
>
>