|
> -----Original Message-----
> From: lvs-users-bounces@xxxxxxxxxxxxxxxxxxxxxx [mailto:lvs-users-
> bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Neal B
> Sent: Friday, August 06, 2010 2:16 PM
> To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Subject: [lvs-users] Kernel Tuning for LVS
>
> Over the past day I have been searching the net for the definitive answer for
> the recommended tweaks to the kernel for ipvs. If you are doing this then
> change this, but that does not exist. :)
>
Hey,
I've done a few things to tune the network for large-traffic here, and I'm
going to share with you the values I set, what I set them to, and why. Please
remember YMMV and don't modify something unless you completely understand what
it does. In a nutshell, if you break stuff using these values, it's not my
fault :).
These values are on 2.6.27.46 -- I'm unsure as to what they do or translate to
in newer or older kernels.
SYSCTLS:
These three values make the ARP cache larger. This is not needed in all
environments, but generally only costs ram so it doesn't hurt to increase it:
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.neigh.default.gc_thresh2 = 4096
net.ipv4.neigh.default.gc_thresh1 = 1024
The next options are for TCP window scaling, I won't go into details because
there are so many internet howtos on how to tweak your window scaling, but here
are the places and what I usually set them to:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65535 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
Next, I always disable proxy arp (RHEL, by default, sets the kernel to arp any
IP on the box on any interface; usually this is undesirable and can break
things for edge routers) -- make sure to set these values for
net.ipv4.conf.[interface].arp_[ignore|announce] as well.
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
A couple of other important sysctls imo:
net.core.netdev_max_backlog = 2500 # Increases the receive queue in the kernel
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 86400 # lowers
default timeout (default: 5d - I think it's a little silly)
Non-SYSCTL tweaks:
/sbin/ifconfig eth[#] txqueuelen 2000 # increases the size of the transmit queue
Check to see what the max ring buffer size is for your NIC card (should be
listed under Pre-set Maximums):
ethtool -g eth[#]
If any of the "Current Hardware Settings" values are less than the maximums,
set them to the max:
ethtool -G eth[#] rx [value]
ethtool -G eth[#] tx [value]
The biggest tweak I can advise you to do though is to *run a good network
card*. I don't put a load balancer in production, regardless of load, unless
it's running an e1000[e] driver NIC. These drivers have proven themselves the
most stable of any I've run, and have an additional setting you can set to
lower interrupts on a high-load system:
(inside modprobe.conf)
options e1000[e] InterruptThrottleRate=3000,3000,3000,3000
This tells the e1000 driver to *never* make more than 3000 interrupts per
second. This is good if you want pure volume of traffic at the expense of a
little more latency (very little more, really).
I hope this has been helpful. Please let me know if you have any questions.
--
Jason Faulkner
Linux Engineer, Rackspace Email & Apps
jason.faulkner@xxxxxxxxxxxxx
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
