LVS
lvs-users
Google
 
Web LinuxVirtualServer.org

[lvs-users] IPVS-NAT: having to restart iptables daily to prevent 100% s

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: [lvs-users] IPVS-NAT: having to restart iptables daily to prevent 100% system CPU usage
From: daryl herzmann <akrherz@xxxxxxxxxxx>
Date: Wed, 9 Jul 2014 08:14:48 -0500 (CDT)
Hello,

I have a IPVS-NAT setup on RHEL6.5 64 bit (2.6.32-431.20.3.el6.x86_64). 
The system has 4 GB of memory (a Dell R200).  This setup primarily 
services HTTP to real-servers and peaks around 10,000 http req/sec. 
Anyway, I am finding that I need to restart iptables every few days to 
prevent system CPU usage from going to 100%.  When this happens, the 
cluster starts getting super sluggish and things grind to a halt.  Here's 
my current iptables

Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Table: nat
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    SNAT       all  --  192.168.0.0/24       0.0.0.0/0 
to:MY.PUBLIC.IP.HERE
2    MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Here's an example of 'sar' output when I reached near 100% system usage

04:20:01 PM     CPU     %user     %nice   %system   %iowait    %steal 
%idle
04:00:04 PM     all      0.36      0.00     98.50      0.02      0.00 
1.12
04:10:01 PM     all      0.71      0.00     98.87      0.03      0.00 
0.39
04:20:01 PM     all      0.38      0.00     95.10      0.09      0.00 
4.43
04:30:01 PM     all      0.38      0.00     51.68      0.14      0.00 
47.79
04:40:01 PM     all      0.37      0.00     47.86      0.12      0.00 
51.65
04:50:01 PM     all      0.34      0.00     48.89      0.12      0.00 
50.65

So I restarted iptables around 4:20 PM and you can see the %system usage 
drop in half and things returned to 'normal' when I did that.

I only have two setting changed in /etc/sysctl.conf

net.netfilter.nf_conntrack_max = 768288
net.ipv4.ip_forward = 1

I've noticed from 'iptables -L' output, that my inactive connections has 
reached around 1.3 million during peak usage.

As I write this email, my system usage percentage is about 30%, for 6,000 
http requests per second, 100 Mb/s bandwidth through the NAT and 
/proc/sys/net/netfilter/nf_conntrack_count reports 278515

Perhaps I am exhausting nf_conntrack_max ?  Googling around, perhaps I 
should configure iptables to NOTRACK the http(s) traffic?  Anybody with 
thoughts on what I am doing wrong and how I could support 1 Gb/s worth of 
http traffic (~ 20,000 req/sec) through this setup?

Maybe I should give up on NAT and figure out IPVS-DR? :)  NAT is just so 
convenient to use! :)

Thank you!
   daryl

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

<Prev in Thread] Current Thread [Next in Thread>
  • [lvs-users] IPVS-NAT: having to restart iptables daily to prevent 100% system CPU usage, daryl herzmann <=