|
Hello,
Jeremy Kusnetz wrote:
Not sure if this an LVS problem, but maybe someone here is smart enough to
at least get me looking in the right direction.
I'll try to act smart, not sure it will anything useful will happen.
My primary director box had been running kernel 2.4.7 with the LVS version
that was current at the time 2.4.7 came out. It was running perfectly
stable for at least 6 months, never a single problem.
Ok. 2.4.7 was stable.
I'm running LVS-NAT. There are 53 VIPs on the box, pointing to 6
Ok. You're running LVS-NAT with 53 VIPS redirecting to 6 RS.
The director consists of a SMP PIII 1gig, with 512 ram. Two built in Intel
nics, eth0 has the 53 VIPs, eth1 is the gateway to the realservers. Then
there is a 3com nic which is running heartbeat with our secondary director.
Ok, and the DGW for the director goes over eth0?
About a month ago I got an alert that the primary director was down. I
logged in, and it was up, I could ping out, I could ping the RIPs, but I
couldn't ping any of it's own interfaces, even the loopback. (the
realservers could ping those interfaces on the director) Pinging these
interfaces from the director gave me the following error:
ping: sendto: No buffer space available
Ok, I have some ideas on where you need to search at the bottom of my
reply. But thanks for the detailed report.
A reboot fixed the problem.
Hmm, routing cache overflow maybe.
This problem started happening more and more frequently, until it was
happening at least once a day, usually in the middle of the night.
Did you see an increased traffic over your load balancer also or did
nothing else increase in proportion to the annoying message so that you
could say "when A happens more frequently, those messages happen more
often too"?
I figured it was time to upgrade the kernel and LVS version. I upgraded to
2.4.19 and LVS 1.0.6. I hoped this would fix the problem, but it did not.
Ok, new kernel, no change of problem, I start suspecting your network
setup? Did you change any of this between the time when it worked and
now? Also, how do you do your failover/failback and do you set up LVS by
hand (an own script that calls ipvsadm) or by one of the various tools?
Next I swapped out all the hardware, everything but the drives, and the
cables. This box had the same amount of memory, but slightly slower CPUs,
800mhz, but I figured even those are probably overkill.
Yes :) ... well actually it depends on your Internet connection. If
you're having a 155Mbit/s Internet connection and such a big amount of
request you might want to reconsider your hardware.
This did not help either.
Ok, not the kernel and not the hardware.
I changed the driver for the intel NICs from eepro100, to the latest e100
from intel. I've always had problems with eepro100 drivers, but when I was
running the old version of LVS, it had problems with the e100 drivers. But
now with the latest version, e100 seems to work.
Thanks to Julian.
But alas this did not fix the problem either.
Boy, I really think then this must be your network topology setup somehow.
The only thing that had changed before I started upgrading everything was
the amount of VIPs on the director. A couple of realservers had been added
to the mix too, along with more RIPs. We had also added some iptable rules
to drop SMTP connections from some some external IPs that were really bad
spammers. This list grew to about 50 chains.
50 chains is nothing, start worrying when you have 4000+ chain entries.
After changing drivers to e100, and it not fixing the problem, I changed the
iptable rules to reject the packets instead of dropping them. This had a
slight change to the symptoms. Instead of not being able to ping any of
it's own interfaces on the director, I can no longer ping random RIPs, to
the point where I start losing services because the LVS can't forward
connections to those IPs.
Any other interesting kernlog output besides from the one in your subject?
I've now removed the iptable rules completely, hoping that was the cause.
Didn't help.
It must be the routing or some weird interface setup then IMHO.
The only thing I can do besides rebooting that helps is I can bring down and
up eth2, which is the heartbeat interface. Sometimes that will clear the
problem for a few hours or longer, but sometime it will only clear up the
Ok, this will flush the cache for eth2 as a side effect.
problem for a few seconds. I've now reverted to a cron job that brings down
and up this interface once a minute. It sort of helps, but I still end up
Could you add a 'ip route flush cache' call to your crontab instead and
see if the problem vanishes?
having to reboot. Bringing up and down the loopback sometimes works too for
a few seconds. I tried bring up and down the eth0 and eth1, but that didn't
seem to have any affect.
So you might have run into problems with routing related to lo and eth0.
You got me very much interested in being a test rabbit (or how is it
again in English?)
Is there some sort of tuning I need to do to the /proc file system. I have
Maybe. Let's find out the cause first.
no idea what else to do, I've upgraded everything I can think of.
One might think so.
Apparently it's not an issue of buggy software. Please help, being paged at
all hours of the night for the past few weeks is getting really old! If you
can't help, what would be a good list to post this question to?
I think this question is ok here, although maybe the netdev mailinglist
is better. But lets first get some more information so when you hit that
list, you can write a very detailed explanation of what happens and what
you already tried. Maybe be doing that we find the real cause for your
problems.
Analysis:
=========
This is a very vague analysis of the above mentioned problem of the
reoccurring "ping: sendto: no buffer space available". It seems to show
up on a more or less regular basis. Let's delve into it ...
A. Possible Causes for this Problem
-----------------------------------
o fragmentation
You might have a big amount of fragmented packets coming in an your
buffer space simply can't cope with it anymore before all packets are
assembled.
o MTU(route output) < MTU(needed)
In case this occurs, the kernel needs to fragment the packets and
send them off then causing a lot of skb_alloc() and memory allocations
in that direction.
o wrong default gateway configuration
You might run your box with a wrong DGW pointing to an internal IP
address configured on your box. The reason why it still works could be
an intelligent router before your box.
o neigbour table overflow
Your arp cache experienced some kind of overflow due to strange
settings in /proc/sys/net/ipv4/neigh/<$INTERFACE>/* or due to circular
routing entries.
o generally a weird /proc/sys/net/* configuration
It could be anything there, let's hope it's not there.
B. How to Hunt Down those Problems
----------------------------------
I need the following outputs _when_ the problems occurs:
o ip -s -s link show
o ip addr show
o ip link show dev lo
o ip rule show
o ip route show table main
o cat /proc/net/softnet_stat
o cat /proc/net/rt_cache_stat
o cat /proc/net/raw (especially after a ping)
o cat /proc/net/netlink
o cat /proc/slabinfo
Could you also test if starting a tcpdump on either one of the
interfaces triggers the problem somehow in that that the error messages
show up more frequently?
C. Possible Band-Aid for the Problem
------------------------------------
o reboot (this is not really a solution)
o crontab that sets link state down/up (also not very nice)
o crontab that does 'ip route flush cache'
I hope you can do something with my information and that it will reveil
a hint on where to search next. Sorry that we don't have a solution for
this right now.
Best regards and thank you for your detailed description,
Roberto Nibali, ratz
--
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc
| 