Re: No buffer space available

To: lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Subject: Re: No buffer space available
Cc: Jeremy Kusnetz <JKusnetz@xxxxxxxx>
From: Roberto Nibali <ratz@xxxxxxxxxxxx>
Date: Mon, 30 Sep 2002 10:33:53 +0200

Jeremy Kusnetz wrote:
Not sure if this an LVS problem, but maybe someone here is smart enough to
at least get me looking in the right direction.

I'll try to act smart, not sure it will anything useful will happen.

My primary director box had been running kernel 2.4.7 with the LVS version
that was current at the time 2.4.7 came out.  It was running perfectly
stable for at least 6 months, never a single problem.

Ok. 2.4.7 was stable.

I'm running LVS-NAT.  There are 53 VIPs on the box, pointing to 6

Ok. You're running LVS-NAT with 53 VIPS redirecting to 6 RS.

The director consists of a SMP PIII 1gig, with 512 ram.  Two built in Intel
nics, eth0 has the 53 VIPs, eth1 is the gateway to the realservers.  Then
there is a 3com nic which is running heartbeat with our secondary director.

Ok, and the DGW for the director goes over eth0?

About a month ago I got an alert that the primary director was down.  I
logged in, and it was up, I could ping out, I could ping the RIPs, but I
couldn't ping any of it's own interfaces, even the loopback.  (the
realservers could ping those interfaces on the director) Pinging these
interfaces from the director gave me the following error:
ping: sendto: No buffer space available

Ok, I have some ideas on where you need to search at the bottom of my reply. But thanks for the detailed report.

A reboot fixed the problem.

Hmm, routing cache overflow maybe.

This problem started happening more and more frequently, until it was
happening at least once a day, usually in the middle of the night.

Did you see an increased traffic over your load balancer also or did nothing else increase in proportion to the annoying message so that you could say "when A happens more frequently, those messages happen more often too"?

I figured it was time to upgrade the kernel and LVS version.  I upgraded to
2.4.19 and LVS 1.0.6.  I hoped this would fix the problem, but it did not.

Ok, new kernel, no change of problem, I start suspecting your network setup? Did you change any of this between the time when it worked and now? Also, how do you do your failover/failback and do you set up LVS by hand (an own script that calls ipvsadm) or by one of the various tools?

Next I swapped out all the hardware, everything but the drives, and the
cables.  This box had the same amount of memory, but slightly slower CPUs,
800mhz, but I figured even those are probably overkill.

Yes :) ... well actually it depends on your Internet connection. If you're having a 155Mbit/s Internet connection and such a big amount of request you might want to reconsider your hardware.

This did not help either.

Ok, not the kernel and not the hardware.

I changed the driver for the intel NICs from eepro100, to the latest e100
from intel.  I've always had problems with eepro100 drivers, but when I was
running the old version of LVS, it had problems with the e100 drivers.  But
now with the latest version, e100 seems to work.

Thanks to Julian.

But alas this did not fix the problem either.

Boy, I really think then this must be your network topology setup somehow.

The only thing that had changed before I started upgrading everything was
the amount of VIPs on the director.  A couple of realservers had been added
to the mix too, along with more RIPs.  We had also added some iptable rules
to drop SMTP connections from some some external IPs that were really bad
spammers.  This list grew to about 50 chains.

50 chains is nothing, start worrying when you have 4000+ chain entries.

After changing drivers to e100, and it not fixing the problem, I changed the
iptable rules to reject the packets instead of dropping them.  This had a
slight change to the symptoms.  Instead of not being able to ping any of
it's own interfaces on the director, I can no longer ping random RIPs, to
the point where I start losing services because the LVS can't forward
connections to those IPs.

Any other interesting kernlog output besides from the one in your subject?

I've now removed the iptable rules completely, hoping that was the cause.
Didn't help.

It must be the routing or some weird interface setup then IMHO.

The only thing I can do besides rebooting that helps is I can bring down and
up eth2, which is the heartbeat interface.  Sometimes that will clear the
problem for a few hours or longer, but sometime it will only clear up the

Ok, this will flush the cache for eth2 as a side effect.

problem for a few seconds.  I've now reverted to a cron job that brings down
and up this interface once a minute.  It sort of helps, but I still end up

Could you add a 'ip route flush cache' call to your crontab instead and see if the problem vanishes?

having to reboot.  Bringing up and down the loopback sometimes works too for
a few seconds.  I tried bring up and down the eth0 and eth1, but that didn't
seem to have any affect.

So you might have run into problems with routing related to lo and eth0. You got me very much interested in being a test rabbit (or how is it again in English?)

Is there some sort of tuning I need to do to the /proc file system.  I have

Maybe. Let's find out the cause first.

no idea what else to do, I've upgraded everything I can think of.

One might think so.

Apparently it's not an issue of buggy software.  Please help, being paged at
all hours of the night for the past few weeks is getting really old!  If you
can't help, what would be a good list to post this question to?

I think this question is ok here, although maybe the netdev mailinglist is better. But lets first get some more information so when you hit that list, you can write a very detailed explanation of what happens and what you already tried. Maybe be doing that we find the real cause for your problems.

This is a very vague analysis of the above mentioned problem of the reoccurring "ping: sendto: no buffer space available". It seems to show up on a more or less regular basis. Let's delve into it ...

A. Possible Causes for this Problem
o fragmentation
  You might have a big amount of fragmented packets coming in an your
  buffer space simply can't cope with it anymore before all packets are
o MTU(route output) < MTU(needed)
  In case this occurs, the kernel needs to fragment the packets and
  send them off then causing a lot of skb_alloc() and memory allocations
  in that direction.
o wrong default gateway configuration
  You might run your box with a wrong DGW pointing to an internal IP
  address configured on your box. The reason why it still works could be
  an intelligent router before your box.
o neigbour table overflow
  Your arp cache experienced some kind of overflow due to strange
  settings in /proc/sys/net/ipv4/neigh/<$INTERFACE>/* or due to circular
  routing entries.
o generally a weird /proc/sys/net/* configuration
  It could be anything there, let's hope it's not there.

B. How to Hunt Down those Problems
I need the following outputs _when_ the problems occurs:
o ip -s -s link show
o ip addr show
o ip link show dev lo
o ip rule show
o ip route show table main
o cat /proc/net/softnet_stat
o cat /proc/net/rt_cache_stat
o cat /proc/net/raw (especially after a ping)
o cat /proc/net/netlink
o cat /proc/slabinfo

Could you also test if starting a tcpdump on either one of the interfaces triggers the problem somehow in that that the error messages show up more frequently?

C. Possible Band-Aid for the Problem
o reboot (this is not really a solution)
o crontab that sets link state down/up (also not very nice)
o crontab that does 'ip route flush cache'

I hope you can do something with my information and that it will reveil a hint on where to search next. Sorry that we don't have a solution for this right now.

Best regards and thank you for your detailed description,
Roberto Nibali, ratz
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc

<Prev in Thread] Current Thread [Next in Thread>