|
Karl wrote:
>
> Derek Glidden wrote:
> >
> > Is it crazy that we should see _used_ (not just "allocated")
> > file-handles on a config like this grow into the thousands and nearing
> > the ten-thousands mark? Or does something somewhere, possibly one of
> > the RedHat-supplied packages, have a leak that I should try to track
> > down?
> >
>
> you might want to run lsof and see what process(es) are maintaining
> large numbers of open files. we had a problem similar to this that was
> caused by some errant logging in our code.
Here's the problem:
cat /proc/sys/fs/file-nr
16793 16555 262144
lsof | wc -l
394
So there's 16161 handles that the kernel is reporting as actively used
that don't show up from lsof. From what I understand, the numbers from
the proc filesystem are "file handles allocated, file handles in use,
and file-max" respectively. I may be wrong about these numbers, but
after the first crash, we increased file-max from default of 4096 to
16384, whereupon it ran for three whole days before crashing again with
"file-max" and "inode-max limit reached" errors, at which point we
increased both values to 262144 as it currently stands, so I'm betting
that we really are using 16K filehandles as it shows. The numbers also
seem to steadily increase over the life of the LVS, at least up to a
point, although I haven't been able to keep a close enough eye on the
box over a long enough period of time to see if it's based on number of
connections, number of ipvs/ipchains rulesets, lifetime of the box, etc.
Even if anyone else could look at their LVS box doing a good amount of
work redirecting lots of VIPs to lots of internal boxes, (rather than
just redirecting one VIP to lots of internal boxes) and see what
/proc/sys/fs/file-nr and /proc/sys/fs/inode-nr report, I'd appreciate
the extra datapoints to tell me if it's something in the LVS or
IPchains, ipportfw or somewhere else.
Thanks!
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
With Microsoft products, failure is not Derek Glidden
an option - it's a standard component. http://3dlinux.org/
Choose your life. Choose your http://www.tbcpc.org/
future. Choose Linux. http://www.illusionary.com/
|
