|
Hi,
> We have had several servers kernel panic on a variety of different
> hardware, maybe one or two kernel panics a week on various servers. None
> of them are under high load. Some are running
> keepalived+ldirector/ipvsadm, and some plain firewalls just running
> keepalived and shorewall/iptables.
>
> Kernel panic: not syncing: fs/block_dev.c:396: spin_lock
> (fs/block_dev.c:c035fc40) already locked by fs/block_dev.c/440
Is that all of the stack dump? If not, please provide the whole output.
> It looks like I need to apply this patch to the kernel,
> http://lkml.org/lkml/2004/11/24/375 ... which has been included in the
> kernel.org kernels.
... if your oops really is related to the bug report you've mentioned
before.
> I'm sure the problem is, all these boxes that are
> crashing sometimes are running CentOS 4 which is a free version of
> RedHat Enterprise 4. They are all running the latest redhat/centos
> kernel, 2.6.9-22.0.1.ELsmp. Thanks to Redhat's insane back porting and
> weird version numbering I'm not quite sure which kernel.org kernel that
> equates to. Other than these occasional kernel panics everything is
> working great. I'm guessing Redhat isn't including that patch yet
> because I'm still seeing these kernel panics. I tried to check the
> redhat kernel source code but that leads me to a problem:
>
> The ipvs source code doesn't seem to be included when I install Redhat's
> kernel source. My folder
You need to install the kernel source rpm provided with your distribution.
> /usr/src/kernels/2.6.9-22.0.1.EL-smp-i686/net/ipv4/ipvs just contains
> two files, a Makefile and Kconfig ... I did find
> /usr/src/kernels/2.6.9-22.0.1.EL-smp-i686/include/net/ip_vs.h but its
> just a header file.
So this means in the default RHEL (similar to other distros) they only
have the kernel headers installed on the system; which is enough to be
able to compile third-party kernel modules or broken user space
applications. You need to install the kernel source rpm as well to get
the whole kernel source + patches applied by the RedHat engineers.
> 1) Think I could grab 2.6.14.3 (latest stable kernel from kernels.org)
> and borrow the net/ipv4/ipvs folder/files out of there and try using
> that to recompile the centos/redhat kernel?
You'd be lucky if it compiled, and really lucky if it ran ;).
> 2) Should I forget about Redhat's crazy kernel and just try using a
> kernel.org kernel on top of the existing centos/redhat installations?
> I'm afraid that has a high chance of breaking things though.
This is dependent on your use case: if you need RedHat support, you
should not do this, if you need special third-party modules, you should
not do this, ...
> 3) The LVS website does not seem to offer the 2.6 kernel module source
> code, it just says 'its already built in to the 2.6 kernel'... there is
> ipvsadm source for the 2.6 kernel but no plain ipvs source.
That's because it's been merged with mainline a while ago. So there is
no need to keep the source on the linux-vs.org webpage anymore.
> Any way I
> could just download the 2.6 ipvs module from someplace, make sure its
> patched patch and use it to recompile the centos/redhat kernel?
This you might be able to do, however, I'm not sure if either the API
changed or the hooks IPVS needs to the standard kernel framework are
provided by your kernel.
> 4) Any other solutions to solve the kernel panic other than that patch I
> found above? It's odd that we have some firewalls running keepalived but
> no ipvsadm/ldirector crashing though keepalived does seem to use ipvs
> even when it's not being a load balancer.
Show us such a complete oops please and maybe we can even rule out IPVS
:). If it really kills your kernel because of the missing patch you've
mentioned, than another suggestion would be to disable the tcp defense
setting in the proc-fs.
Best regards,
Roberto Nibali, ratz
--
echo
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc
|
