Re: System crash

To: " users mailing list." <lvs-users@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: System crash
From: Sébastien BONNET <sebastien.bonnet@xxxxxxxxxxx>
Date: Mon, 18 Jul 2005 11:09:34 +0200
Hi folks,

Here is the latest news about this (still) unsolved problem. I've reactivated IPVS a few days ago and machines are crashing every 3 days.

Since sept. 2004, a few changes have been made : now running 2.6.10, UP instead of SMP. Still using bcm5700 NIC drivers. Kernel args : acpi=off. I've just applyed pci=noapic (as initially proposed) but it's not yet used (server not yet rebooted).

The serial console is now operationnal. Unfortunately, Alt-SysRq was also panic'ed this morning, I was only able to save a screen height of kernel dump :

update_defense_level+0x10/0x372 [ip_vs]
defense_timer_handler+0x0/0x29 [ip_vs]
defense_timer_handler+0x5/0x29 [ip_vs]
Kernel panic - not syncing: fs/block_dev.c:396: spin_lock(fs/block_dev.c:c035d880) already locked by fs/block_dev.c/396

As one can see, it's still talking about ip_vs defense something. Running ksymoops on the above is not significant :

[root@martinique ksymoops-2.4.11]# ./ksymoops -m /boot/ < ~/ksymoops.log

ksymoops 2.4.11 on i686 2.6.10-1.771_FC2.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.6.10-1.771_FC2/ (default)
     -m /boot/ (specified)

Error (regular_file): read_ksyms stat /proc/ksyms failed
./ksymoops: No such file or directory
No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Kernel panic - not syncing: fs/block_dev.c:396: spin_lock(fs/block_dev.c:c035d880) already locked by fs/block_dev.c/396

1 error issued.  Results may not be reliable.

I'm setting up a permanent serial console, recording every kernel message. I'm hoping to get a full kernel panic dump in less than 3 days.

If some of you have ideas in the meantime ...


Sébastien BONNET wrote:
Hi list,

I've been using IVPS 1.0.x for years now and I never had any problem.

I recently built a new cluster, running Fedora Core 2 and IPVS 1.2.0. Since then, I'm having big problems. Eventhough the servers are not loaded, the director periodically crashed.

No more network, no more interactive console access ... the only thing left is Alt-SysRq ! No single char logged (to disk or to the console) that could help :(

Alt-SysRq-p has recently shown that something related to ipvs tcp defense was found in the stack trace. Unfornunately, I wasn't able to read enough infos as they scroll way too fast.

Right now, I don't know what I can try and test ? I've changed the NIC driver, I've stopped the backup director, I've just stopped the sync daemon ... and still hoping it won't crash again. If it crashes again, I'll run without ipvs for a few days to see if it's really the cause.

Has anybody already faced such a situation ? Any help appreciated.

Kind regards.

[root@frioul root]# uname -a
Linux frioul 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004 i686 i686 i386 GNU/Linux

[root@hawai root]# ipvsadm -ln
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  a.b.c.78:80 wrr
  -> a.b.c.104:80             Local   981    0          0
TCP  a.b.c.79:80 wrr
  -> a.b.c.104:80             Local   981    0          0
TCP  a.b.c.74:80 wrr
  -> a.b.c.104:80             Local   1000   0          0
TCP  a.b.c.78:443 wrr
  -> a.b.c.104:443            Local   1000   0          0
TCP  a.b.c.79:443 wrr
  -> a.b.c.104:443            Local   1000   0          0
TCP  a.b.c.74:443 wrr
  -> a.b.c.104:443            Local   1000   0          70

Setup : 2 servers with one NIC each, both acting as director and realserver, forwarding method is Direct Routing, MAC pb solved using net.ipv4.conf.{lo,all}.arp_{ignore,announce} The above ipvsadm output shows the current situation with only one active server as I investigate on the other one.

Sébastien BONNET     --    Ingénieur système
Tel:      GSM:

<Prev in Thread] Current Thread [Next in Thread>