|
Hi! :)
We've been having a large number of LVS boxes fall over (hang) since
upgrading from 2.6.35 or earlier to 2.6.37. They seem to hard lock with
nothing on the console. The ones I've enabled nmi_watchdog on haven't
died yet, but I did enable remote syslog and managed to catch the kernel
log below.
It seems to be spinning on &dest->dst_lock. On this pair that this was
recorded on, we have keepalived running with a MISC_CHECK script that
actually fiddles with the weights with an ipvsadm command, hence the
"ipvsadm" command running, but we have also seen hangs on other master
and backup nodes without such scripts. I'm not sure if this is a useful
trace or not, since it has all '?'. There seem to be not many changes
except quite a few locking changes around the new features introduced in
2.6.37, and then not much since. Any ideas?
Simon-
[14315.510017] BUG: soft lockup - CPU#0 stuck for 67s! [ipvsadm:8382]
[14315.510104] Modules linked in: ip_vs_rr ip_vs_lblc ip_vs nf_conntrack
xt_hashlimit ts_kmp xt_string xt_recent ipmi_devintf ipmi_si ipmi_msghandler
bnx2
[14315.510111] CPU 0
[14315.510112] Modules linked in: ip_vs_rr ip_vs_lblc ip_vs nf_conntrack
xt_hashlimit ts_kmp xt_string xt_recent ipmi_devintf ipmi_si ipmi_msghandler
bnx2
[14315.510118]
[14315.510121] Pid: 8382, comm: ipvsadm Not tainted 2.6.37-hw #8
01V648/PowerEdge R410
[14315.510122] RIP: 0010:[<ffffffff816b3f66>] [<ffffffff816b3f66>]
_raw_spin_lock+0x16/0x20
[14315.510128] RSP: 0018:ffff8800cee037d0 EFLAGS: 00000293
[14315.510130] RAX: 000000000000fffd RBX: ffff8800cee037d0 RCX: 0000000000000000
[14315.510131] RDX: 000000001a204740 RSI: ffff88011fe83200 RDI: ffff88011fe832dc
[14315.510133] RBP: ffffffff8100b853 R08: 0000000000000003 R09: ffff8800cee038b0
[14315.510135] R10: ffff880111b3cf98 R11: 0000000000000000 R12: ffff8800cee03750
[14315.510136] R13: ffff88011fe83200 R14: ffff88012a2b0c54 R15: ffffffff8102a5a7
[14315.510138] FS: 00007eff00e526e0(0000) GS:ffff8800cee00000(0000)
knlGS:0000000000000000
[14315.510140] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[14315.510142] CR2: 00007eff00805b40 CR3: 0000000110a90000 CR4: 00000000000006f0
[14315.510144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14315.510146] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[14315.510147] Process ipvsadm (pid: 8382, threadinfo ffff8801106fe000, task
ffff880110f11640)
[14315.510149] Stack:
[14315.510209] ffff8800cee038b0 ffffffffa008370d ffff88012a2b0c68
1a20474000000003
[14315.510212] 000000081fcab800 0000000000000246 ffff8800cee03850
ffffffff815e02e1
[14315.510215] ffff88011fcab800 0000000000000000 ffff8800cee03890
0000000000000000
[14315.510218] Call Trace:
[14315.510279] <IRQ>
[14315.510388] [<ffffffffa008370d>] ? __ip_vs_get_out_rt+0x4d/0x380 [ip_vs]
[14315.510391] [<ffffffff815e02e1>] ? dev_queue_xmit+0x171/0x460
[14315.510395] [<ffffffff810647a6>] ? lock_timer_base+0x36/0x70
[14315.510400] [<ffffffffa0084b60>] ? ip_vs_dr_xmit+0x50/0x420 [ip_vs]
[14315.510404] [<ffffffffa007cdfa>] ? ip_vs_in+0x15a/0x5a0 [ip_vs]
[14315.510407] [<ffffffffa007c3e5>] ? ip_vs_out+0x675/0x9b0 [ip_vs]
[14315.510411] [<ffffffffa007d28e>] ? ip_vs_remote_request4+0xe/0x10 [ip_vs]
[14315.510414] [<ffffffff815fe8f7>] ? nf_iterate+0x67/0xa0
[14315.510417] [<ffffffff81608a50>] ? ip_local_deliver_finish+0x0/0x1d0
[14315.510420] [<ffffffff815fe9db>] ? nf_hook_slow+0xab/0x100
[14315.510422] [<ffffffff81608a50>] ? ip_local_deliver_finish+0x0/0x1d0
[14315.510425] [<ffffffff81608460>] ? ip_rcv_finish+0x0/0x310
[14315.510427] [<ffffffff81608c75>] ? ip_local_deliver+0x55/0xa0
[14315.510430] [<ffffffff816085b2>] ? ip_rcv_finish+0x152/0x310
[14315.510433] [<ffffffff816089de>] ? ip_rcv+0x26e/0x2e0
[14315.510436] [<ffffffff815dd833>] ? __netif_receive_skb+0x433/0x520
[14315.510439] [<ffffffff815dd885>] ? __netif_receive_skb+0x485/0x520
[14315.510444] [<ffffffff8107f5d4>] ? getnstimeofday+0x64/0xf0
[14315.510446] [<ffffffff815de848>] ? netif_receive_skb+0x78/0x80
[14315.510448] [<ffffffff815de960>] ? napi_skb_finish+0x50/0x70
[14315.510450] [<ffffffff815deee5>] ? napi_gro_receive+0xc5/0xd0
[14315.510454] [<ffffffffa000a9d8>] ? bnx2_poll_work+0x608/0x1540 [bnx2]
[14315.510458] [<ffffffff810cfe79>] ? free_one_page+0x1e9/0x460
[14315.510463] [<ffffffff81109ed1>] ? add_partial+0x21/0x80
[14315.510465] [<ffffffff8110b923>] ? __slab_free+0x123/0x130
[14315.510468] [<ffffffff81109ed1>] ? add_partial+0x21/0x80
[14315.510471] [<ffffffffa000b94e>] ? bnx2_poll_msix+0x3e/0xd0 [bnx2]
[14315.510474] [<ffffffff815df0db>] ? net_rx_action+0x11b/0x2e0
[14315.510478] [<ffffffff8105c4b0>] ? __do_softirq+0xf0/0x1f0
[14315.510481] [<ffffffff8100bd9c>] ? call_softirq+0x1c/0x30
[14315.510483] [<ffffffff8100d7a5>] ? do_softirq+0x65/0xa0
[14315.510485] [<ffffffff8105c315>] ? irq_exit+0x75/0xa0
[14315.510487] [<ffffffff8100cd0f>] ? do_IRQ+0x6f/0xf0
[14315.510490] [<ffffffff816b4653>] ? ret_from_intr+0x0/0x11
[14315.510492] <EOI>
[14315.510597] [<ffffffff8103f2ee>] ? mutex_spin_on_owner+0x8e/0xb0
[14315.510601] [<ffffffff816b2915>] ? __mutex_lock_slowpath+0x65/0x180
[14315.510604] [<ffffffff815d668a>] ? __alloc_skb+0x7a/0x170
[14315.510606] [<ffffffff816b27de>] ? mutex_lock+0x1e/0x40
[14315.510609] [<ffffffff815fa2f9>] ? netlink_dump+0x49/0x200
[14315.510612] [<ffffffff815d539d>] ? consume_skb+0x5d/0x80
[14315.510614] [<ffffffff815fa998>] ? netlink_recvmsg+0x268/0x3a0
[14315.510617] [<ffffffff815fc1fb>] ? netlink_sendmsg+0x10b/0x370
[14315.510620] [<ffffffff815ccc77>] ? sock_recvmsg+0xf7/0x130
[14315.510623] [<ffffffff810caa7b>] ? find_get_page+0x1b/0x90
[14315.510625] [<ffffffff810cb461>] ? filemap_fault+0x81/0x400
[14315.510628] [<ffffffff810cad85>] ? unlock_page+0x25/0x30
[14315.510630] [<ffffffff815cc8ae>] ? __sys_recvmsg+0x13e/0x2e0
[14315.510634] [<ffffffff816b7d74>] ? do_page_fault+0x254/0x4a0
[14315.510636] [<ffffffff815fc99f>] ? netlink_bind+0x15f/0x230
[14315.510638] [<ffffffff815cb775>] ? move_addr_to_user+0x65/0x70
[14315.510641] [<ffffffff815cd879>] ? sys_getsockname+0x99/0xa0
[14315.510644] [<ffffffff815cd294>] ? sys_recvmsg+0x44/0x70
[14315.510647] [<ffffffff8100af12>] ? system_call_fastpath+0x16/0x1b
[14315.510648] Code: c2 85 c0 c9 0f 95 c0 0f b6 c0 c3 66 2e 0f 1f 84 00 00 00
00 00 55 b8 00 01 00 00 48 89 e5 f0 66 0f c1 07 38 e0 74 06 f3 90 8a 07 <eb> f6
c9 c3 66 0f 1f 44 00 00 55 48 89 e5 9c 58 0f 1f 44 00 00
[14315.513532] Call Trace:
[14315.513533] <IRQ> [<ffffffffa008370d>] ? __ip_vs_get_out_rt+0x4d/0x380
[ip_vs]
[14315.513540] [<ffffffff815e02e1>] ? dev_queue_xmit+0x171/0x460
[14315.513542] [<ffffffff810647a6>] ? lock_timer_base+0x36/0x70
[14315.513546] [<ffffffffa0084b60>] ? ip_vs_dr_xmit+0x50/0x420 [ip_vs]
[14315.513550] [<ffffffffa007cdfa>] ? ip_vs_in+0x15a/0x5a0 [ip_vs]
[14315.513554] [<ffffffffa007c3e5>] ? ip_vs_out+0x675/0x9b0 [ip_vs]
[14315.513558] [<ffffffffa007d28e>] ? ip_vs_remote_request4+0xe/0x10 [ip_vs]
[14315.513560] [<ffffffff815fe8f7>] ? nf_iterate+0x67/0xa0
[14315.513563] [<ffffffff81608a50>] ? ip_local_deliver_finish+0x0/0x1d0
[14315.513565] [<ffffffff815fe9db>] ? nf_hook_slow+0xab/0x100
[14315.513568] [<ffffffff81608a50>] ? ip_local_deliver_finish+0x0/0x1d0
[14315.513570] [<ffffffff81608460>] ? ip_rcv_finish+0x0/0x310
[14315.513573] [<ffffffff81608c75>] ? ip_local_deliver+0x55/0xa0
[14315.513575] [<ffffffff816085b2>] ? ip_rcv_finish+0x152/0x310
[14315.513578] [<ffffffff816089de>] ? ip_rcv+0x26e/0x2e0
[14315.513580] [<ffffffff815dd833>] ? __netif_receive_skb+0x433/0x520
[14315.513583] [<ffffffff815dd885>] ? __netif_receive_skb+0x485/0x520
[14315.513586] [<ffffffff8107f5d4>] ? getnstimeofday+0x64/0xf0
[14315.513588] [<ffffffff815de848>] ? netif_receive_skb+0x78/0x80
[14315.513590] [<ffffffff815de960>] ? napi_skb_finish+0x50/0x70
[14315.513592] [<ffffffff815deee5>] ? napi_gro_receive+0xc5/0xd0
[14315.513596] [<ffffffffa000a9d8>] ? bnx2_poll_work+0x608/0x1540 [bnx2]
[14315.513598] [<ffffffff810cfe79>] ? free_one_page+0x1e9/0x460
[14315.513601] [<ffffffff81109ed1>] ? add_partial+0x21/0x80
[14315.513604] [<ffffffff8110b923>] ? __slab_free+0x123/0x130
[14315.513606] [<ffffffff81109ed1>] ? add_partial+0x21/0x80
[14315.513609] [<ffffffffa000b94e>] ? bnx2_poll_msix+0x3e/0xd0 [bnx2]
[14315.513612] [<ffffffff815df0db>] ? net_rx_action+0x11b/0x2e0
[14315.513614] [<ffffffff8105c4b0>] ? __do_softirq+0xf0/0x1f0
[14315.513616] [<ffffffff8100bd9c>] ? call_softirq+0x1c/0x30
[14315.513618] [<ffffffff8100d7a5>] ? do_softirq+0x65/0xa0
[14315.513621] [<ffffffff8105c315>] ? irq_exit+0x75/0xa0
[14315.513623] [<ffffffff8100cd0f>] ? do_IRQ+0x6f/0xf0
[14315.513625] [<ffffffff816b4653>] ? ret_from_intr+0x0/0x11
[14315.513626] <EOI> [<ffffffff8103f2ee>] ? mutex_spin_on_owner+0x8e/0xb0
[14315.513631] [<ffffffff816b2915>] ? __mutex_lock_slowpath+0x65/0x180
[14315.513633] [<ffffffff815d668a>] ? __alloc_skb+0x7a/0x170
[14315.513636] [<ffffffff816b27de>] ? mutex_lock+0x1e/0x40
[14315.513638] [<ffffffff815fa2f9>] ? netlink_dump+0x49/0x200
[14315.513641] [<ffffffff815d539d>] ? consume_skb+0x5d/0x80
[14315.513643] [<ffffffff815fa998>] ? netlink_recvmsg+0x268/0x3a0
[14315.513645] [<ffffffff815fc1fb>] ? netlink_sendmsg+0x10b/0x370
[14315.513648] [<ffffffff815ccc77>] ? sock_recvmsg+0xf7/0x130
[14315.513650] [<ffffffff810caa7b>] ? find_get_page+0x1b/0x90
[14315.513653] [<ffffffff810cb461>] ? filemap_fault+0x81/0x400
[14315.513655] [<ffffffff810cad85>] ? unlock_page+0x25/0x30
[14315.513657] [<ffffffff815cc8ae>] ? __sys_recvmsg+0x13e/0x2e0
[14315.513660] [<ffffffff816b7d74>] ? do_page_fault+0x254/0x4a0
[14315.513662] [<ffffffff815fc99f>] ? netlink_bind+0x15f/0x230
[14315.513665] [<ffffffff815cb775>] ? move_addr_to_user+0x65/0x70
[14315.513667] [<ffffffff815cd879>] ? sys_getsockname+0x99/0xa0
[14315.513670] [<ffffffff815cd294>] ? sys_recvmsg+0x44/0x70
[14315.513672] [<ffffffff8100af12>] ? system_call_fastpath+0x16/0x1b
[14315.785537] BUG: soft lockup - CPU#3 stuck for 67s! [ipvsadm:8405]
[14315.785608] Modules linked in: ip_vs_rr ip_vs_lblc ip_vs nf_conntrack
xt_hashlimit ts_kmp xt_string xt_recent ipmi_devintf ipmi_si ipmi_msghandler
bnx2
[14315.785617] CPU 3
[14315.785618] Modules linked in: ip_vs_rr ip_vs_lblc ip_vs nf_conntrack
xt_hashlimit ts_kmp xt_string xt_recent ipmi_devintf ipmi_si ipmi_msghandler
bnx2
[14315.785627]
[14315.785630] Pid: 8405, comm: ipvsadm Not tainted 2.6.37-hw #8
01V648/PowerEdge R410
[14315.785632] RIP: 0010:[<ffffffff816b3f64>] [<ffffffff816b3f64>]
_raw_spin_lock+0x14/0x20
[14315.785638] RSP: 0018:ffff8800cee637d0 EFLAGS: 00000286
[14315.785640] RAX: 00000000000001fd RBX: ffff8800cee637d0 RCX: 0000000000000000
[14315.785643] RDX: 000000001a204740 RSI: ffff88011fe83200 RDI: ffff88011fe832dc
[14315.785646] RBP: ffffffff8100b853 R08: 0000000000000003 R09: ffff8800cee638b0
[14315.785648] R10: ffff880119568a98 R11: 0000000000000400 R12: ffff8800cee63750
[14315.785651] R13: ffff88011fe83200 R14: ffff88012a02c454 R15: ffffffff8102a5a7
[14315.785654] FS: 00007f476e92b6e0(0000) GS:ffff8800cee60000(0000)
knlGS:0000000000000000
[14315.785657] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[14315.785659] CR2: 00000000011a3488 CR3: 0000000110a25000 CR4: 00000000000006e0
[14315.785662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14315.785665] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[14315.785668] Process ipvsadm (pid: 8405, threadinfo ffff880111bb2000, task
ffff880110edc2c0)
[14315.785670] Stack:
[14315.785731] ffff8800cee638b0 ffffffffa008370d ffff88012a02c468
1a20474000000003
[14315.785736] 0000000800000006 0000000000000000 0000000000000001
000000000000024d
[14315.785740] 0000000000000400 0000000000000003 ffff8800cee63940
ffffffff81046349
[14315.785744] Call Trace:
[14315.785806] <IRQ>
[14315.785915] [<ffffffffa008370d>] ? __ip_vs_get_out_rt+0x4d/0x380 [ip_vs]
[14315.785920] [<ffffffff81046349>] ? select_task_rq_fair+0x989/0x9f0
[14315.785924] [<ffffffff810647a6>] ? lock_timer_base+0x36/0x70
[14315.785931] [<ffffffffa0084b60>] ? ip_vs_dr_xmit+0x50/0x420 [ip_vs]
[14315.785936] [<ffffffffa007cdfa>] ? ip_vs_in+0x15a/0x5a0 [ip_vs]
[14315.785941] [<ffffffffa007c3e5>] ? ip_vs_out+0x675/0x9b0 [ip_vs]
[14315.785947] [<ffffffffa007d28e>] ? ip_vs_remote_request4+0xe/0x10 [ip_vs]
[14315.785951] [<ffffffff815fe8f7>] ? nf_iterate+0x67/0xa0
[14315.785955] [<ffffffff81608a50>] ? ip_local_deliver_finish+0x0/0x1d0
[14315.785958] [<ffffffff815fe9db>] ? nf_hook_slow+0xab/0x100
[14315.785962] [<ffffffff81608a50>] ? ip_local_deliver_finish+0x0/0x1d0
[14315.785965] [<ffffffff81608460>] ? ip_rcv_finish+0x0/0x310
[14315.785969] [<ffffffff81608c75>] ? ip_local_deliver+0x55/0xa0
[14315.785973] [<ffffffff816085b2>] ? ip_rcv_finish+0x152/0x310
[14315.785976] [<ffffffff816089de>] ? ip_rcv+0x26e/0x2e0
[14315.785980] [<ffffffff815dd833>] ? __netif_receive_skb+0x433/0x520
[14315.785984] [<ffffffff815dd885>] ? __netif_receive_skb+0x485/0x520
[14315.785989] [<ffffffff8107f5d4>] ? getnstimeofday+0x64/0xf0
[14315.785992] [<ffffffff815de848>] ? netif_receive_skb+0x78/0x80
[14315.785995] [<ffffffff815de960>] ? napi_skb_finish+0x50/0x70
[14315.785998] [<ffffffff815deee5>] ? napi_gro_receive+0xc5/0xd0
[14315.786004] [<ffffffffa000a9d8>] ? bnx2_poll_work+0x608/0x1540 [bnx2]
[14315.786008] [<ffffffff81045cd7>] ? select_task_rq_fair+0x317/0x9f0
[14315.786012] [<ffffffff810b23bc>] ? trace_ctxwake_hex+0x7c/0x140
[14315.786016] [<ffffffff810c79ff>] ? perf_event_task_tick+0x9f/0x2d0
[14315.786020] [<ffffffffa000b94e>] ? bnx2_poll_msix+0x3e/0xd0 [bnx2]
[14315.786024] [<ffffffff815df0db>] ? net_rx_action+0x11b/0x2e0
[14315.786028] [<ffffffff8105c4b0>] ? __do_softirq+0xf0/0x1f0
[14315.786031] [<ffffffff8100bd9c>] ? call_softirq+0x1c/0x30
[14315.786034] [<ffffffff8100d7a5>] ? do_softirq+0x65/0xa0
[14315.786038] [<ffffffff8105c315>] ? irq_exit+0x75/0xa0
[14315.786041] [<ffffffff8100cd0f>] ? do_IRQ+0x6f/0xf0
[14315.786044] [<ffffffff816b4653>] ? ret_from_intr+0x0/0x11
[14315.786046] <EOI>
[14315.786153] [<ffffffff8103f2df>] ? mutex_spin_on_owner+0x7f/0xb0
[14315.786157] [<ffffffff816b2915>] ? __mutex_lock_slowpath+0x65/0x180
[14315.786161] [<ffffffff816b27de>] ? mutex_lock+0x1e/0x40
[14315.786164] [<ffffffff815fdd38>] ? genl_rcv+0x18/0x40
[14315.786167] [<ffffffff815fba53>] ? netlink_unicast+0x2d3/0x2e0
[14315.786171] [<ffffffff815d7ba3>] ? memcpy_fromiovec+0x63/0x80
[14315.786174] [<ffffffff815fc426>] ? netlink_sendmsg+0x336/0x370
[14315.786179] [<ffffffff815ccdf3>] ? sock_sendmsg+0xe3/0x110
[14315.786182] [<ffffffff810cb461>] ? filemap_fault+0x81/0x400
[14315.786187] [<ffffffff810e9149>] ? __do_fault+0x409/0x520
[14315.786191] [<ffffffff810d5748>] ? lru_cache_add_lru+0x28/0x50
[14315.786194] [<ffffffff815cb6fc>] ? move_addr_to_kernel+0x2c/0x40
[14315.786198] [<ffffffff815d7d03>] ? verify_iovec+0x43/0xf0
[14315.786201] [<ffffffff815cdbb4>] ? sys_sendmsg+0x184/0x320
[14315.786205] [<ffffffff816b7d74>] ? do_page_fault+0x254/0x4a0
[14315.786209] [<ffffffff815fc99f>] ? netlink_bind+0x15f/0x230
[14315.786212] [<ffffffff815cb775>] ? move_addr_to_user+0x65/0x70
[14315.786216] [<ffffffff815cd879>] ? sys_getsockname+0x99/0xa0
[14315.786220] [<ffffffff8100af12>] ? system_call_fastpath+0x16/0x1b
[14315.786222] Code: 0f b6 c2 85 c0 c9 0f 95 c0 0f b6 c0 c3 66 2e 0f 1f 84 00
00 00 00 00 55 b8 00 01 00 00 48 89 e5 f0 66 0f c1 07 38 e0 74 06 f3 90 <8a> 07
eb f6 c9 c3 66 0f 1f 44 00 00 55 48 89 e5 9c 58 0f 1f 44
[14315.789140] Call Trace:
[14315.789142] <IRQ> [<ffffffffa008370d>] ? __ip_vs_get_out_rt+0x4d/0x380
[ip_vs]
[14315.789150] [<ffffffff81046349>] ? select_task_rq_fair+0x989/0x9f0
[14315.789154] [<ffffffff810647a6>] ? lock_timer_base+0x36/0x70
[14315.789160] [<ffffffffa0084b60>] ? ip_vs_dr_xmit+0x50/0x420 [ip_vs]
[14315.789165] [<ffffffffa007cdfa>] ? ip_vs_in+0x15a/0x5a0 [ip_vs]
[14315.789170] [<ffffffffa007c3e5>] ? ip_vs_out+0x675/0x9b0 [ip_vs]
[14315.789176] [<ffffffffa007d28e>] ? ip_vs_remote_request4+0xe/0x10 [ip_vs]
[14315.789179] [<ffffffff815fe8f7>] ? nf_iterate+0x67/0xa0
[14315.789183] [<ffffffff81608a50>] ? ip_local_deliver_finish+0x0/0x1d0
[14315.789186] [<ffffffff815fe9db>] ? nf_hook_slow+0xab/0x100
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
