LVS
lvs-devel
Google
 
Web LinuxVirtualServer.org

Re: Module crash due to broken count

To: Net Filter <netfilternetfilter@xxxxxxxxx>
Subject: Re: Module crash due to broken count
Cc: lvs-devel@xxxxxxxxxxxxxxx, netfilter-devel@xxxxxxxxxxxxxxx
From: Julian Anastasov <ja@xxxxxx>
Date: Wed, 25 Apr 2018 22:27:55 +0300 (EEST)
        Hello,

On Mon, 23 Apr 2018, Net Filter wrote:

> I was requested by Pablo Neira Ayuso to report the netfilter bugzilla
> ticket https://bugzilla.netfilter.org/show_bug.cgi?id=1247 here.
> 
> The ticket is replicated below for your convenience.
> 
> Start of ticket content:
> 
> The configuration I use is a NixOS keepalived setup on AWS (which uses
> Xen) configured with per packet round robin routing in a direct
> routing setup, which in turn uses ipvs.
> 
> Symptoms are ksoftirqd using 100% of the CPU resulting in what looks
> like a DoS attack on the machine.
> 
> The module repeatedly crashes when in this mode with a message similar
> to the one below.
> 
> [root@keepalive:~]# uname -a
> Linux keepalive 4.15.17 #1-NixOS SMP Thu Apr 12 10:31:21 UTC 2018
> x86_64 GNU/Linux
> 
> It is the same problem as described on
> https://github.com/NixOS/nixpkgs/issues/39078.
> 
> [  782.525457] ------------[ cut here ]------------
> [  782.525467] refcount_t hit zero at ip_vs_conn_put+0x31/0x40 [ip_vs]

        Looks like the One-packet scheduling mode (-o, --ops)
runs with refcnt=0 which upsets the refcount_dec() checks.
The solution would be to replace the __ip_vs_conn_put() call
in __ip_vs_conn_put_notimer() with refcount_dec_if_one() in
ip_vs_conn_unlink(). This will make the refcount subsystem
happy. Will test a patch in the following days...

> in sh[15519], uid/euid: 497/497
> [  782.525477] WARNING: CPU: 0 PID: 15519 at ../kernel/panic.c:657
> refcount_error_report+0x94/0x9e
> [  782.525477] Modules linked in: ip_vs_rr cirrus ttm sb_edac
> edac_core drm_kms_helper crct10dif_pclmul crc32_pclmul
> ghash_clmulni_intel pcbc mousedev drm aesni_intel aes_x86_64
> crypto_simd glue_helper cryptd psmouse evdev input_leds led_class
> intel_agp fb_sys_fops syscopyarea sysfillrect intel_rapl_perf mac_hid
> intel_gtt serio_raw sysimgblt agpgart i2c_piix4 i2c_core ata_generic
> pata_acpi floppy cfg80211 rfkill button loop macvlan ip_vs
> nf_conntrack libcrc32c crc32c_generic ip_tables x_tables ipv6
> crc_ccitt autofs4 ext4 crc16 mbcache jbd2 fscrypto ata_piix libata
> atkbd libps2 scsi_mod crc32c_intel i8042 rtc_cmos serio af_packet
> dm_mod dax fuse xen_netfront xen_blkfront
> [  782.525532] CPU: 0 PID: 15519 Comm: sh Tainted: G        W
> 4.15.17 #1-NixOS
> [  782.525533] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
> [  782.525535] RIP: 0010:refcount_error_report+0x94/0x9e
> [  782.525536] RSP: 0000:ffffa344dde039c8 EFLAGS: 00010296
> [  782.525537] RAX: 0000000000000057 RBX: ffffffff92f20e06 RCX: 
> 0000000000000006
> [  782.525541] RDX: 0000000000000007 RSI: 0000000000000086 RDI: 
> ffffa344dde165c0
> [  782.525541] RBP: ffffa344dde03b08 R08: 0000000000000218 R09: 
> 0000000000000004
> [  782.525542] R10: ffffffff93006a80 R11: 0000000000000001 R12: 
> ffffa344d68cd100
> [  782.525543] R13: 00000000000001f1 R14: ffffffff92f12fb0 R15: 
> 0000000000000004
> [  782.525544] FS:  00007fc9d2040fc0(0000) GS:ffffa344dde00000(0000)
> knlGS:0000000000000000
> [  782.525545] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  782.525546] CR2: 000000000262a000 CR3: 0000000016a0c004 CR4: 
> 00000000001606f0
> [  782.525548] Call Trace:
> [  782.525551]  <IRQ>
> [  782.525557]  ex_handler_refcount+0x4e/0x80
> [  782.525558]  fixup_exception+0x33/0x40
> [  782.525563]  do_trap+0x83/0x140
> [  782.525565]  do_error_trap+0x83/0xf0
> [  782.525568]  ? ip_vs_conn_drop_conntrack+0x120/0x1a5 [ip_vs]
> [  782.525574]  ? ip_finish_output2+0x29c/0x390
> [  782.525575]  ? ip_finish_output2+0x1a2/0x390
> [  782.525580]  invalid_op+0x1b/0x40
> [  782.525582] RIP: 0010:ip_vs_conn_put+0x31/0x40 [ip_vs]
> [  782.525583] RSP: 0000:ffffa344dde03bb8 EFLAGS: 00010246
> [  782.525584] RAX: 0000000000000001 RBX: ffffa344df31cf00 RCX: 
> ffffa344d7450198
> [  782.525585] RDX: 0000000000000003 RSI: 00000000fffffe01 RDI: 
> ffffa344d7450140
> [  782.525586] RBP: 0000000000000002 R08: 0000000000000476 R09: 
> 0000000000000000
> [  782.525587] R10: ffffa344dde03b28 R11: ffffa344df200000 R12: 
> ffffa344d7d09000
> [  782.525587] R13: ffffa344def3a980 R14: ffffffffc04f6e20 R15: 
> 0000000000000008
> [  782.525591]  ip_vs_in.part.29.constprop.36+0x34f/0x640 [ip_vs]
> [  782.525593]  ? ip_vs_conn_out_get+0xe0/0xe0 [ip_vs]
> [  782.525596]  ip_vs_remote_request4+0x47/0xa0 [ip_vs]
> [  782.525598]  ? ip_vs_in.part.29.constprop.36+0x640/0x640 [ip_vs]
> [  782.525600]  nf_hook_slow+0x43/0xc0
> [  782.525602]  ip_local_deliver+0xac/0xc0
> [  782.525604]  ? ip_rcv_finish+0x400/0x400
> [  782.525606]  ip_rcv+0x26c/0x380
> [  782.525610]  __netif_receive_skb_core+0x3a0/0xb10
> [  782.525629]  ? inet_gro_receive+0x23c/0x2b0
> [  782.525631]  ? netif_receive_skb_internal+0x24/0xb0
> [  782.525633]  netif_receive_skb_internal+0x24/0xb0
> [  782.525635]  napi_gro_receive+0xb8/0xe0
> [  782.525638]  xennet_poll+0x676/0xb40 [xen_netfront]
> [  782.525641]  net_rx_action+0x139/0x3a0
> [  782.525644]  __do_softirq+0xde/0x2b4
> [  782.525646]  irq_exit+0xae/0xb0
> [  782.525651]  xen_evtchn_do_upcall+0x2c/0x40
> [  782.525653]  xen_hvm_callback_vector+0x7d/0x90
> [  782.525654]  </IRQ>
> [  782.525656] RIP: 0033:0x7fc9d11c91f9
> [  782.525656] RSP: 002b:00007ffebe8a2ea0 EFLAGS: 00000202 ORIG_RAX:
> ffffffffffffff0c
> [  782.525658] RAX: 00000000ffffffff RBX: 0000000002609808 RCX: 
> 0000000000000054
> [  782.525658] RDX: 0000000000000001 RSI: 0000000002605440 RDI: 
> 00000000025f940e
> [  782.525659] RBP: 00000000025f940e R08: 000000000260213d R09: 
> 1999999999999999
> [  782.525660] R10: 000000000262a808 R11: 00000000025f942d R12: 
> 00000000025f940e
> [  782.525661] R13: 00007fc9d1301e20 R14: 00000000025f9408 R15: 
> 00007fc9d1302720
> [  782.525662] Code: 48 8b 95 80 00 00 00 41 55 49 8d 8c 24 e0 05 00
> 00 45 8b 84 24 38 04 00 00 41 89 c1 48 89 de 48 c7 c7 a8 2f f2 92 e8
> 7c fa ff ff <0f> 0b 58 5b 5d 41 5c 41 5d c3 0f 1f 44 00 00 55 48 89 e5
> 41 56
> [  782.525684] ---[ end trace 429a7a27ee858cfb ]---
> --

Regards

--
Julian Anastasov <ja@xxxxxx>
--
To unsubscribe from this list: send the line "unsubscribe lvs-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

<Prev in Thread] Current Thread [Next in Thread>