[Bug 1109880] New: iscsi crashes by high load
http://bugzilla.suse.com/show_bug.cgi?id=1109880 Bug ID: 1109880 Summary: iscsi crashes by high load Classification: openSUSE Product: openSUSE Distribution Version: Leap 42.3 Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-maintainers@forge.provo.novell.com Reporter: varkoly@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- On openLeap 42.3 if I mount a iscsi device and start to copy big amount of data after 2 minutes the iscsi connection crashes with kernel trace. Sep 23 21:41:10 admin kernel: ------------[ cut here ]------------ Sep 23 21:41:10 admin kernel: WARNING: CPU: 0 PID: 767 at ../kernel/softirq.c:161 __local_bh_enable_ip+0x5f/0x80() Sep 23 21:41:10 admin kernel: Modules linked in: fuse nf_log_ipv6 ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_nat xt_comment xt_TCPMSS nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_nat nf_nat_ipv4 nf_nat iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi af_packet vmw_vsock_vmci_transport vsock iscsi_ibft iscsi_boot_sysfs ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT nf_reject_ipv4 xt_pkttype xt_tcpudp iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables xfs libcrc32c sb_edac edac_core crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drbg ansi_cprng aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd pcspkr joydev shpchp Sep 23 21:41:10 admin kernel: i2c_piix4 vmw_vmci fjes processor ac button ext4 crc16 jbd2 mbcache ata_generic hid_generic usbhid sd_mod sr_mod cdrom ata_piix vmwgfx(O) serio_raw drm_kms_helper(O) syscopyarea sysfillrect sysimgblt fb_sys_fops uhci_hcd ehci_pci ahci ttm(O) ehci_hcd libahci mptspi scsi_transport_spi usbcore mptscsih vmxnet3 libata drm(O) usb_common mptbase dm_mod sg scsi_mod autofs4 Sep 23 21:41:10 admin kernel: CPU: 0 PID: 767 Comm: kworker/0:1H Tainted: G O 4.4.155-68-default #1 Sep 23 21:41:10 admin kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016 Sep 23 21:41:10 admin kernel: Workqueue: kblockd blk_timeout_work Sep 23 21:41:10 admin kernel: 0000000000000000 ffffffff81346687 0000000000000000 ffffffff81a23362 Sep 23 21:41:10 admin kernel: ffffffff810842b1 0000000000000201 ffff8818c14053f0 ffff882fdc399428 Sep 23 21:41:10 admin kernel: ffff8818c1419000 0000000000000000 ffffffff810893ef ffff8818c1405338 Sep 23 21:41:10 admin kernel: Call Trace: Sep 23 21:41:10 admin kernel: [<ffffffff8101a049>] dump_trace+0x59/0x350 Sep 23 21:41:10 admin kernel: [<ffffffff8101a43a>] show_stack_log_lvl+0xfa/0x180 Sep 23 21:41:10 admin kernel: [<ffffffff8101b201>] show_stack+0x21/0x40 Sep 23 21:41:10 admin kernel: [<ffffffff81346687>] dump_stack+0x5c/0x85 Sep 23 21:41:10 admin kernel: [<ffffffff810842b1>] warn_slowpath_common+0x81/0xb0 Sep 23 21:41:10 admin kernel: [<ffffffff810893ef>] __local_bh_enable_ip+0x5f/0x80 Sep 23 21:41:10 admin kernel: [<ffffffffa06a21f3>] __iscsi_conn_send_pdu+0x1b3/0x390 [libiscsi] Sep 23 21:41:10 admin kernel: [<ffffffffa06a24e9>] iscsi_send_nopout+0xb9/0x100 [libiscsi] Sep 23 21:41:10 admin kernel: [<ffffffffa06a36f1>] iscsi_eh_cmd_timed_out+0x2b1/0x310 [libiscsi] Sep 23 21:41:10 admin kernel: [<ffffffffa001230f>] scsi_times_out+0x5f/0x260 [scsi_mod] Sep 23 21:41:10 admin kernel: [<ffffffff8131ceee>] blk_rq_timed_out+0x1e/0x60 Sep 23 21:41:10 admin kernel: [<ffffffff8131cfaa>] blk_timeout_work+0x7a/0x120 Sep 23 21:41:10 admin kernel: [<ffffffff8109dd0b>] process_one_work+0x15b/0x450 Sep 23 21:41:10 admin kernel: [<ffffffff8109e8d6>] worker_thread+0x116/0x4c0 Sep 23 21:41:10 admin kernel: [<ffffffff810a3f34>] kthread+0xd4/0xf0 Sep 23 21:41:10 admin kernel: [<ffffffff81645a55>] ret_from_fork+0x55/0x80 Sep 23 21:41:10 admin kernel: DWARF2 unwinder stuck at ret_from_fork+0x55/0x80 Sep 23 21:41:10 admin kernel: Sep 23 21:41:10 admin kernel: Leftover inexact backtrace: Sep 23 21:41:10 admin kernel: [<ffffffff810a3e60>] ? kthread_park+0x50/0x50 Sep 23 21:41:10 admin kernel: ---[ end trace 3f368ae90b5e003c ]--- -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1109880
Hannes Reinecke
http://bugzilla.suse.com/show_bug.cgi?id=1109880
http://bugzilla.suse.com/show_bug.cgi?id=1109880#c1
Lee Duncan
From just the segment of log output you shared, it looks like you have a heavy IO load, and you have open-iscsi NOPs enabled, which is a bad idea to start with. This is because, under heavy IO load, the NOPs can get congested along with the IO, but the NOP timeout is unforgiving of congestion and can signal a
http://bugzilla.suse.com/show_bug.cgi?id=1109880
http://bugzilla.suse.com/show_bug.cgi?id=1109880#c2
Lee Duncan
http://bugzilla.suse.com/show_bug.cgi?id=1109880
http://bugzilla.suse.com/show_bug.cgi?id=1109880#c3
--- Comment #3 from Lee Duncan
http://bugzilla.suse.com/show_bug.cgi?id=1109880
http://bugzilla.suse.com/show_bug.cgi?id=1109880#c4
Lee Duncan
participants (1)
-
bugzilla_noreply@novell.com