Environment:
3 virtual machines, connected with FCoE,
2 of them are workers and 1 is a RAID-Server.
Issue:
This issue occurs always onto the RAID-Server
after a while, depend to the load situation.
>From the worksystems (SLES11SP4) are 4 path's
configured and used to the VM which is the RAID
system.
A reconnect with:
fcoeadm -r eth1/2
doesn't work and hangs.
RAID:~ # hostnamectl
Static hostname: RAID
Icon name: computer-vm
Chassis: vm
Machine ID: 8861646bb10c9582ad7be1f6583d364a
Boot ID: e31e33c369584f389cddd61d83b96e2d
Virtualization: kvm
Operating System: openSUSE Leap 42.2
CPE OS Name: cpe:/o:opensuse:leap:42.2
Kernel: Linux 4.4.70-18.9-default
Architecture: x86-64
RAID:~ # uname -a
Linux RAID 4.4.70-18.9-default #1 SMP Wed May 31 09:09:25 UTC 2017 (c1231a7)
x86_64 x86_64 x86_64 GNU/Linux
RAID:~ #
----------------------------
Jun 13 22:25:59 RAID kernel: BUG: sleeping function called from invalid context
at ../kernel/locking/mutex.c:97
Jun 13 22:25:59 RAID kernel: in_atomic(): 1, irqs_disabled(): 0, pid: 11442,
name: kworker/3:1
Jun 13 22:25:59 RAID kernel: CPU: 3 PID: 11442 Comm: kworker/3:1 Not tainted
4.4.70-18.9-default #1
Jun 13 22:25:59 RAID kernel: Hardware name: QEMU Standard PC (Q35 + ICH9,
2009), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
Jun 13 22:25:59 RAID kernel: Workqueue: events fcoe_ctlr_timer_work [libfcoe]
Jun 13 22:25:59 RAID kernel: 0000000000000000 ffffffff81329447
ffff88007b5fa048 ffff88007b5fa048
Jun 13 22:25:59 RAID kernel: ffffffff8160cf6c ffff88007b5fa000
ffffffffa00945f1 ffff88007b5fa010
Jun 13 22:25:59 RAID kernel: 0000000101d6b8c0 ffff880078d74820
ffff880078d747d8 ffffffffa00b0fe9
Jun 13 22:25:59 RAID kernel: Call Trace:
Jun 13 22:25:59 RAID kernel: [<ffffffff81019ea9>] dump_trace+0x59/0x320
Jun 13 22:25:59 RAID kernel: [<ffffffff8101a26a>]
show_stack_log_lvl+0xfa/0x180
Jun 13 22:25:59 RAID kernel: [<ffffffff8101b011>] show_stack+0x21/0x40
Jun 13 22:25:59 RAID kernel: [<ffffffff81329447>] dump_stack+0x5c/0x85
Jun 13 22:25:59 RAID kernel: [<ffffffff8160cf6c>] mutex_lock+0x1c/0x38
Jun 13 22:25:59 RAID kernel: [<ffffffffa00945f1>] fc_rport_logoff+0x21/0xe0
[libfc]
Jun 13 22:25:59 RAID kernel: [<ffffffffa00b0fe9>]
fcoe_ctlr_timer_work+0x6a9/0xca0 [libfcoe]
Jun 13 22:25:59 RAID kernel: [<ffffffff81097775>] process_one_work+0x155/0x440
Jun 13 22:25:59 RAID kernel: [<ffffffff810982b6>] worker_thread+0x116/0x4b0
Jun 13 22:25:59 RAID kernel: [<ffffffff8109d8a2>] kthread+0xd2/0xf0
Jun 13 22:25:59 RAID kernel: [<ffffffff8160f58f>] ret_from_fork+0x3f/0x70
Jun 13 22:25:59 RAID kernel: DWARF2 unwinder stuck at ret_from_fork+0x3f/0x70
Jun 13 22:25:59 RAID kernel:
Jun 13 22:25:59 RAID kernel: Leftover inexact backtrace:
Jun 13 22:25:59 RAID kernel: [<ffffffff8109d7d0>] ? kthread_park+0x50/0x50
Jun 13 22:26:03 RAID kernel: BUG: sleeping function called from invalid context
at ../kernel/locking/mutex.c:97
Jun 13 22:26:03 RAID kernel: in_atomic(): 1, irqs_disabled(): 0, pid: 11442,
name: kworker/3:1
Jun 13 22:26:03 RAID kernel: CPU: 3 PID: 11442 Comm: kworker/3:1 Not tainted
4.4.70-18.9-default #1
Jun 13 22:26:03 RAID kernel: Hardware name: QEMU Standard PC (Q35 + ICH9,
2009), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
Jun 13 22:26:03 RAID kernel: Workqueue: events fcoe_ctlr_timer_work [libfcoe]
Jun 13 22:26:03 RAID kernel: 0000000000000000 ffffffff81329447
ffff880036a96048 ffff880036a96048
Jun 13 22:26:03 RAID kernel: ffffffff8160cf6c ffff880036a96000
ffffffffa00945f1 ffff880036a96010
Jun 13 22:26:03 RAID kernel: 0000000101d706e2 ffff880078d74820
ffff880078d747d8 ffffffffa00b0fe9
Jun 13 22:26:03 RAID kernel: Call Trace:
Jun 13 22:26:03 RAID kernel: [<ffffffff81019ea9>] dump_trace+0x59/0x320
Jun 13 22:26:03 RAID kernel: [<ffffffff8101a26a>]
show_stack_log_lvl+0xfa/0x180
Jun 13 22:26:03 RAID kernel: [<ffffffff8101b011>] show_stack+0x21/0x40
Jun 13 22:26:03 RAID kernel: [<ffffffff81329447>] dump_stack+0x5c/0x85
Jun 13 22:26:03 RAID kernel: [<ffffffff8160cf6c>] mutex_lock+0x1c/0x38
Jun 13 22:26:03 RAID kernel: [<ffffffffa00945f1>] fc_rport_logoff+0x21/0xe0
[libfc]
Jun 13 22:26:03 RAID kernel: [<ffffffffa00b0fe9>]
fcoe_ctlr_timer_work+0x6a9/0xca0 [libfcoe]
Jun 13 22:26:03 RAID kernel: [<ffffffff81097775>] process_one_work+0x155/0x440
Jun 13 22:26:03 RAID kernel: [<ffffffff810982b6>] worker_thread+0x116/0x4b0
Jun 13 22:26:03 RAID kernel: [<ffffffff8109d8a2>] kthread+0xd2/0xf0
Jun 13 22:26:03 RAID kernel: [<ffffffff8160f58f>] ret_from_fork+0x3f/0x70
Jun 13 22:26:03 RAID kernel: DWARF2 unwinder stuck at ret_from_fork+0x3f/0x70
Jun 13 22:26:03 RAID kernel:
Jun 13 22:26:03 RAID kernel: Leftover inexact backtrace:
Jun 13 22:26:03 RAID kernel: [<ffffffff8109d7d0>] ? kthread_park+0x50/0x50
Jun 13 22:26:03 RAID kernel: BUG: scheduling while atomic:
kworker/3:1/11442/0x00000002
Jun 13 22:26:03 RAID kernel: Modules linked in: fuse target_core_user tcm_loop
tcm_qla2xxx qla2xxx vhost_scsi vhost iscsi_target_mod ib_srpt ib_cm ib_sa
ib_mad ib_core ib_addr tcm_fc target_core_file target_core_iblock
target_core_pscsi target_core_mod configfs iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi af_packet iscsi_ibft iscsi_boot_sysfs iTCO_wdt
iTCO_vendor_support virtio_balloon ppdev acpi_cpufreq pcspkr i2c_i801 joydev
lpc_ich mfd_core processor parport_pc parport shpchp button ext4 crc16 jbd2
mbcache sr_mod cdrom sd_mod virtio_gpu drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm virtio_console virtio_scsi hid_generic usbhid
8021q garp stp llc mrp bnx2fc cnic uio xhci_pci xhci_hcd usbcore usb_common
ahci libahci virtio_pci virtio_ring virtio e1000 libata serio_raw fcoe libfcoe
Jun 13 22:26:03 RAID kernel: libfc scsi_transport_fc sg scsi_mod autofs4
Jun 13 22:26:03 RAID kernel: CPU: 3 PID: 11442 Comm: kworker/3:1 Not tainted
4.4.70-18.9-default #1
Jun 13 22:26:03 RAID kernel: Hardware name: QEMU Standard PC (Q35 + ICH9,
2009), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
Jun 13 22:26:03 RAID kernel: Workqueue: events fcoe_ctlr_timer_work [libfcoe]
Jun 13 22:26:03 RAID kernel: 0000000000000000 ffffffff81329447
ffff88017fd95c40 ffff8801022cfca0
Jun 13 22:26:03 RAID kernel: ffffffff8118a8ee ffff8801022cfce8
ffffffff8160afbe ffff8801470460c4
Jun 13 22:26:03 RAID kernel: ffff8801444c0d40 ffff8801022d0000
ffff8801444c0d40 ffff880178f9984c
Jun 13 22:26:03 RAID kernel: Call Trace:
Jun 13 22:26:03 RAID kernel: [<ffffffff81019ea9>] dump_trace+0x59/0x320
Jun 13 22:26:03 RAID kernel: [<ffffffff8101a26a>]
show_stack_log_lvl+0xfa/0x180
Jun 13 22:26:03 RAID kernel: [<ffffffff8101b011>] show_stack+0x21/0x40
Jun 13 22:26:03 RAID kernel: [<ffffffff81329447>] dump_stack+0x5c/0x85
Jun 13 22:26:03 RAID kernel: [<ffffffff8118a8ee>] __schedule_bug+0x4b/0x59
Jun 13 22:26:03 RAID kernel: [<ffffffff8160afbe>] thread_return+0x5eb/0x6bd
Jun 13 22:26:03 RAID kernel: [<ffffffff8160b0cc>] schedule+0x3c/0x90
Jun 13 22:26:03 RAID kernel: [<ffffffff8160b3f5>]
schedule_preempt_disabled+0x15/0x20
Jun 13 22:26:03 RAID kernel: [<ffffffff8160ced6>]
__mutex_lock_slowpath+0xb6/0x130
Jun 13 22:26:03 RAID kernel: [<ffffffff8160cf79>] mutex_lock+0x29/0x38
Jun 13 22:26:03 RAID kernel: [<ffffffffa00945f1>] fc_rport_logoff+0x21/0xe0
[libfc]
Jun 13 22:26:03 RAID kernel: [<ffffffffa00b0fe9>]
fcoe_ctlr_timer_work+0x6a9/0xca0 [libfcoe]
Jun 13 22:26:03 RAID kernel: [<ffffffff81097775>] process_one_work+0x155/0x440
Jun 13 22:26:03 RAID kernel: [<ffffffff810982b6>] worker_thread+0x116/0x4b0
Jun 13 22:26:03 RAID kernel: [<ffffffff8109d8a2>] kthread+0xd2/0xf0
Jun 13 22:26:03 RAID kernel: [<ffffffff8160f58f>] ret_from_fork+0x3f/0x70
Jun 13 22:26:03 RAID kernel: DWARF2 unwinder stuck at ret_from_fork+0x3f/0x70
Jun 13 22:26:03 RAID kernel:
Jun 13 22:26:03 RAID kernel: Leftover inexact backtrace:
Jun 13 22:26:03 RAID kernel: [<ffffffff8109d7d0>] ? kthread_park+0x50/0x50
Jun 13 22:26:19 RAID kernel: BUG: sleeping function called from invalid context
at ../kernel/locking/mutex.c:97
Jun 13 22:26:19 RAID kernel: in_atomic(): 1, irqs_disabled(): 0, pid: 11499,
name: kworker/3:3
Jun 13 22:26:19 RAID kernel: CPU: 3 PID: 11499 Comm: kworker/3:3 Tainted: G
W 4.4.70-18.9-default #1
Jun 13 22:26:19 RAID kernel: Hardware name: QEMU Standard PC (Q35 + ICH9,
2009), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
Jun 13 22:26:19 RAID kernel: Workqueue: events fcoe_ctlr_timer_work [libfcoe]
Jun 13 22:26:19 RAID kernel: 0000000000000000 ffffffff81329447
ffff880178801848 ffff880178801848
Jun 13 22:26:19 RAID kernel: ffffffff8160cf6c
Jun 13 22:26:19 RAID kernel: ffff880178801800 ffffffffa00945f1
ffff880178801810
Jun 13 22:26:19 RAID kernel: 0000000101d6d440
Jun 13 22:26:19 RAID kernel: ffff880036a17820 ffff880036a177d8
ffffffffa00b0fe9
Jun 13 22:26:19 RAID kernel: Call Trace:
Jun 13 22:26:19 RAID kernel: [<ffffffff81019ea9>] dump_trace+0x59/0x320
Jun 13 22:26:19 RAID kernel: [<ffffffff8101a26a>]
show_stack_log_lvl+0xfa/0x180
Jun 13 22:26:19 RAID kernel: [<ffffffff8101b011>] show_stack+0x21/0x40
Jun 13 22:26:19 RAID kernel: [<ffffffff81329447>] dump_stack+0x5c/0x85
Jun 13 22:26:19 RAID kernel: [<ffffffff8160cf6c>] mutex_lock+0x1c/0x38
Jun 13 22:26:19 RAID kernel: [<ffffffffa00945f1>] fc_rport_logoff+0x21/0xe0
[libfc]
Jun 13 22:26:19 RAID kernel: [<ffffffffa00b0fe9>]
fcoe_ctlr_timer_work+0x6a9/0xca0 [libfcoe]
Jun 13 22:26:19 RAID kernel: [<ffffffff81097775>] process_one_work+0x155/0x440
Jun 13 22:26:19 RAID kernel: [<ffffffff810982b6>] worker_thread+0x116/0x4b0
Jun 13 22:26:19 RAID kernel: [<ffffffff8109d8a2>] kthread+0xd2/0xf0
Jun 13 22:26:19 RAID kernel: [<ffffffff8160f58f>] ret_from_fork+0x3f/0x70
Jun 13 22:26:19 RAID kernel: DWARF2 unwinder stuck at ret_from_fork+0x3f/0x70
Jun 13 22:26:19 RAID kernel:
Jun 13 22:26:19 RAID kernel: Leftover inexact backtrace:
Jun 13 22:26:19 RAID kernel: [<ffffffff8109d7d0>] ? kthread_park+0x50/0x50
Jun 13 22:26:51 RAID sshd[11500]: Accepted publickey for root from
192.168.101.100 port 33413 ssh2: DSA
SHA256:p/NSgZGhYmtt2zdIWjlOuyPFGKDIf6OaB2FTiW8CV2c
Jun 13 22:26:51 RAID sshd[11500]: pam_unix(sshd:session): session opened for
user root by (uid=0)
Jun 13 22:26:51 RAID systemd[1]: Created slice User Slice of root.
Jun 13 22:26:51 RAID systemd[1]: Starting User Manager for UID 0...