Comment # 1
on bug 1106005
from Markus Zimmermann
It happend again, and since this takes a whole server offline that is
important, I mark this as critical.
[604205.233980] ------------[ cut here ]------------
[604205.233988] WARNING: CPU: 4 PID: 617 at ../drivers/nvme/host/core.c:571
nvme_setup_cmd+0x3dd/0x430 [nvme_core]
[604205.233994] WARNING: CPU: 0 PID: 568 at ../drivers/nvme/host/core.c:571
nvme_setup_cmd+0x3dd/0x430 [nvme_core]
[604205.233995] Modules linked in:
[604205.233996] Modules linked in:
[604205.233997] tcp_diag
[604205.233997] tcp_diag udp_diag
[604205.233999] udp_diag
[604205.234000] inet_diag fuse
[604205.234001] inet_diag fuse
[604205.234002] vhost_net
[604205.234003] vhost_net vhost
[604205.234004] vhost tap
[604205.234005] tap xt_CHECKSUM
[604205.234006] xt_CHECKSUM
[604205.234007] iptable_mangle
[604205.234008] iptable_mangle ipt_REJECT
[604205.234009] ipt_REJECT tun
[604205.234010] tun devlink
[604205.234011] devlink ebtable_filter
[604205.234013] ebtable_filter ebtables
[604205.234014] ebtables
[604205.234015] af_packet
[604205.234015] af_packet iscsi_ibft
[604205.234016] iscsi_ibft iscsi_boot_sysfs
[604205.234018] iscsi_boot_sysfs nf_conntrack_ipv6
[604205.234019] nf_conntrack_ipv6 nf_defrag_ipv6
[604205.234020] nf_defrag_ipv6 ip6table_filter
[604205.234021] ip6table_filter ip6_tables
[604205.234022] ip6_tables ipt_MASQUERADE
[604205.234024] ipt_MASQUERADE nf_nat_masquerade_ipv4
[604205.234025] nf_nat_masquerade_ipv4 xt_nat
[604205.234026] xt_nat iptable_nat
[604205.234027] iptable_nat nf_nat_ipv4
[604205.234028] nf_nat_ipv4 nf_nat
[604205.234029] nf_nat xt_physdev
[604205.234030] xt_physdev br_netfilter
[604205.234032] br_netfilter bridge
[604205.234033] bridge stp
[604205.234034] stp llc
[604205.234035] llc xt_tcpudp
[604205.234036] xt_tcpudp nf_conntrack_ipv4
[604205.234037] nf_conntrack_ipv4 nf_defrag_ipv4
[604205.234039] nf_defrag_ipv4
[604205.234039] xt_conntrack
[604205.234040] xt_conntrack nf_conntrack
[604205.234041] nf_conntrack libcrc32c
[604205.234042] libcrc32c iptable_filter
[604205.234043] iptable_filter ip_tables
[604205.234045] ip_tables x_tables
[604205.234046] x_tables intel_rapl
[604205.234047] intel_rapl x86_pkg_temp_thermal
[604205.234048] x86_pkg_temp_thermal intel_powerclamp
[604205.234049] intel_powerclamp
[604205.234050] coretemp
[604205.234051] coretemp kvm_intel
[604205.234052] kvm_intel
[604205.234053] iTCO_wdt
[604205.234053] iTCO_wdt mei_wdt
[604205.234054] mei_wdt iTCO_vendor_support
[604205.234056] iTCO_vendor_support ppdev
[604205.234057] ppdev kvm
[604205.234058] kvm irqbypass
[604205.234059] irqbypass pcspkr
[604205.234060] pcspkr i2c_i801
[604205.234061] i2c_i801 mei_me
[604205.234062] mei_me shpchp
[604205.234063] shpchp mei
[604205.234065] mei ie31200_edac
[604205.234066] ie31200_edac intel_pch_thermal
[604205.234067] intel_pch_thermal wmi
[604205.234068] wmi parport_pc
[604205.234069] parport_pc video
[604205.234070] video parport
[604205.234071] parport acpi_pad
[604205.234072] acpi_pad button
[604205.234074] button raid1
[604205.234075] raid1
[604205.234075] md_mod
[604205.234076] md_mod
[604205.234077] dm_service_time
[604205.234078] dm_service_time
[604205.234078] scsi_transport_iscsi
[604205.234079] scsi_transport_iscsi
[604205.234080] crct10dif_pclmul
[604205.234081] crct10dif_pclmul
[604205.234081] crc32_pclmul
[604205.234082] crc32_pclmul
[604205.234083] crc32c_intel ghash_clmulni_intel
[604205.234084] crc32c_intel ghash_clmulni_intel
[604205.234085] pcbc
[604205.234086] pcbc aesni_intel
[604205.234087] aesni_intel aes_x86_64
[604205.234088] aes_x86_64 crypto_simd
[604205.234089] crypto_simd glue_helper
[604205.234090] glue_helper cryptd
[604205.234092] cryptd
[604205.234093] xhci_pci
[604205.234093] xhci_pci nvme
[604205.234094] nvme e1000e
[604205.234095] e1000e xhci_hcd
[604205.234096] xhci_hcd ptp
[604205.234098] ptp serio_raw
[604205.234099] serio_raw pps_core
[604205.234100] pps_core
[604205.234101] ahci
[604205.234101] ahci
[604205.234102] nvme_core
[604205.234103] nvme_core libahci
[604205.234104] libahci usbcore
[604205.234105] usbcore
[604205.234106] sunrpc
[604205.234107] sunrpc dm_mirror
[604205.234108] dm_mirror dm_region_hash
[604205.234109] dm_region_hash dm_log
[604205.234110] dm_log sg
[604205.234112] sg dm_multipath
[604205.234113] dm_multipath dm_mod
[604205.234114] dm_mod
[604205.234115] scsi_dh_rdac
[604205.234115] scsi_dh_rdac scsi_dh_emc
[604205.234116] scsi_dh_emc scsi_dh_alua
[604205.234118] scsi_dh_alua
[604205.234122] CPU: 0 PID: 568 Comm: kdmwork-254:0 Not tainted
4.12.14-lp150.12.16-default #1 openSUSE Leap 15.0
[604205.234124] CPU: 4 PID: 617 Comm: kdmwork-254:1 Not tainted
4.12.14-lp150.12.16-default #1 openSUSE Leap 15.0
[604205.234125] Hardware name: FUJITSU /D3417-B2, BIOS V5.0.0.12 R1.8.0.SR.1
for D3417-B2x 05/16/2017
[604205.234126] Hardware name: FUJITSU /D3417-B2, BIOS V5.0.0.12 R1.8.0.SR.1
for D3417-B2x 05/16/2017
[604205.234128] task: ffff880fe0508100 task.stack: ffffc90006fe0000
[604205.234129] task: ffff880fe0500040 task.stack: ffffc90007114000
[604205.234134] RIP: 0010:nvme_setup_cmd+0x3dd/0x430 [nvme_core]
[604205.234137] RIP: 0010:nvme_setup_cmd+0x3dd/0x430 [nvme_core]
[604205.234139] RSP: 0018:ffffc90006fe3ca0 EFLAGS: 00010202
[604205.234140] RSP: 0018:ffffc90007117ca0 EFLAGS: 00010202
[604205.234143] RAX: 0000000000000000 RBX: ffffc90007117d38 RCX:
0000000000000000
[604205.234144] RAX: 0000000000000000 RBX: ffffc90006fe3d38 RCX:
0000000000000000
[604205.234145] RDX: 0000000000000002 RSI: ffff880d75ed9e20 RDI:
0000000004040c18
[604205.234146] RDX: 0000000000000002 RSI: ffff880d75ed9480 RDI:
0000000004040c18
[604205.234148] RBP: ffff880fdefd6600 R08: ffff880d75ed9e20 R09:
0000000000000008
[604205.234149] R10: ffff880fe0961200 R11: 0000000000000032 R12:
ffff880fe0582100
[604205.234150] RBP: ffff880fdf50f800 R08: ffff880d75ed9480 R09:
0000000000000008
[604205.234152] R13: 0000000000000001 R14: 0000000000000001 R15:
ffff880fdefd6600
[604205.234153] R10: ffff880fde91bac0 R11: 0000000000000031 R12:
ffff880fe0472880
[604205.234154] R13: 0000000000000001 R14: 0000000000000001 R15:
ffff880fdf50f800
[604205.234156] FS: 0000000000000000(0000) GS:ffff88102e400000(0000)
knlGS:0000000000000000
[604205.234157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[604205.234159] FS: 0000000000000000(0000) GS:ffff88102e500000(0000)
knlGS:0000000000000000
[604205.234160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[604205.234162] CR2: 00007ff3af64c000 CR3: 000000000200a002 CR4:
00000000003626f0
[604205.234163] CR2: 00007f1d336d0458 CR3: 000000000200a001 CR4:
00000000003626e0
[604205.234165] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[604205.234166] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[604205.234167] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[604205.234168] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[604205.234169] Call Trace:
[604205.234169] Call Trace:
[604205.234178] ? blk_mq_run_hw_queue+0x10/0x10
[604205.234179] ? blk_mq_run_hw_queue+0x10/0x10
[604205.234184] nvme_queue_rq+0x52/0xbc0 [nvme]
[604205.234186] nvme_queue_rq+0x52/0xbc0 [nvme]
[604205.234191] ? kmem_cache_alloc+0xea/0x510
[604205.234193] ? kmem_cache_alloc+0xea/0x510
[604205.234197] ? __sbitmap_queue_get+0x24/0x90
[604205.234198] ? __sbitmap_queue_get+0x24/0x90
[604205.234202] ? wait_woken+0x80/0x80
[604205.234203] ? wait_woken+0x80/0x80
[604205.234207] ? mempool_alloc+0x55/0x160
[604205.234209] ? mempool_alloc+0x55/0x160
[604205.234212] ? blk_mq_run_hw_queue+0x10/0x10
[604205.234214] ? blk_mq_run_hw_queue+0x10/0x10
[604205.234216] blk_mq_dispatch_rq_list+0x194/0x300
[604205.234218] blk_mq_dispatch_rq_list+0x194/0x300
[604205.234222] blk_mq_sched_dispatch_requests+0x11c/0x1d0
[604205.234224] blk_mq_sched_dispatch_requests+0x11c/0x1d0
[604205.234228] __blk_mq_delay_run_hw_queue+0x73/0x80
[604205.234229] __blk_mq_delay_run_hw_queue+0x73/0x80
[604205.234233] blk_insert_cloned_request+0x9a/0x1e0
[604205.234235] blk_insert_cloned_request+0x9a/0x1e0
[604205.234243] map_request+0xc1/0x1d0 [dm_mod]
[604205.234248] map_request+0xc1/0x1d0 [dm_mod]
[604205.234253] ? kthread_create_worker_on_cpu+0x50/0x50
[604205.234255] ? kthread_create_worker_on_cpu+0x50/0x50
[604205.234261] map_tio_request+0x12/0x30 [dm_mod]
[604205.234265] map_tio_request+0x12/0x30 [dm_mod]
[604205.234269] kthread_worker_fn+0xe5/0x190
[604205.234271] kthread_worker_fn+0xe5/0x190
[604205.234274] kthread+0x11a/0x130
[604205.234276] kthread+0x11a/0x130
[604205.234279] ? kthread_create_on_node+0x40/0x40
[604205.234281] ? kthread_create_on_node+0x40/0x40
[604205.234284] ? do_syscall_64+0x7b/0x150
[604205.234286] ? do_syscall_64+0x7b/0x150
[604205.234290] ? SyS_exit_group+0x10/0x10
[604205.234293] ? SyS_exit_group+0x10/0x10
[604205.234296] ret_from_fork+0x1f/0x40
[604205.234298] ret_from_fork+0x1f/0x40
[604205.234301] Code:
[604205.234302] Code: 5b
[604205.234303] 5b 89
[604205.234305] 89 c8
[604205.234306] c8
[604205.234307] 83
[604205.234307] 83 ce
[604205.234309] ce
[604205.234309] 10
[604205.234310] 10 c1
[604205.234311] c1 e0
[604205.234312] e0 10
[604205.234313] 10 09
[604205.234315] 09 c2
[604205.234316] c2
[604205.234317] 83
[604205.234317] 83 f9
[604205.234318] f9 04
[604205.234319] 04 0f
[604205.234320] 0f 87
[604205.234322] 87 b1
[604205.234323] b1 fe
[604205.234324] fe ff
[604205.234325] ff ff
[604205.234326] ff 8b
[604205.234327] 8b 45
[604205.234328] 45 58
[604205.234329] 58 48
[604205.234330] 48 8b
[604205.234331] 8b 7d
[604205.234332] 7d 30
[604205.234333] 30 c1
[604205.234334] c1 e8
[604205.234335] e8 09
[604205.234336] 09 48
[604205.234337] 48 01
[604205.234338] 01 84
[604205.234340] 84 cf
[604205.234341] cf 20
[604205.234342] 20 08
[604205.234343] 08 00
[604205.234344] 00 00
[604205.234345] 00 e9
[604205.234346] e9 9a
[604205.234347] 9a fe
[604205.234348] fe ff
[604205.234349] ff ff
[604205.234350] ff <0f>
[604205.234352] <0f> 0b
[604205.234353] 0b 4c
[604205.234354] 4c 89
[604205.234355] 89
[604205.234356] c7
[604205.234357] c7
[604205.234357] 41 bd
[604205.234358] 41 bd
[604205.234359] 0a 00
[604205.234360] 0a 00
[604205.234361] 00 00
[604205.234363] 00 00
[604205.234364] e8 f3
[604205.234365] e8 f3
[604205.234366] b6 f9
[604205.234367] b6 f9
[604205.234368] e0 e9
[604205.234369] e0 e9
[604205.234370] 9d fc
[604205.234371] 9d fc
[604205.234372] ff ff
[604205.234373] ff ff
[604205.234375] ---[ end trace 86232acc5cc90666 ]---
[604205.234377] ---[ end trace 86232acc5cc90667 ]---
[604205.234379] print_req_error: I/O error, dev nvme1n1, sector 67374104
[604205.234382] print_req_error: I/O error, dev nvme0n1, sector 67374104
[604205.234577] device-mapper: multipath: Failing path 259:1.
[604205.234687] device-mapper: multipath: Failing path 259:0.
[604205.234689] print_req_error: I/O error, dev dm-1, sector 67374104
[604205.234808] print_req_error: I/O error, dev dm-0, sector 67374104
[604205.234954] print_req_error: I/O error, dev dm-1, sector 67382370
[604205.234955] print_req_error: I/O error, dev dm-0, sector 67382370
[604205.235223] print_req_error: I/O error, dev dm-1, sector 67409782
[604205.235225] print_req_error: I/O error, dev dm-0, sector 67409782
[604205.235468] print_req_error: I/O error, dev dm-1, sector 67409888
[604205.235471] print_req_error: I/O error, dev dm-0, sector 67409888
[604205.235805] md/raid1:md1: dm-3: rescheduling sector 262166
[604205.235822] md/raid1:md1: dm-3: rescheduling sector 262168
[604205.236070] md/raid1:md1: redirecting sector 262166 to other mirror: dm-6
[604205.236082] md/raid1:md1: dm-6: rescheduling sector 262166
[604205.270041] md: super_written gets error=10
[604205.270153] md/raid1:md2: Disk failure on dm-7, disabling device.
md/raid1:md2: Operation continuing on 1 devices.
[604205.270354] md: super_written gets error=10
[604205.270474] md/raid1:md1: Disk failure on dm-6, disabling device.
md/raid1:md1: Operation continuing on 1 devices.
[604205.275292] md: super_written gets error=10
[604205.275437] md: super_written gets error=10
[604205.275553] md: super_written gets error=10
[604205.275676] md: super_written gets error=10
[604205.275811] md: super_written gets error=10
[604205.275933] md: super_written gets error=10
[604205.276060] md/raid1:md1: dm-3: unrecoverable I/O read error for block
262168
[604205.276077] md: super_written gets error=10
[604205.276350] md: super_written gets error=10
[604205.276468] md: super_written gets error=10
[604205.276586] md: super_written gets error=10
[604205.276587] md/raid1:md1: dm-6: unrecoverable I/O read error for block
262166
[604205.276612] EXT4-fs error (device md1): ext4_wait_block_bitmap:523: comm
fstrim: Cannot read block bitmap - block_group = 26, block_bitmap = 131083
[604205.276741] md: super_written gets error=10
[604205.277168] md: super_written gets error=10
[604205.277461] md: super_written gets error=10
[604205.314548] md: super_written gets error=10
[604205.318749] md: super_written gets error=10
[604205.319058] md: super_written gets error=10
[604205.319274] md: super_written gets error=10
[604205.334705] md: super_written gets error=10
[604205.334915] md: super_written gets error=10
[604205.335069] md: super_written gets error=10
[604205.335225] Buffer I/O error on dev md1, logical block 1, lost sync page
write
[604205.335281] md: super_written gets error=10
[604205.335455] EXT4-fs error (device md1): ext4_wait_block_bitmap:523: comm
fstrim: Cannot read block bitmap - block_group = 27, block_bitmap = 131084
[604205.335574] md: super_written gets error=10
[604205.335788] EXT4-fs (md1): previous I/O error to superblock detected
[604205.335799] Buffer I/O error on dev md1, logical block 1, lost sync page
write
[604205.350745] md: super_written gets error=10
[604205.351005] md: super_written gets error=10
[604205.351207] md: super_written gets error=10
[604205.370660] md: super_written gets error=10
[604205.370907] md: super_written gets error=10
[604205.371072] md: super_written gets error=10
[604205.371299] md: super_written gets error=10
[604205.371583] EXT4-fs error (device md2): ext4_wait_block_bitmap:523: comm
fstrim: Cannot read block bitmap - block_group = 1, block_bitmap = 1026