http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c1 Neil Rickert <nwr10cst-oslnx@yahoo.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nwr10cst-oslnx@yahoo.com --- Comment #1 from Neil Rickert <nwr10cst-oslnx@yahoo.com> --- I am probably seeing the same errors. I haven't reported, because they go past too quickly to grab a copy. I've tried to reproduce in a KVM virtual machine, but without success. I don't worry much about it, because the file systems check out as clean. So nothing is being corrupted. Maybe it is trying to disassemble something that does not need to be disassembled (i.e. could just be abandoned). I get this on both Tumbleweed and Leap 15.0. And occasionally I see a clean shutdown after updates. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c9 Ronnie Bailey <purevw@wtxs.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |purevw@wtxs.net --- Comment #9 from Ronnie Bailey <purevw@wtxs.net> --- There are 2 primary errors on my system. I had to shoot a video to be able to catch and read them. There may be 60 or more repeats over a half second time period. I should note that this happens 80% of the time. So there are times when shutdown is without errors. Error 1: device-mapper: remove ioctl on system-root failed: Device or resource busy (system-root is my encrypted LVM Luks root partition) Error 2: device-mapper: remove ioctl on cr_scsi_hard_drive_partition_containing_root My /home is also a partition within the same LVM, but never produces errors. SLES 11 SP4 had a problem that produced the same error in 2016, but it's possible that the cause is not identical. Suse document is 7017390 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c11 --- Comment #11 from Martin Wilck <martin.wilck@suse.com> --- Could you try this: (https://mirrors.edge.kernel.org/pub/linux/utils/boot/dracut/dracut.html#debu...) # mkdir -p /run/initramfs/etc/cmdline.d # echo "rd.debug rd.break=pre-shutdown rd.break=shutdown" > /run/initramfs/etc/cmdline.d/debug.conf # touch /run/initramfs/.need_shutdown You should now be able to watch better what's going on.

remove ioctl on system-root failed

Once we're back in the initramfs, shutting down the root LV should work - unless it's still in use by some process. This is only a real problem if the situation persists (e.g. because a file system hasn't been unmounted). bug 1083392 is quite interesting in this respect. It seems to be quite common that such "remove ioctl" errors occur transiently, and succeed after a few retries. The error messages are printed nonetheless. If it's just that, we may want to simply silence the error message and only print a message after the last retry has failed. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c13 --- Comment #13 from Martin Wilck <martin.wilck@suse.com> --- Please try setting STARTMODE=nfsroot for the network connection that is used for netconsole. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c15 --- Comment #15 from Martin Wilck <martin.wilck@suse.com> --- Affected people, please check if /var/run and /var/lock on your systems are directories (bind mounts) or symlinks. The former may be the case if the system is an old installation with a long history of being updated. The latter would be the case on new installations. It they are directories, please move them away (will require an umount -l first) and create symlinks /var/run -> /run /var/lock -> /run/lock I've recently been debugging a similar case where this workaround helped. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c17 --- Comment #17 from Martin Wilck <martin.wilck@suse.com> --- (In reply to Christoph Obexer from comment #16)

Both are symlinks on my system but I still experience the disassembly problem. My Symlinks are apparently from 28. June 2017.

Ok, so my guess was wrong. Maybe we can make progresss if someone could gather logs as described under "Shutdown Completes Eventually" on https://freedesktop.org/wiki/Software/systemd/Debugging/. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c19 --- Comment #19 from Michiel Janssens <michiel@nexigon.net> --- Created attachment 769906 --> http://bugzilla.opensuse.org/attachment.cgi?id=769906&action=edit fstab -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c21 --- Comment #21 from Michiel Janssens <michiel@nexigon.net> --- Created attachment 769908 --> http://bugzilla.opensuse.org/attachment.cgi?id=769908&action=edit rdsosreport_shutdown -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c23 --- Comment #23 from Michiel Janssens <michiel@nexigon.net> --- As I was also was getting these shutdown messages, I tried to get some logging info as requested. see attached files. I hope this will be helpful in determining if the messages should be suppressed or not. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c24 --- Comment #24 from Ronnie Bailey <purevw@wtxs.net> --- I had opened a thread in the openSUSE Tumbleweed forum regarding this problem. A user has found a workaround that may at least give a hint as to what is going on. It was posted that: "Adding "plymouth.enable=0" to the kernel parameters makes the "device-mapper: remove ioctl on XXXXX failed: Device or resource busy" messages disappear." I have no idea if Plymouth is the problem or just a direction to look. I can verify that adding the above kernel parameter has stopped the error in my Tumbleweed machine. My original thread may be found at: https://forums.opensuse.org/showthread.php/530530-device-mapper-remove-ioctl... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c26 --- Comment #26 from Martin Wilck <martin.wilck@suse.com> --- My guess would be that disabling plymouth only hides the messages. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c28 --- Comment #28 from Martin Wilck <martin.wilck@suse.com> --- (In reply to Neil Rickert from comment #27)

I am now seeing these errors on every shutdown. This is with a KVM virtual machine, where I previously never could produced the problem. And I do have "plymouth.enable=0".

IIUC use see these messages for a while, then they stop and the reboot succeeds (how long? how many messages)? If actually some library from the root FS was used by some process running outside of it, then the error should persist and the remove ioctl should consistently fail, causing the shutdown process to time out.

My personal opinion: this is all a mistake. The system should not try to disassemble device mapper setup.

I'm not sure where this happens. I guess it's the stop job of "blk-availability.service". Could you try to mask that service and see if the problem disappears? Wrt this being a "mistake", the blkdeactivate script was introduced to LVM by Red Hat in 2012. I didn't find an explicit rationale, but as they're unlikely to have written that code without a purpose, I guess they were seeing data corruption of some sort if it's not done. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c30 --- Comment #30 from Martin Wilck <martin.wilck@suse.com> --- (In reply to Neil Rickert from comment #29)

...

(how long? how many messages)?

They go past so fast, that it is very hard to tell.

So it's really a transient condition, and the problem is mainly the irritating log messages, no real damage done. Might be solvable by adding a short sleep somewhere (but I can't say where :-/ ).

...

Could you try to mask that service and see if the problem disappears?

How and when should I do that?

When the system is up, run "systemctl mask blk-availability.service". You probably need to reboot after that. But wait - first check whether that service is enabled at all ("systemctl status blk-availability.service"). If it isn't, we have to find another culprit. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c32 Martin Wilck <martin.wilck@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS --- Comment #32 from Martin Wilck <martin.wilck@suse.com> ---

I agree that no real damage is done.

OK, good. If you're willing to invest some more work nonetheless: As you're seeing it on a VM, maybe you could provide a serial console log taken with "systemd.log_target=console systemd.journald.forward_to_console=1 console=ttyS0,57600n8"? In libvirt, you can connect e.g. "screen" to the serial port of the VM (which appears as /dev/pts/$X device on the host, you can see it e.g. with "virsh dumpxml"), and log the screen session. qemu has various methods to redirect a virtual serial console to a file. The reason I'm asking is to identify at which point exactly in the shutdown sequence the messages occur. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c34 --- Comment #34 from Neil Rickert <nwr10cst-oslnx@yahoo.com> --- Created attachment 780732 --> http://bugzilla.opensuse.org/attachment.cgi?id=780732&action=edit serial console log Well, that was easier than expected. I have attached the serial console log. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 Maximilen Bullett <mlbullett@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mlbullett@gmail.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c36 --- Comment #36 from Achim Gratz <Stromeko@NexGo.DE> --- The problem has stopped appearing somewhere around the 4.18.5 kernel update on Tumbleweed and has not returned so far. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c38 --- Comment #38 from Ronnie Bailey <purevw@wtxs.net> --- I also continue to see the messages, although they disappear during shutdown immediately after a kernel update. After the following boot, the problem is back. I might note that my practice is to always remove the existing kernel at the same time I update to a newer kernel and not wait for the system to do it at another time. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c40 - - <mikccc@tutanota.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mikccc@tutanota.com --- Comment #40 from - - <mikccc@tutanota.com> --- https://openqa.opensuse.org/tests/799295#step/shutdown/6 On my system: With 'rd.break=shutdown' (after the error messages) /oldroot remains mounted with a 'ro' option, but one time it was 'rw' (because something related with btrfs balance or fs sync timed out). It seems that dracut doesn't succeed after many failed retries but gives up. (Adding the 'rd.debug' option results in a complete freeze of my system every time) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c41 --- Comment #41 from - - <mikccc@tutanota.com> --- There are 252 (42 * 6) lines of device-mapper's error messages in the Neil's log. Dracut tries 42 times and gives up: _cnt=0 while [ $_cnt -le 40 ]; do _check_shutdown && break _cnt=$(($_cnt+1)) done [ $_cnt -ge 40 ] && _check_shutdown final https://github.com/dracutdevs/dracut/blob/master/modules.d/99shutdown/shutdo... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c43 --- Comment #43 from - - <mikccc@tutanota.com> --- After running `kill -s TERM 1454` I could unmount /oldroot with no error. I found also a line identyfying the mysterious kthread named [none] in logs:

kernel: bpfilter: Loaded bpfilter_umh pid 1454

-- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 Hanns-Joachim Uhl <hannsj_uhl@de.ibm.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hannsj_uhl@de.ibm.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c49 --- Comment #49 from - - <mikccc@tutanota.com> --- First of all, I have to correct my last comment - the 'none' process wasn't a kthread, because it lacked the PF_KTHREAD flag. (In reply to Martin Wilck from comment #46)

Here's something of interest:

https://lkml.org/lkml/2018/10/20/259

There is also a patch in systemd 240: https://github.com/systemd/systemd/commit/e45154c770567507cc51eb78b28a1fae1f... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c52 --- Comment #52 from - - <mikccc@tutanota.com> --- (In reply to Martin Wilck from comment #50)

Could you please test with 4.20.1 as requested in comment 47?

I'm sorry, but I would prefer to wait for the release in the official repo. (Won't it appear any day now?) I had recently even more issues with the rd.break=shutdown option, so I'll try to give you a deatailed answer then, but I cannot guarantee it. Maybe could you consider adding an extra test to openqa, which could show you most reliably if the patches have had the desired effect? It would also be useful for detecting similar issues in the future.

...

There is also a patch in systemd 240: https://github.com/systemd/systemd/commit/ e45154c770567507cc51eb78b28a1fae1fcdf396

That one looks orthogonal with what we are discussing here. It deals with _not killing_ kernel threads, where we must ensure that systemd _does kill_ this user mode helper.

It corrects kthread distinguishing to ensure killing user mode helpers. Checking the PF_KTHREAD flag is the right solution IMO and dracut too should implement it. The old hacky way - checking if /proc/{PID}/cmdline is empty, isn't reliable enough as we all have seen ;) I consider the kernel patches as a workaround of the problem. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c54 --- Comment #54 from - - <mikccc@tutanota.com> --- (In reply to Martin Wilck from comment #53)

...

I consider the kernel patches as a workaround of the problem.

Not sure what you meant to say, but judging from the other testers' feedback, it looks as if these patches were actually the solution to the problem described in this bug. I wouldn't call this a workaround.

Workarounds are solutions by definition. Poor solutions. Anyway, I can confirm that the issue has been solved in the Tumbleweed 20190115 snapshot: kernel-default-4.20.0-1.5.x86_64 systemd-239-3.1.x86_64 dracut-044.1-22.1.x86_64 The error messages have stopped appearing. /oldroot and /oldroot/var mounts: - both ro at the pre-shutdown breakpoint; - not listed at the shutdown breakpoint. The cmdline entry in /proc/<pid> for the bpfilter umh process contains "bpfilter_umh" followed by a null byte. The dir isn't available at the shutdown breakpoint (or - in other words - the process has been killed earlier, as expected).

@all, do we have a fool-proof setup procedure for a VM to reproduce this problem?

I suspect that the issue affects every system where bpfilter_umh is started, but verbose errors appear only if the dracut's shutdown hook is enabled. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c56 --- Comment #56 from Ronnie Bailey <purevw@wtxs.net> --- I upgraded to kernel-default-4-20 this morning. After reboot into the new kernel, and shut-down, the problem still exists. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c58 --- Comment #58 from - - <mikccc@tutanota.com> --- (In reply to Martin Wilck from comment #55)

...

...

@all, do we have a fool-proof setup procedure for a VM to reproduce this problem?

I suspect that the issue affects every system where bpfilter_umh is started, but verbose errors appear only if the dracut's shutdown hook is enabled.

I haven't yet figured out which service starts the bpfilter_umh process. Have you? I suspected firewalld, but I've seen bpfilter_umh running on systems with firewalld disabled.

I suspect NetworkManager (on the basis of the order of log entries and boot messages). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c60 --- Comment #60 from Martin Wilck <martin.wilck@suse.com> --- (In reply to Ronnie Bailey from comment #59)

I set network management to "Wicked" via Yast. NetworkManager closed. I still got the messages on shutdown. Something curious though. On reboot, NetworkManager was running. I went to Yast and it still showed Wicked as the service to use.

Different issue, please open a separate bug.

Earlier, I removed the bpfilter module. lsmod had showed it being used by 0. When I initiated shutdown, the first console message I got was "starting bpfilter". But during that shutdown there were no messages.

Could you try that after activating systemd debugging ("systemd-analyze log-level debug")? It might show us which service loads the module.

Shutdowns on my system typically take 90 seconds or so, which seems quite long to me.

Yes indeed. 90s is the standard systemd timeout for jobs that don't finish. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 http://bugzilla.opensuse.org/show_bug.cgi?id=1080485#c62 --- Comment #62 from Martin Wilck <martin.wilck@suse.com> --- (In reply to - - from comment #61)

After a little research I found that it can be anything that calls `(set|get)sockopt`.

Thanks for digging that up. More precisely, getsockopt() or setsockopt() with level == IPPROTO_IP and optname supported by bpfilter (https://elixir.bootlin.com/linux/v5.0-rc4/source/include/uapi/linux/bpfilter...) automagically loads the bpfilter module and starts the UMH. This happens when iptables initializes itself: it calls getsockopt(IPPROTO_IP, IPT_SO_GET_INFO), and IPT_SO_GET_INFO has the same value as BPFILTER_IPT_SO_GET_INFO (=64). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1080485 fons jongh <fons.dejongh@microfocus.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |1172006 -- You are receiving this mail because: You are on the CC list for the bug.