http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c2 --- Comment #2 from Olli Tuominen --- What logs I should provide to pin down the issue? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c5 --- Comment #5 from Neil Rickert ---

I'm wondering how you can deduce from the screenshot that this is happening after all file systems have been unmounted.

I made that comment about "unmounted". It is based on what I see when shutting down Tumbleweed. In my grub boot line, I have removed "splash=silent". So I see lots of boot messages and lots of shutdown messages. When I see that series of "Command failed", I have already seen the unmount messages, including one "failed to unmount" message (which seems an unimportant "bug"). Otherwise, what I see is almost identical to the screenshot, except that the last line is usually "restarting" instead of "powering off". -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c7 Olli Tuominen changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(fbui@suse.com) --- Comment #7 from Olli Tuominen --- Is "systemd-analyze set-log-level debug" permanent? Is "systemd-analyze set-log-level notice" default or how I revert it back? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c9 --- Comment #9 from Franck Bui --- It would be great if you could have a look at: https://freedesktop.org/wiki/Software/systemd/Debugging/#index2h1 More accurately: - Does "reboot -f" work ? - Does the reboot finally happen after waiting a few minutes (at least 3min) ? Thanks. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c11 --- Comment #11 from Franck Bui --- (In reply to Franck Bui from comment #9)

- Does the reboot finally happen after waiting a few minutes (at least 3min) ?

-- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c13 --- Comment #13 from Franck Bui --- (In reply to Olli Tuominen from comment #12)

(In reply to Franck Bui from comment #11)

...

(In reply to Franck Bui from comment #9)

...

- Does the reboot finally happen after waiting a few minutes (at least 3min) ?

Yes, almost immediately.

I meant when doing "systemctl shutdown" or "shutdown" (without the '-f' switch). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c15 --- Comment #15 from Olli Tuominen --- Created attachment 740189 --> http://bugzilla.opensuse.org/attachment.cgi?id=740189&action=edit dmesg -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c17 --- Comment #17 from Olli Tuominen --- failed to get those to usb. :( -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c19 --- Comment #19 from Olli Tuominen --- Dualboot Leap + Win10. Disk is SSD. dm-0 is root and btrfs. fdisk -l Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xf6c468e5 Device Boot Start End Sectors Size Id Type /dev/sda1 * 2048 1026047 1024000 500M 7 HPFS/NTFS/exFAT /dev/sda2 1026048 214845439 213819392 102G 7 HPFS/NTFS/exFAT /dev/sda3 214845440 976773119 761927680 363.3G 8e Linux LVM Disk /dev/mapper/system-root: 40 GiB, 42949672960 bytes, 83886080 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/system-swap: 2 GiB, 2147483648 bytes, 4194304 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/system-home: 200 GiB, 214748364800 bytes, 419430400 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c21 --- Comment #21 from Neil Rickert ---

OK so apparently there's something wrong with /dev/dm-0

Since I reported seeing the same sort of messages with Tumbleweed, I'll comment here. I use an encrypted LVM. And "/dev/dm-0" is the access to that LVM. Inside the LVM, I have volumes "root1", "root2", "root3", "swap", "home". I use "ext4" for home and root. If the root file system is the "root1" volume, then the "home" volume is mounted at "/home". If the root file system is either the "root2" or "root3" volume, then the "home" volume is mounted at "/xhome". The shutdown messages seem to show the crypto being properly undone. And that happens before the "Command failed" messages show up. If I mount the "home" volume at "/xhome", then I usually see a message "Failed to unmount /xhome". I have checked by booting rescue media, and that file system shows as clean. So I think it fails to unmount it because it has already unmounted it, though I cannot see where. If I have given more detail than is relevant, please ignore that extra detail. Hmm, here's another point I should mention. I also have Tumbleweed on a spare partition on a laptop. In that case, the root file system is "/dev/sda8" and is not encrypted at all. But I do still mount "home" from the encrypted LVM at "/xhome" and I still use swap from the encrypted LVM. On that system, I still get the "Command failed" messages on shutdown. And I still see the "failed to unmount /xhome" (before the "Command failed"). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c23 --- Comment #23 from Neil Rickert --- Following the instructions from C14 The last few lines of the "dmesg" output (abbreviated) were: all DM devices detached spawned ... mdadm.shutdown as 3650 ... shutdown succeeded Successfully changed to root pivot returned to initrd watchdog: watchdog0: watchdog did not stop Break before pre-shutdown (shell prompt) I then entered CTRL-D That gave me another debug prompt, and a suggestion to save a file to a USB or elsewhere. However, it rebooted before I had time to try that. I'm guessing that the "watchdog did not stop" is related to the "Command failed" messages. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Command failed on shutdown |"Command failed" messages |phase |on shutdown phase -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c26 --- Comment #26 from Franck Bui --- (In reply to Franck Bui from comment #18)

OK so apparently there's something wrong with /dev/dm-0.

After some research it appears that the warnings: systemd-shutdown[1]: Detaching DM 254:0. systemd-shutdown[1]: Could not detach DM /dev/dm-0: Device or resource busy systemd-shutdown[1]: Not all DM devices detached, 1 left. are emitted because we're trying to detach the device that contains the rootfs. And this is doomed to fail as we're currently using it. Normally we should detect this case and skip detaching the DM device hosting the rootfs but in your case BTRFS is used and the detection was failing. I submitted a patch to fix that: https://github.com/systemd/systemd/pull/6811 However the original issue remains... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c28 --- Comment #28 from Olli Tuominen --- Created attachment 740587 --> http://bugzilla.opensuse.org/attachment.cgi?id=740587&action=edit journalctl -b -1 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c30 --- Comment #30 from Franck Bui --- (In reply to Olli Tuominen from comment #29)

So I did a set -x to /usr/lib/dracut/modules.d/90dm/dm-shutdown.sh

Did you recreated the initrd at that point (by running mkinitrd) ?

Rebooted and used "plymouth..." for booting option. Did C#14 and rebooted.

Attached the journalctl after rebooting.

Nothing interesting is supposed to be recorded in the journal since all happens when journald is off... only on the console or in dmesg. BTW can you still see the "Command failed" messages ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c32 --- Comment #32 from Olli Tuominen --- ok - so I got following: ... ++ devname=system-root ++ dmsetup -v --noudevsync remove system-root Command failed ++ return 1 ++ ret=1 ++ 'xfinal '!=' x ']' ++ info 'dmsetup ls --tree' ++ check_quiet ++ '[' -z yes ']' ++ echo '<30>dracut: dmsetup ls --tree' ++ dmsetup ls --tree ++ vinfo ++ read line ++ info system-root '(254:1)' ++ check_quiet ... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c34 --- Comment #34 from Franck Bui --- (In reply to Olli Tuominen from comment #32)

ok - so I got following:

... ++ devname=system-root ++ dmsetup -v --noudevsync remove system-root Command failed

Good we at least spotted the command that fails... What if you manually try to execute the failing command in the debug shell spawned by dracut ? Can you see more messages ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c36 --- Comment #36 from Olli Tuominen --- I check with lsof dm-0 but it didn't show anything. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c38 --- Comment #38 from Franck Bui --- (In reply to Daniel Molkentin from comment #37)

Can you prefix the command with "strace -f" and attach the output (goes to stderr by default)?

I think you should explain how to embed strace in the initrd as it's not really obvious. Also if a (shutdown) hook fails, couldn't dracut emit a warning which includes the name of the hook or the relevant module ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c40 --- Comment #40 from Thomas Blume --- (In reply to Olli Tuominen from comment #36)

I check with lsof dm-0 but it didn't show anything.

Might be good to get the status of the lv, can you please edit: /usr/lib/dracut/modules.d/90dm/dm-shutdown.sh and just before the row: dmsetup -v --noudevsync remove "$devname" || return $? put: dmsetup info -v "$devname" ? Afterwards please recreate the initrd (you might want to backup the old one before) and reproduce the issue. The console output should show the status of the lv, just before removal. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c42 --- Comment #42 from Olli Tuominen --- Created attachment 740630 --> http://bugzilla.opensuse.org/attachment.cgi?id=740630&action=edit reply for C#41 dmsetup info -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c44 --- Comment #44 from Franck Bui --- Olli can you try to get the debug shell at the "shutdown" phase and show the output of the following command: - cat /proc/mounts - umount /oldroot Thanks. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c46 --- Comment #46 from Olli Tuominen --- Once I unmounted that /oldroot I was able to execute "dmsetup -v --noudevsync remove system-root" command It seems to be somekind of snapshot of the root fs? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c48 --- Comment #48 from Franck Bui --- (In reply to Olli Tuominen from comment #46)

Once I unmounted that /oldroot I was able to execute "dmsetup -v --noudevsync remove system-root" command

It seems to be somekind of snapshot of the root fs?

Huh ? you were able to umount /oldroot yourself without any error ? That would mean that dracut skips the unmount command entirely for some reasons in your case. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c50 --- Comment #50 from Franck Bui --- (In reply to Olli Tuominen from comment #49)

Can it be timing issue?

I don't think so as everything is executed sequentially during the shutdown process of dracut. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c52 --- Comment #52 from Olli Tuominen --- Just to verify that my 54 line starts with _cnt=0 and before it I'll put set -x exec 2>/dev/kmsg ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c54 --- Comment #54 from Olli Tuominen --- So today came the dracut patch - and it seems to fix it (something). At least I did just reboot and I didn't see the message(s) anymore. The /oldroot is mounted on <pre-shutdown> stage but when I <Ctrl>-d to <shutdown> stage it has disappeared. I copied <pre-shutdown> dmesg to /oldroot/tmp/ but I cannot find it from the /tmp/ now. Is it actually a snapshot? So I think I'll test boot couple of times and if Neil Rickert hasn't encounter issues anymore we can close this. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c56 --- Comment #56 from Olli Tuominen --- (In reply to Franck Bui from comment #55)

...

The /oldroot is mounted on <pre-shutdown> stage but when I <Ctrl>-d to <shutdown> stage it has disappeared.

I copied <pre-shutdown> dmesg to /oldroot/tmp/ but I cannot find it from the /tmp/ now. Is it actually a snapshot?

You were asked to copy the output of dmesg in /oldroot, not in /oldroot/tmp...

Can you please follow the instructions and provide the output of dmesg ?

Can you provide instructions how to do so as in shutdown stage there is no more /oldroot I can only do it in pre-shutdown when /oldroot is still mounted. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c58 --- Comment #58 from Olli Tuominen --- Created attachment 740748 --> http://bugzilla.opensuse.org/attachment.cgi?id=740748&action=edit pre-shutdown stage dracut debug log -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c60 --- Comment #60 from Franck Bui --- (In reply to Olli Tuominen from comment #59)

(In reply to Franck Bui from comment #57)

...

Please at the *shutdown* stage, issue the following command:

- cat /proc/mounts

missing the /oldroot

...

- dmsetup info -v system-root

Device does not exist

I'm puzzled... According to your comment #32, "dmsetup remove" is supposed to the failing command. And this command is executed by lib/dracut/hooks/shutdown/25-dm-shutdown.sh. In its turn this script is supposed to be called between "pre-shutdown" and "shutdown" by /shutdown. Can you comment the changes in /usr/lib/dracut/modules.d/99shutdown/shutdown.sh requested in comment #51, recreate the initrd and retry ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c62 --- Comment #62 from Olli Tuominen --- (In reply to Franck Bui from comment #60)

I'm puzzled...

According to your comment #32, "dmsetup remove" is supposed to the failing command. And this command is executed by lib/dracut/hooks/shutdown/25-dm-shutdown.sh.

Yep, I did the change to C#29 (included sleep so got that message out)

In its turn this script is supposed to be called between "pre-shutdown" and "shutdown" by /shutdown.

Can you comment the changes in /usr/lib/dracut/modules.d/99shutdown/shutdown.sh requested in comment #51, recreate the initrd and retry ?

I'll attach two pic which systemd-shutdown fails to unmount but /shutdown is able to do so. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c64 --- Comment #64 from Olli Tuominen --- Created attachment 740848 --> http://bugzilla.opensuse.org/attachment.cgi?id=740848&action=edit dmesg shutdown -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c67 Olli Tuominen changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(otuominen@suse.co | |m) | --- Comment #67 from Olli Tuominen --- So I'm little bit lost. In Comment 27/29 we figured out that /usr/lib/dracut/modules.d/90dm/dm-shutdown.sh causes the messages, right? As the set -x and sleep reveal that it was the case. In pre-shutdown the mount is still there and in shutdown it's not. And if I manually try unmount /oldroot (in pre-shutdown) it works but not with the "dmsetup -v --noudevsync remove system-root" (as it's active?). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c70 --- Comment #70 from Olli Tuominen --- Or actually to comment #68 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c72 --- Comment #72 from Neil Rickert --- Just a note. After recent Tumbleweed updates, I am no longer seeing those "Command failed" messages. However, I am now sometimes getting messages: remove ioctl for seagate1-root1 failed device or resource busy That's approximate wording. The messages go past too fast to read them clearly. I'll note that this is evidently referring to the logical volume for the root file system (which is part of an encrypted LVM). I'll also note that if I check after the shutdown has completed, that file system seems to be clean. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c74 Olli Tuominen changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(otuominen@suse.co | |m) | --- Comment #74 from Olli Tuominen --- (In reply to Franck Bui from comment #73)

@Olli, could you test the procedure given in comment #71 without updating your system for now ?

Nope. It didn't print anything. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c89 --- Comment #89 from Olli Tuominen --- Created attachment 742990 --> http://bugzilla.opensuse.org/attachment.cgi?id=742990&action=edit dmesg when issue happens -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c91 --- Comment #91 from Olli Tuominen --- Created attachment 743004 --> http://bugzilla.opensuse.org/attachment.cgi?id=743004&action=edit dmesg v5 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 http://bugzilla.opensuse.org/show_bug.cgi?id=1055641#c93 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|systemd-maintainers@suse.de |daniel.molkentin@suse.com --- Comment #93 from Franck Bui --- Ok I think we made some progress here and thanks to Olli for all the feedback and testing he provided so far. So here is what's happening from my understanding: During shutdown, systemd stops all services and kills all remaining processes which are still alive (systemd killing spree). However some processes are excluded from the killing spree: all processes whose first character of the zeroth command line argument is '@'. And plymouth is one of those since it still wants to run after dracut takes over. Once dracut becomes PID1, it kills all remaining processes that still references /oldroot (the path where the rootfs is now mounted RO) in order to umount it definitively. Plymouth is the process that still holds reference to /oldroot at that point and therefore is killed by dracut. And I think this is where the problem lives: dracut doesn't wait for plymouth to exit. Instead it sends the KILL signal (done by "killall_proc_mountpoint /oldroot") and then right after tries to umount /oldroot (done by umount_a()). But in this case, plymouth still exists and the umount fails. Another strange part is the way dracut tries to umount /oldroot in a loop: _cnt=0 while [ $_cnt -le 40 ]; do umount_a 2>/dev/null || break _cnt=$(($_cnt+1)) done [ $_cnt -ge 40 ] && umount_a unlike its name suggests, "umount_a" only unmounts /oldroot. If umount_a() succeeds, dracut will continue to umount /oldroot even if the exit status suggests that /oldroot was unmounted and there's no point to continue. OTOH if umount_a() fails then dracut won't do any other attempts and will break the loop. Note also that all errors are hidden since stderr of umount_a() is redirected to /dev/null. Anyways this hopefully shows that the issue lives in dracut so I'm reassigning this bug to Daniel. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1055641 Michiel Janssens changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |michiel@nexigon.net -- You are receiving this mail because: You are on the CC list for the bug.