[opensuse-factory] [Tumbleweed] dracut wails about not being able to disassemble device-mapper
Since a few weeks dracut spews about two pages of errors/warnings on shutdown. Yesterday I halted the system instead of poweroff to be able to read those and it seems that the unmount of /oldroot fails and then the disassembly of the device-mapper devices errors out (maybe there'd been other errors before that that already rolled off the screeb). Does anybody have an idea what is going on and how to fix it? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
08.07.2017 19:17, Achim Gratz пишет:
Since a few weeks dracut spews about two pages of errors/warnings on shutdown. Yesterday I halted the system instead of poweroff to be able to read those and it seems that the unmount of /oldroot fails and then the disassembly of the device-mapper devices errors out (maybe there'd been other errors before that that already rolled off the screeb). Does anybody have an idea what is going on and how to fix it?
Similar report on forums: https://forums.opensuse.org/showthread.php/525843-reboot-problem?p=2829373#p... I guess bug report is needed. I do not see it, but then this could well be yet another race condition. Both of you seem to use LVM or crypt while I have just plain (virtual) disk. Do you have any chance to use serial or netconsole? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Andrei Borzenkov writes:
08.07.2017 19:17, Achim Gratz пишет:
Since a few weeks dracut spews about two pages of errors/warnings on shutdown. Yesterday I halted the system instead of poweroff to be able to read those and it seems that the unmount of /oldroot fails and then the disassembly of the device-mapper devices errors out (maybe there'd been other errors before that that already rolled off the screeb). Does anybody have an idea what is going on and how to fix it?
Similar report on forums: https://forums.opensuse.org/showthread.php/525843-reboot-problem?p=2829373#p...
That seems to be a different problem. I have no problems restarting or shutting down, aside from dracut complaining about not being able to unmount /oldroot and then getting lots of errors from the device-mapper.
I guess bug report is needed. I do not see it, but then this could well be yet another race condition.
OK, I'll try to report it tomorrow.
Both of you seem to use LVM or crypt while I have just plain (virtual) disk.
It's a standard Tumbleweed install on bare metal. No crypt, but LVM is used (again, was offered as default) to split the full disk into three volumes for root, swap and home.
Do you have any chance to use serial or netconsole?
Not easily. The system has serial ports, but I've never used them for booting and I'd have to use another computer as my terminal. I have no idea what netconsole is or how to use it. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2017-07-08 22:52, Achim Gratz wrote:
Andrei Borzenkov writes:
Do you have any chance to use serial or netconsole?
Not easily. The system has serial ports, but I've never used them for booting and I'd have to use another computer as my terminal. I have no idea what netconsole is or how to use it.
netconsole is a method that the kernel can dump messages via the ethernet connection to another computer, bypassing some of the IP stack. I have used it, so I can dig out my notes on it. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
08.07.2017 23:52, Achim Gratz пишет:
Andrei Borzenkov writes:
08.07.2017 19:17, Achim Gratz пишет:
Since a few weeks dracut spews about two pages of errors/warnings on shutdown. Yesterday I halted the system instead of poweroff to be able to read those and it seems that the unmount of /oldroot fails and then the disassembly of the device-mapper devices errors out (maybe there'd been other errors before that that already rolled off the screeb). Does anybody have an idea what is going on and how to fix it?
Similar report on forums: https://forums.opensuse.org/showthread.php/525843-reboot-problem?p=2829373#p...
That seems to be a different problem. I have no problems restarting or shutting down, aside from dracut complaining about not being able to unmount /oldroot and then getting lots of errors from the device-mapper.
The very fact that system jumps back to dracut is common. It is not done everywhere.
I guess bug report is needed. I do not see it, but then this could well be yet another race condition.
OK, I'll try to report it tomorrow.
Both of you seem to use LVM or crypt while I have just plain (virtual) disk.
It's a standard Tumbleweed install on bare metal. No crypt, but LVM is used (again, was offered as default) to split the full disk into three volumes for root, swap and home.
Yes, and as far as I can tell if LVM is used it automatically jumps back into initrd on shutdown. I briefly tried to force it on my system, but do not see any errors, so I assume it is related to using LVM or similar.
Do you have any chance to use serial or netconsole?
Not easily. The system has serial ports, but I've never used them for booting and I'd have to use another computer as my terminal. I have no idea what netconsole is or how to use it.
https://www.kernel.org/doc/Documentation/networking/netconsole.txt -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Andrei Borzenkov writes:
Not easily. The system has serial ports, but I've never used them for booting and I'd have to use another computer as my terminal. I have no idea what netconsole is or how to use it.
https://www.kernel.org/doc/Documentation/networking/netconsole.txt
Hmm. Doesn't seem to be present in Tumbleweed: # insmod netconsole netconsole=@/,@… insmod: ERROR: could not load module netconsole: No such file or directory Also, /sys/kernel/config does not exist, so it's either someplace else or dynamic configuration isn't possible as well (if it was compiled into the kernel rather than a module). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Waldorf MIDI Implementation & additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
09.07.2017 11:25, Achim Gratz пишет:
Andrei Borzenkov writes:
Not easily. The system has serial ports, but I've never used them for booting and I'd have to use another computer as my terminal. I have no idea what netconsole is or how to use it.
https://www.kernel.org/doc/Documentation/networking/netconsole.txt
Hmm. Doesn't seem to be present in Tumbleweed:
bor@tw:~> zgrep NETCON /proc/config.gz CONFIG_NETCONSOLE=m CONFIG_NETCONSOLE_DYNAMIC=y bor@tw:~> /usr/sbin/modinfo netconsole filename: /lib/modules/4.11.8-1-default/kernel/drivers/net/netconsole.ko license: GPL description: Console driver for network interfaces author: Maintainer: Matt Mackall <mpm@selenic.com> srcversion: 518A3CA9740FF6BF6BA9715 depends: configfs intree: Y vermagic: 4.11.8-1-default SMP preempt mod_unload modversions parm: netconsole: netconsole=[src-port]@[src-ip]/[dev],[tgt-port]@<tgt-ip>/[tgt-macaddr] (string) parm: oops_only:Only log oops messages (bool) bor@tw:~>
# insmod netconsole netconsole=@/,@… insmod: ERROR: could not load module netconsole: No such file or directory
Of course. You should never use insmod unless you know exactly what you are doing. You should use modprobe.
Also, /sys/kernel/config does not exist,
It does after you load netconsole. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Andrei Borzenkov writes:
Of course. You should never use insmod unless you know exactly what you are doing. You should use modprobe.
Well, that's what the text you pointed to said I should try.
Also, /sys/kernel/config does not exist,
It does after you load netconsole.
Yes, but luckily I don't need it. I can just modprobe with the correct parameters. It does not use the correct interface (tries eth0, which I don't have since it gets renamed to em1 early in the boot sequence) unless I explicitly name that. The remote system then sees console messages, e.g. when I plug in a USB stick or remove it. However, none of the shutdown messages ever make it to the remote system. I suspect the network get already unconfigured before it gets there (the system uses wicked). Anyway, I've then tried to stop the scrolling of the messages in an attempt to read more of what's going on and that seems to have triggered plymouthd to not terminate. The result was that it still failed to unmount /oldroot, but identified plymouthd as the source now. About half of the device-mapper teardown the suceeded and the rest still failed, so your suspicion of a race somewhere might well be true. The last few lines from dracut are: dmsetup ls --tree system-root (254:0) `- (8:2) ...which probably tells me the same thing as dracut not being able to unmount /oldroot. [I'd have filed a bug, but I get a gateway error when accessing bugzilla.] Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf rackAttack: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
08.07.2017 19:17, Achim Gratz пишет:
Since a few weeks dracut spews about two pages of errors/warnings on shutdown. Yesterday I halted the system instead of poweroff to be able to read those and it seems that the unmount of /oldroot fails and then the disassembly of the device-mapper devices errors out (maybe there'd been other errors before that that already rolled off the screeb). Does anybody have an idea what is going on and how to fix it?
Sorry, but you will need to debug it. If unmounting of /oldroot fails, some process was stuck and not killed. device-mapper are just followup errors, as LVM cannot be stopped at this point. Any debugging will involve some way to capture late console output. Here is example of console output in my case (clean shutdown). Are you sure you do not misinterpret "Warning: Unmounted /oldroot" or "Failed to read reboot parameter file" or preceding "device or resource busy" as errors? [ OK ] Reached target Shutdown. [ 32.943171] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 32.945135] systemd-journald[775]: Received SIGTERM from PID 1 (systemd-shutdow). [ 33.076748] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 33.078616] systemd-shutdown[1]: Unmounting file systems. [ 33.079348] systemd-shutdown[1]: Remounting '/var/log' read-only with options 'space_cache,subvolid=274,subvol=/@/var/log'. [ 33.200819] BTRFS info (device dm-1): disk space caching is enabled [ 33.283588] systemd-shutdown[1]: Unmounting /var/log. [ 33.295335] systemd-shutdown[1]: Remounting '/' read-only with options 'space_cache,subvolid=257,subvol=/@'. [ 33.296214] BTRFS info (device dm-1): disk space caching is enabled [ 33.296947] systemd-shutdown[1]: Remounting '/' read-only with options 'space_cache,subvolid=257,subvol=/@'. [ 33.298869] BTRFS info (device dm-1): disk space caching is enabled [ 33.299555] systemd-shutdown[1]: Remounting '/' read-only with options 'space_cache,subvolid=257,subvol=/@'. [ 33.300564] BTRFS info (device dm-1): disk space caching is enabled [ 33.301212] systemd-shutdown[1]: All filesystems unmounted. [ 33.301785] systemd-shutdown[1]: Deactivating swaps. [ 33.302337] systemd-shutdown[1]: All swaps deactivated. [ 33.302877] systemd-shutdown[1]: Detaching loop devices. [ 33.303530] systemd-shutdown[1]: All loop devices detached. [ 33.304111] systemd-shutdown[1]: Detaching DM devices. [ 33.304923] systemd-shutdown[1]: Detaching DM 254:1. [ 33.305463] systemd-shutdown[1]: Could not detach DM /dev/dm-1: Device or resource busy [ 33.306293] systemd-shutdown[1]: Detaching DM 254:0. [ 33.327328] systemd-shutdown[1]: Not all DM devices detached, 1 left. [ 33.328001] systemd-shutdown[1]: Detaching DM devices. [ 33.328643] systemd-shutdown[1]: Detaching DM 254:1. [ 33.329073] systemd-shutdown[1]: Could not detach DM /dev/dm-1: Device or resource busy [ 33.329733] systemd-shutdown[1]: Not all DM devices detached, 1 left. [ 33.330289] systemd-shutdown[1]: Cannot finalize remaining DM devices, continuing. [ 33.334868] systemd-shutdown[1]: Successfully changed into root pivot. [ 33.335494] systemd-shutdown[1]: Returning to initrd... [ 33.364999] dracut Warning: Killing all remaining processes dracut Warning: Killing all remaining processes [ 33.423657] dracut Warning: Unmounted /oldroot. [ 33.436812] dracut: Disassembling device-mapper devices Failed to read reboot parameter file: No such file or directory Rebooting. [ 33.468035] Unregister pv shared memory for cpu 0 [ 33.468670] sd 6:0:0:0: [sda] Synchronizing SCSI cache [ 33.470750] reboot: Restarting system [ 33.471122] reboot: machine restart -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
09.07.2017 10:26, Andrei Borzenkov пишет:
08.07.2017 19:17, Achim Gratz пишет:
Since a few weeks dracut spews about two pages of errors/warnings on shutdown. Yesterday I halted the system instead of poweroff to be able to read those and it seems that the unmount of /oldroot fails and then the disassembly of the device-mapper devices errors out (maybe there'd been other errors before that that already rolled off the screeb). Does anybody have an idea what is going on and how to fix it?
Sorry, but you will need to debug it. If unmounting of /oldroot fails, some process was stuck and not killed. device-mapper are just followup errors, as LVM cannot be stopped at this point. Any debugging will involve some way to capture late console output.
What you can do is to boot with rd.break=shutdown. This will stop during shutdown after it tried to unmount root but immediately before it would do reboot/poweroff. This will let you poke around what is left - which mount points, which processes etc. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Achim Gratz writes:
Since a few weeks dracut spews about two pages of errors/warnings on shutdown. Yesterday I halted the system instead of poweroff to be able to read those and it seems that the unmount of /oldroot fails and then the disassembly of the device-mapper devices errors out (maybe there'd been other errors before that that already rolled off the screeb). Does anybody have an idea what is going on and how to fix it?
I've now filed a bug for this: https://bugzilla.opensuse.org/show_bug.cgi?id=1080485 Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (3)
-
Achim Gratz -
Andrei Borzenkov -
Carlos E. R.