[Bug 1194199] New: zypper dup on multiple HPE servers freezes, resulting system will not start
http://bugzilla.opensuse.org/show_bug.cgi?id=1194199 Bug ID: 1194199 Summary: zypper dup on multiple HPE servers freezes, resulting system will not start Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: 64bit OS: openSUSE Tumbleweed Status: NEW Severity: Major Priority: P5 - None Component: Upgrade Problems Assignee: screening-team-bugs@suse.de Reporter: bob@muhlenberg.edu QA Contact: jsrain@suse.com Found By: --- Blocker: --- Created attachment 854887 --> http://bugzilla.opensuse.org/attachment.cgi?id=854887&action=edit Rapidly scrolling errors after startup, post fozen zypper dup On two different HPE DL-380 and DL-360 ( Gen 8 and Gen 9 ) servers I performed zypper dup and after installing perhaps half of the packages, the zypper dup stalls and the system becomes unresponsive. The system looses network connectivity and appears to be frozen. After waiting a long time ( a day ) we rebooted the system, on booting the system boots through GRUB and after a few seconds haangs ( usually at a green "three dots" screen ). On booting via Advanced Options --> Recovery It is observed various dracut errors a couple seconds into startup, followed by a continuous stream of errors about 10 a second: dracut-initqueue[PID]: udevadm: error while loading shared libraries: libsystemd-shared-246.so cannot open shared object file: no such file or directory This scrolls continuously with a new PID on every line. This never ends and the system is non-responsive. I am going to attempt to do an in place upgrade with the latest DVD ISO. In both instances the prior release installed was old ( 2020 or early 2021 vintage? ). If this is a known issue is there a way to recoup the system? If not, what do you need, log wise? Given this is two different physical systems and two different prior releases, it may be a serious issue. The first time I saw this was about 2 weeks ago. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1194199 http://bugzilla.opensuse.org/show_bug.cgi?id=1194199#c1 Andreas Stieger <Andreas.Stieger@gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dracut-maintainers@suse.de Assignee|screening-team-bugs@suse.de |dracut-maintainers@suse.de --- Comment #1 from Andreas Stieger <Andreas.Stieger@gmx.de> --- (In reply to Robert Mahar from comment #0)
dracut-initqueue[PID]: udevadm: error while loading shared libraries: libsystemd-shared-246.so cannot open shared object file: no such file or directory [...] In both instances the prior release installed was old ( 2020 or early 2021 vintage? ).
Looks like the system was in an intermediate state that prevented the dracut scripts from running successfully. Dracut maintainers, can anything be done here? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1194199 http://bugzilla.opensuse.org/show_bug.cgi?id=1194199#c2 Andreas Stieger <Andreas.Stieger@gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Andreas.Stieger@gmx.de --- Comment #2 from Andreas Stieger <Andreas.Stieger@gmx.de> --- (In reply to Robert Mahar from comment #0)
is there a way to recoup the system
Boot into a rescue system and operate with zypper -R / --root -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1194199 http://bugzilla.opensuse.org/show_bug.cgi?id=1194199#c3 --- Comment #3 from Robert Mahar <bob@muhlenberg.edu> --- I will try the zypper -R ... suggestion on one of the systems once physically back at work. Performing an in-place upgrade with the 20211228 Tumbleweed ISO DVD media fixed the issue on another one of the broken systems. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1194199 http://bugzilla.opensuse.org/show_bug.cgi?id=1194199#c4 --- Comment #4 from Robert Mahar <bob@muhlenberg.edu> --- I am not up to 4 systems, including two laptops, where zypper dup leaves the system unusable. One was last updated on Dec 1. 2021 and has been otherwise working. When I issued zypper dup, upgrade proceeded normally until numerous dracut-intall errors were displayed ( it took about 5 minutes for this to stop ) and then soon after the update was aborted. The dracut-install error indicated that it could not copy a file. The cp command failed. However the cp binary in /usr/bin/ had been overwritten by a link to busybox. On looking at other binaries in /usr/bin/ many had been replaced by busybox. Busybox has bot previously been installed on this system. - - - - bob@BOBSTOP:~> ls -lia ls: invalid option -- 'N' BusyBox v1.34.1 () multi-call binary. Usage: ls [-1AaCxdLHRFplinshrSXvctukZ] [-w WIDTH] [FILE]... - - - - bob@BOBSTOP:~> cp -v BusyBox v1.34.1 () multi-call binary. Usage: cp [-arPLHpfinlsTu] SOURCE DEST or: cp [-arPLHpfinlsu] SOURCE... { -t DIRECTORY | DIRECTORY } - - - - from the zypper dup screen output 2022-01-02 21:09:02 <1> BOBSTOP(30241) [zypp::posttrans++] RpmPostTransCollector.cc(executeScripts):140 dracut-install: dracut_install '/lib/modules/5.14.14-1-default//kernel/crypto/sm3_generic.ko.xz' '/lib/modules/5.14.14-1-default//kernel/crypto/sm3_generic.ko.xz' ERROR 2022-01-02 21:09:02 <1> BOBSTOP(30241) [zypp::posttrans++] RpmPostTransCollector.cc(executeScripts):140 cp: unrecognized option '--sparse=auto' 2022-01-02 21:09:02 <1> BOBSTOP(30241) [zypp::posttrans++] RpmPostTransCollector.cc(executeScripts):140 BusyBox v1.34.1 () multi-call binary. 2022-01-02 21:09:02 <1> BOBSTOP(30241) [zypp::posttrans++] RpmPostTransCollector.cc(executeScripts):140 2022-01-02 21:09:02 <1> BOBSTOP(30241) [zypp::posttrans++] RpmPostTransCollector.cc(executeScripts):140 Usage: cp [-arPLHpfinlsTu] SOURCE DEST 2022-01-02 21:09:02 <1> BOBSTOP(30241) [zypp::posttrans++] RpmPostTransCollector.cc(executeScripts):140 or: cp [-arPLHpfinlsu] SOURCE... { -t DIRECTORY | DIRECTORY } So basically busybox somehow overwrites /usr/bin on these systems and then everything after that is a total disaster. I am attaching zypper logs for one system. I checked the other system I recouped by doing a upgrade from DVD medium, and the snapshots show a similar pattern where busybox supplanted the normal binaries. I suspect that when using the DVD upgrade, the copies of the binaries available to the installer are those from the DVD ( so not busybox ) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1194199 http://bugzilla.opensuse.org/show_bug.cgi?id=1194199#c5 --- Comment #5 from Robert Mahar <bob@muhlenberg.edu> --- Created attachment 854893 --> http://bugzilla.opensuse.org/attachment.cgi?id=854893&action=edit System 1 - zypper log logs showing dracut-install failing with invalid parameter error in cp command and also that busybox cp is being used, etc. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1194199 http://bugzilla.opensuse.org/show_bug.cgi?id=1194199#c6 --- Comment #6 from Robert Mahar <bob@muhlenberg.edu> --- Created attachment 854894 --> http://bugzilla.opensuse.org/attachment.cgi?id=854894&action=edit System 1 - zypper log 2 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1194199 http://bugzilla.opensuse.org/show_bug.cgi?id=1194199#c7 --- Comment #7 from Robert Mahar <bob@muhlenberg.edu> --- Created attachment 854895 --> http://bugzilla.opensuse.org/attachment.cgi?id=854895&action=edit System 1 - zypper log 3 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1194199 http://bugzilla.opensuse.org/show_bug.cgi?id=1194199#c8 --- Comment #8 from Robert Mahar <bob@muhlenberg.edu> --- I was able to manually remove busybox ( which was not installed per rpm -qa or rpm -qf /usr/bin/____ ( insert name here ) but forcing re-install of coreutils procps psmisc with zypper in -f pkg_name then confirming busybox links were no longer present with find / -L -samefile /usr/bin/busybox > /tmp/busy.lst or via rpm -Va -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com