Tumbleweed: how to recover from a botched zypper dup
I have an old AthlonXP system that I decomissioned three years ago. It turned out later that just an SATA cable needed replacement. Today I tried to update to the latest Tumbleweed using zypper dup, which initially went well (if slowly), but then it hit the installation of filesystem, complained about missing lua triggers and admonished me that I would need a rescue system. Nothing in /usr/bin works anymore, although the (old) kernel boots until the first time it needs a shell for something. I do have a rescue system on USB from that time (actually a net install for Tumbleweed that I used to install the replacement system with), but it runs into trouble: the available shell prompt does not seem to offer a chroot command nor zypper and clicking past the language prompt will blink caps and scroll lock on the keyboard for about a minute and then reboots (perhaps it tried to mount the botched filesystem?). I'm not even sure a chroot would work since the filesystem isn't set up… Preferrably I could somehow use zypper (or rpm) to use the already downloaded package and target the failed installation (with --root /mnt or something like that). Is there a way to do that? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves
Achim Gratz composed on 2022-08-26 21:31 (UTC+0200):
I have an old AthlonXP system that I decomissioned three years ago.
For openSUSE purposes, it must stay that way. That CPU doesn't support at least one required instruction provided by newer CPUs and required by some base & optional packages. Debian 11 will run on it. I don't know the status of Debian Testing (12 to be). AntiX is often recommended for Athlons. According to https://get.opensuse.org/tumbleweed/, which IMO is bogus, 2G RAM and 2 core CPU are required. I have it running on several Pentium IVs, and with as little as 1G RAM. -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata
Felix Miata writes:
Achim Gratz composed on 2022-08-26 21:31 (UTC+0200):
I have an old AthlonXP system that I decomissioned three years ago.
For openSUSE purposes, it must stay that way. That CPU doesn't support at least one required instruction provided by newer CPUs and required by some base & optional packages. Debian 11 will run on it. I don't know the status of Debian Testing (12 to be). AntiX is often recommended for Athlons.
I misremembered what it was called, it's actually an Athlon64 (I think a Venice @2.2GHz, rated 3400+). Anyway, it should still be running in Tumbleweed x86_64 I think, but I'm not sure.
According to https://get.opensuse.org/tumbleweed/, which IMO is bogus, 2G RAM and 2 core CPU are required. I have it running on several Pentium IVs, and with as little as 1G RAM.
The system has the max. possible 3G RAM installed, it worked OK with just 2GiB, but then the browser would often trigger swapping. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada
On 26.08.2022 22:31, Achim Gratz wrote:
I have an old AthlonXP system that I decomissioned three years ago. It turned out later that just an SATA cable needed replacement. Today I tried to update to the latest Tumbleweed using zypper dup, which initially went well (if slowly), but then it hit the installation of filesystem, complained about missing lua triggers and admonished me that I would need a rescue system. Nothing in /usr/bin works anymore, although the (old) kernel boots until the first time it needs a shell for something.
I do have a rescue system on USB from that time (actually a net install for Tumbleweed that I used to install the replacement system with), but it runs into trouble: the available shell prompt does not seem to offer a chroot command nor zypper
If "available" means "whatever is available on Net image" that is expected - Net image only contains enough to download the actual installation system including rescue boot.
and clicking past the language prompt will blink caps and scroll lock on the keyboard for about a minute and then reboots (perhaps it tried to mount the botched filesystem?). I'm not
That sounds like kernel panic. You do not really need openSUSE medium, you can use anything (Knoppix, SystemRescueCD, ...) to access Tumbleweed root. Or use full openSUSE DVD.
even sure a chroot would work since the filesystem isn't set up… Preferrably I could somehow use zypper (or rpm) to use the already downloaded package and target the failed installation (with --root /mnt or something like that). Is there a way to do that?
Regards, Achim.
Andrei Borzenkov writes:
That sounds like kernel panic.
Yes, specifically it seems to happen when trying to switch to graphical mode.
You do not really need openSUSE medium, you can use anything (Knoppix, SystemRescueCD, ...) to access Tumbleweed root. Or use full openSUSE DVD.
I can mount the botched root from the netboot just fine (before it hangs up), what I was hoping for was a way to repair it from the outside. I don't think the chroot thing is a possibility in the current state (I've done that before to recover missing kernel and such stuff, but then the system partition was OK). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf Blofeld V1.15B11: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
ASSI writes:
Andrei Borzenkov writes:
That sounds like kernel panic.
Yes, specifically it seems to happen when trying to switch to graphical mode.
…or when it tries to load something off the USB medium. Anyway, I've prepared a stick with the full DVD image of yestaerday and it clears that hurdle and leaves me at the rescue prompt with all the tools available.
You do not really need openSUSE medium, you can use anything (Knoppix, SystemRescueCD, ...) to access Tumbleweed root. Or use full openSUSE DVD.
I can mount the botched root from the netboot just fine (before it hangs up), what I was hoping for was a way to repair it from the outside. I don't think the chroot thing is a possibility in the current state (I've done that before to recover missing kernel and such stuff, but then the system partition was OK).
However as suspected, the chroot just doesn't work for the same reason the system is inoperable after boot. So I'd still need to repair the damage to the installation from the outside or else do a full reinstall. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves
On 27.08.2022 12:21, Achim Gratz wrote:
However as suspected, the chroot just doesn't work for the same reason the system is inoperable after boot. So I'd still need to repair the damage to the installation from the outside or else do a full reinstall.
filesystem package tries to move the content of /bin, /sbin, /lib and /lib64 into corresponding /usr subdirectory and replace these directories with links. It runs script /usr/libexec/convertfs that should be present on your root. You can edit this script, replace ROOT= at the beginning with ROOT=/your-root-mount-point and try to run it (you do not need to chroot). Or you could simply try "chroot /usr/libexec/convertfs", but it depends on /bin/bash being available in your root. You can check the current state of the directories above (and their /usr counterparts).
Andrei Borzenkov writes:
filesystem package tries to move the content of /bin, /sbin, /lib and /lib64 into corresponding /usr subdirectory and replace these directories with links.
Yes, I meanwhile figured out that this might be a related to the UsrMerge story, still digging into all the bug reports.
It runs script /usr/libexec/convertfs that should be present on your root. You can edit this script, replace ROOT= at the beginning with ROOT=/your-root-mount-point and try to run it (you do not need to chroot). Or you could simply try "chroot /usr/libexec/convertfs", but it depends on /bin/bash being available in your root.
No, bash is already unstartable. So do I read you correctly that if I fix this up manually then I should be able to run the script and regain the ability to chroot into the root fs so I can run the zypper dup to completion?
You can check the current state of the directories above (and their /usr counterparts).
Later… Well, now that I knew that I had to look for UsrMerge, I've found this page: https://en.opensuse.org/openSUSE:Usr_merge, especially the FAQ there. Lo and behold just adding the two links made the chroot (or rather the bash it's trying to start) work again. I couldn't get the network up in the rescue system and I didn't manage to get zypper to work just with the already downloaded information in the caches, so I tried booting, which worked. A lot of things didn't come up correctly, but it got to the root prompt and I did the convertfs. Another reboot later and after some manual fixes (DNS wasn't working and somehow the ethernet interface had an rp_filter active on IPv4, SSH ingress still gets filtered out) I've got zypper to re-install the filesystem package. I'm currently letting it to update the remaining ~7000 packages it didn't get to yesterday. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada
Achim Gratz writes:
I've got zypper to re-install the filesystem package. I'm currently letting it to update the remaining ~7000 packages it didn't get to yesterday.
Some 2000 packages into that exercise it hit the final speedbump: rpm got upgraded and wanted to change the database format to ndb, which didn't work while in zypper. I could rebuild the rpmdb, but now zypper doesn't start anymore since it wants an older librpm than is now installed. Presumably I could install a new zypper version via rpm if I could figure out the whole dependency chain, but rpm only seems to tell me the next link when I try. Plus having to specify the full package file names out of the cache dir kind of sucks. Now I'd really like a way of running zypper from the install DVD image targeted at the inactive system root, but it seems zypper is not available there. Absent that I'd need to install a fresh system on a separate disk (which at the moment still has an openSUSE 10.2 on it, LOL) and then try to revive the original system… maybe. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf microQ V2.22R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
On 2022-08-27 21:19, Achim Gratz wrote:
Achim Gratz writes:
I've got zypper to re-install the filesystem package. I'm currently letting it to update the remaining ~7000 packages it didn't get to yesterday.
Some 2000 packages into that exercise it hit the final speedbump: rpm got upgraded and wanted to change the database format to ndb, which didn't work while in zypper. I could rebuild the rpmdb, but now zypper doesn't start anymore since it wants an older librpm than is now installed. Presumably I could install a new zypper version via rpm if I could figure out the whole dependency chain, but rpm only seems to tell me the next link when I try. Plus having to specify the full package file names out of the cache dir kind of sucks. Now I'd really like a way of running zypper from the install DVD image targeted at the inactive system root, but it seems zypper is not available there. Absent that I'd need to install a fresh system on a separate disk (which at the moment still has an openSUSE 10.2 on it, LOL) and then try to revive the original system… maybe.
That can be achieved by booting the DVD (freshly downloaded) and choosing "upgrade" instead of install. -- Cheers / Saludos, Carlos E. R. (from 15.3 x86_64 at Telcontar)
Hello, Am Samstag, 27. August 2022, 21:19:53 CEST schrieb Achim Gratz:
Some 2000 packages into that exercise it hit the final speedbump: rpm got upgraded and wanted to change the database format to ndb, which didn't work while in zypper. I could rebuild the rpmdb, but now zypper doesn't start anymore since it wants an older librpm than is now installed. Presumably I could install a new zypper version via rpm if I could figure out the whole dependency chain, but rpm only seems to tell me the next link when I try. Plus having to specify the full package file names out of the cache dir kind of sucks.
Just wondering - did you try to let rpm upgrade all packages at once? rpm -Uhv $cachedir/repo-oss/*/*.rpm In theory, this should work ;-) and is probably easier than manually solving the dependencies.
[...] Absent that I'd need to install a fresh system on a separate disk (which at the moment still has an openSUSE 10.2 on it,
Sounds like you could do some archeology and look back a bit - were the "good old times" really that good, or is current Tumbleweed better? ;-) Regards, Christian Boltz -- "Wouldn't the sentence 'I want to put a hyphen between the words Fish and And and And and Chips in my Fish-And-Chips sign' have been clearer if quotation marks had been placed before Fish, and between Fish and and, and and and And, and And and and, and and and And, and And and and, and and and Chips, as well as after Chips?" -- BSD fortune file
On 2022-08-28 00:07, Christian Boltz wrote:
Hello,
Am Samstag, 27. August 2022, 21:19:53 CEST schrieb Achim Gratz:
Some 2000 packages into that exercise it hit the final speedbump: rpm got upgraded and wanted to change the database format to ndb, which didn't work while in zypper. I could rebuild the rpmdb, but now zypper doesn't start anymore since it wants an older librpm than is now installed. Presumably I could install a new zypper version via rpm if I could figure out the whole dependency chain, but rpm only seems to tell me the next link when I try. Plus having to specify the full package file names out of the cache dir kind of sucks.
Just wondering - did you try to let rpm upgrade all packages at once?
rpm -Uhv $cachedir/repo-oss/*/*.rpm
In theory, this should work ;-) and is probably easier than manually solving the dependencies.
A friend of mine told me this summer that we computer people use a lot the word "should", so that the thing became a running joke with us ;-) I can imagine many ways in what that command would nor work or fail in unexpected manners. In my machine, for instance, it would find current and old rpms, and several versions of packages. -- Cheers / Saludos, Carlos E. R. (from 15.3 x86_64 at Telcontar)
Achim Gratz writes:
Now I'd really like a way of running zypper from the install DVD image targeted at the inactive system root, but it seems zypper is not available there.
I ended up putting the snapshot DVD on a new USB stick and doing an "Update" from there. It hit two more stops because some assumptions about how the file system should look were no longer true, but these could be ignored and the install finished after a few hours. The next reboot ended up at the rescue prompt because no default target had been set up. Fixing that and booting into graphical target triggered extremely high and prolonged disk activity that didn't really subside. It turns out more systemd units were set up wrongly, network and sshd were not starting correctly and syslog-ng was conflicting with journald. It was also immediately clear that there was a problem with the RV350 graphics card in that system, since the login screen from sddm did not appear until a bit of switching back and forth between VT1 and VT7, then I've had to stop. I took to the system again yesterday, and Tumbleweed presented me with another full update of roughly 7500 packages, which consumed most of the day. When I was finally logging into my KDE/Plasma/X11 desktop plasmashell slowly exhausted the memory and then also the swap and wedged the system pretty hard. I've let it continue overnight to see if it would finish whatever it was trying to do, but no dice (except thoroughly stressing the HD of course, which did survive). After the hard reset and reboot I switched off swap and logged into the KDE session again. Plasmashell again started to consume memory at a relatively slow, but constant rate, so I eventually killed it when it hit max. memory. While I did lose the desktop features, the window manager kept running and the memory consumption was back to its previous (before the update) value of ~1.2GiB, I could even start Firefox. Looking into the logs it seems that the behaviour from Plasma is due to some regression / change in Mesa and or the way OpenGL is configured: --8<---------------cut here---------------start------------->8--- plasmashell[2330]: MESA: warning: r300: WARNING: Shader is trying to use derivatives, but the hardware doesn't support it. Expect possible misrendering (it's not a bug, do not report it). plasmashell[2330]: r300 FP: Compiler Error: plasmashell[2330]: Too many constants. Max: 32, Got: 262 plasmashell[2330]: Using a dummy shader instead. plasmashell[2330]: shader compilation failed: plasmashell[2330]: QOpenGLShader::link: error: fragment shader uses too many input components (92 > 40) plasmashell[2330]: QOpenGLShaderProgram::uniformLocation(matrix): shader program is not linked plasmashell[2330]: QOpenGLShaderProgram::uniformLocation(opacity): shader program is not linked plasmashell[2330]: QOpenGLShaderProgram::uniformLocation(lineWidth): shader program is not linked plasmashell[2330]: QOpenGLShaderProgram::uniformLocation(aspect): shader program is not linked plasmashell[2330]: QOpenGLShaderProgram::uniformLocation(smoothing): shader program is not linked --8<---------------cut here---------------end--------------->8--- Firefox curiously complained about some environment override of VDPAU settings. I do indeed have "VDPAU_DRIVER=va_gl" in my environment, although I do not know yet where it comes from. I dimly remember that VDPAU was a problem in the past, but not how and where I resolved it. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ DIY Stuff: http://Synth.Stromeko.net/DIY.html
On 2022-09-11 10:27, Achim Gratz wrote:
Achim Gratz writes:
Now I'd really like a way of running zypper from the install DVD image targeted at the inactive system root, but it seems zypper is not available there.
I ended up putting the snapshot DVD on a new USB stick and doing an "Update" from there. It hit two more stops because some assumptions about how the file system should look were no longer true, but these could be ignored and the install finished after a few hours.
The next reboot ended up at the rescue prompt because no default target had been set up. Fixing that and booting into graphical target triggered extremely high and prolonged disk activity that didn't really subside. It turns out more systemd units were set up wrongly, network and sshd were not starting correctly and syslog-ng was conflicting with journald.
syslog-ng seems to be a leftover from long ago. I think you should have syslog instead. Telcontar:~ # systemctl status syslog-ng.service Unit syslog-ng.service could not be found. Telcontar:~ # locate syslog-ng.service Telcontar:~ # Telcontar:~ # systemctl status syslog ● rsyslog.service - System Logging Service Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2022-09-09 18:21:21 CEST; 1 day 17h ago TriggeredBy: ● syslog.socket Docs: man:rsyslogd(8) http://www.rsyslog.com/doc/ Process: 2302 ExecStartPre=/usr/sbin/rsyslog-service-prepare (code=exited, status=0/SUCCESS) Process: 11327 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS) Main PID: 2350 (rsyslogd) Tasks: 6 (limit: 4915) CGroup: /system.slice/rsyslog.service └─2350 /usr/sbin/rsyslogd -n -iNONE ...
It was also immediately clear that there was a problem with the RV350 graphics card in that system, since the login screen from sddm did not appear until a bit of switching back and forth between VT1 and VT7, then I've had to stop.
I took to the system again yesterday, and Tumbleweed presented me with another full update of roughly 7500 packages, which consumed most of the day.
Maybe during the DVD upgrade the network repositories were not enabled?
When I was finally logging into my KDE/Plasma/X11 desktop plasmashell slowly exhausted the memory and then also the swap and wedged the system pretty hard. I've let it continue overnight to see if it would finish whatever it was trying to do, but no dice (except thoroughly stressing the HD of course, which did survive). After the hard reset and reboot I switched off swap and logged into the KDE session again. Plasmashell again started to consume memory at a relatively slow, but constant rate, so I eventually killed it when it hit max. memory. While I did lose the desktop features, the window manager kept running and the memory consumption was back to its previous (before the update) value of ~1.2GiB, I could even start Firefox.
Disabling swap in this status is not correct. The idea is that when memory runs away and hits swap you have time to kill some process. Without swap the time limit is shorter, and the kernel may kill processes on its own, not necessarily choosing the correct one. -- Cheers / Saludos, Carlos E. R. (from 15.3 x86_64 at Telcontar)
On 26.08.2022 22:31, Achim Gratz wrote:
I have an old AthlonXP system that I decomissioned three years ago. It turned out later that just an SATA cable needed replacement. Today I tried to update to the latest Tumbleweed using zypper dup, which initially went well (if slowly), but then it hit the installation of filesystem, complained about missing lua triggers and admonished me that I would need a rescue system.
It is rather hard to make any guess based in such precise information, but the only message matching this description comes when a) filesystem needs to perform usrmerge (meaning, /bin etc are directories and not links into /usr) b) ZYPP_SINGLE_RPMTRANS environment variable is set which as far as I can tell still is not default and needs to be set explicitly You did not say from which Tumbleweed snapshot you updated. At the very least it should have been per-usrmerge. I cannot reproduce it updating from relatively recent snapshot.
Andrei Borzenkov writes:
a) filesystem needs to perform usrmerge (meaning, /bin etc are directories and not links into /usr)
Yes.
b) ZYPP_SINGLE_RPMTRANS environment variable is set which as far as I can tell still is not default and needs to be set explicitly
Also yes (out of habit), although the zypper currently running there should not even know about this option, predating its announcement by roughly two years.
You did not say from which Tumbleweed snapshot you updated.
It was 201909xy… Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables
On 27.08.2022 18:45, Achim Gratz wrote:
Andrei Borzenkov writes:
a) filesystem needs to perform usrmerge (meaning, /bin etc are directories and not links into /usr)
Yes.
b) ZYPP_SINGLE_RPMTRANS environment variable is set which as far as I can tell still is not default and needs to be set explicitly
Also yes (out of habit), although the zypper currently running there should not even know about this option, predating its announcement by roughly two years.
Which is irrelevant because this variable is checked in filesystem package scripts. And yes, in this case different utility is needed. The problem is that RPM cannot have dependency conditional on environment variable. Self inflicted wound ... ... still I guess bug report is in order to at least document this. May be some words in release notes/wiki.
You did not say from which Tumbleweed snapshot you updated.
It was 201909xy…
Regards, Achim.
participants (6)
-
Achim Gratz -
Andrei Borzenkov -
ASSI
-
Carlos E. R. -
Christian Boltz -
Felix Miata