[kernel][TW] zypper dup to 20220714 + rpm -i kernel-default-5.18.9-2 x86_64 removed all other kernels
Is $SUBJECT what was supposed to happen? Prior TW version was 20220630. Newest kernel installed prior to 5.18.9-2 was 5.17.9-1. June/July portion of /var/log/zypp/history: https://paste.opensuse.org/74203728 -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata
Felix Miata composed on 2022-07-15 19:20 (UTC-0400):
Is $SUBJECT what was supposed to happen?
Prior TW version was 20220630. Newest kernel installed prior to 5.18.9-2 was 5.17.9-1.
June/July portion of /var/log/zypp/history: https://paste.opensuse.org/74203728
Just happened again on dup from 20220613 to 20220714. Latest previous kernel was 5.18.2. :( -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata
Hi Felix, On Mon, 18 Jul 2022, 07:18:52 +0200, Felix Miata wrote:
Felix Miata composed on 2022-07-15 19:20 (UTC-0400):
Is $SUBJECT what was supposed to happen?
Prior TW version was 20220630. Newest kernel installed prior to 5.18.9-2 was 5.17.9-1.
June/July portion of /var/log/zypp/history: https://paste.opensuse.org/74203728
Just happened again on dup from 20220613 to 20220714. Latest previous kernel was 5.18.2. :(
no problems here, all latest dups left all kernels installed. What does /etc/zypp/zypp.conf contain for multiversion.kernels? This is what I have here: # grep multi /etc/zypp/zypp.conf | egrep -v '^#' multiversion = provides:multiversion(kernel) multiversion.kernels = latest,latest-1,running,oldest HTH, cheers. l8er manfred
Manfred Hollstein composed on 2022-07-18 01:56 (UTC-0400):
Felix Miata wrote:
Felix Miata composed on 2022-07-15 19:20 (UTC-0400):
Is $SUBJECT what was supposed to happen?
Prior TW version was 20220630. Newest kernel installed prior to 5.18.9-2 was 5.17.9-1.
June/July portion of /var/log/zypp/history: https://paste.opensuse.org/74203728
Just happened again on dup from 20220613 to 20220714. Latest previous kernel was 5.18.2. :(
no problems here, all latest dups left all kernels installed. What does /etc/zypp/zypp.conf contain for multiversion.kernels? This is what I have here:
# grep multi /etc/zypp/zypp.conf | egrep -v '^#' multiversion = provides:multiversion(kernel) multiversion.kernels = latest,latest-1,running,oldest
Same here. I forgot to mention, because I keep kernels locked, none added, none removed, unless and until I choose, this is/was a particularly huge surprise: # grep -B3 kernel /etc/zypp/locks type: package match_type: glob case_sensitive: on solvable_name: kernel-de* -- type: package match_type: glob case_sensitive: on solvable_name: kernel-firmwar? I know rpm doesn't pay any attention to zypp locks, but I don't recall ever seeing this kind of burn, leaving no kernel to fall back on if the fresh one fails to boot. I also forgot to mention, I installed 5.18.9-2 using rpm -ivh, which didn't mention anything about any plan to remove anything that caught my attention. :( -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata
On Mon, 18 Jul 2022, 08:28:31 +0200, Felix Miata wrote:
Manfred Hollstein composed on 2022-07-18 01:56 (UTC-0400):
Felix Miata wrote:
Felix Miata composed on 2022-07-15 19:20 (UTC-0400):
Is $SUBJECT what was supposed to happen?
Prior TW version was 20220630. Newest kernel installed prior to 5.18.9-2 was 5.17.9-1.
June/July portion of /var/log/zypp/history: https://paste.opensuse.org/74203728
Just happened again on dup from 20220613 to 20220714. Latest previous kernel was 5.18.2. :(
no problems here, all latest dups left all kernels installed. What does /etc/zypp/zypp.conf contain for multiversion.kernels? This is what I have here:
# grep multi /etc/zypp/zypp.conf | egrep -v '^#' multiversion = provides:multiversion(kernel) multiversion.kernels = latest,latest-1,running,oldest
Same here.
I forgot to mention, because I keep kernels locked, none added, none removed, unless and until I choose, this is/was a particularly huge surprise: # grep -B3 kernel /etc/zypp/locks type: package match_type: glob case_sensitive: on solvable_name: kernel-de* -- type: package match_type: glob case_sensitive: on solvable_name: kernel-firmwar?
I know rpm doesn't pay any attention to zypp locks, but I don't recall ever seeing this kind of burn, leaving no kernel to fall back on if the fresh one fails to boot.
I do remember such a problem - it was caused by "zypper purge-kernels" which happily removed all other kernels than the current one. This is why I disabled purge-kernels.service and also always remove the file /boot/do_purge_kernels after every "zypper dup". Don't know if this is actually still necessary, but could explain what you are seeing now.
I also forgot to mention, I installed 5.18.9-2 using rpm -ivh, which didn't mention anything about any plan to remove anything that caught my attention. :(
I don't have any locks, but I protect the kernels differently as described above. Cheers. l8er manfred
Manfred Hollstein composed on 2022-07-18 08:42 (UTC+0200):
Felix Miata wrote:
Manfred Hollstein composed on 2022-07-18 07:56 (UTC+0200):
Felix Miata wrote:
Felix Miata composed on 2022-07-15 19:20 (UTC-0400):
Is $SUBJECT what was supposed to happen?
Prior TW version was 20220630. Newest kernel installed prior to 5.18.9-2 was 5.17.9-1.
June/July portion of /var/log/zypp/history: https://paste.opensuse.org/74203728
Just happened again on dup from 20220613 to 20220714. Latest previous kernel was 5.18.2. :(
no problems here, all latest dups left all kernels installed. What does /etc/zypp/zypp.conf contain for multiversion.kernels? This is what I have here:
# grep multi /etc/zypp/zypp.conf | egrep -v '^#' multiversion = provides:multiversion(kernel) multiversion.kernels = latest,latest-1,running,oldest
Same here.
I forgot to mention, because I keep kernels locked, none added, none removed, unless and until I choose, this is/was a particularly huge surprise: # grep -B3 kernel /etc/zypp/locks type: package match_type: glob case_sensitive: on solvable_name: kernel-de* -- type: package match_type: glob case_sensitive: on solvable_name: kernel-firmwar?
I know rpm doesn't pay any attention to zypp locks, but I don't recall ever seeing this kind of burn, leaving no kernel to fall back on if the fresh one fails to boot.
I do remember such a problem - it was caused by "zypper purge-kernels" which happily removed all other kernels than the current one. This is why I disabled purge-kernels.service and also always remove the file /boot/do_purge_kernels after every "zypper dup". Don't know if this is actually still necessary, but could explain what you are seeing now.
AFAIK, do_purge_kernels is run /after/ booting the newest kernel the first time. That's not what happened here. I too have purge-kernels.service disabled, and masked. I also remove /boot/do_purge_kernels every time I see it, which normally follows immediately after a new kernel installation completes, when I goto /boot/ to adjust the symlinks, all but one of which for kernels are now dead: # ls -Gg ini* lrwxrwxrwx 1 23 Jul 18 01:09 initrd -> initrd-5.18.9-2-default -rw------- 1 16564455 Jan 18 01:21 initrd-5.15.12-1-default -rw------- 1 15086540 May 17 03:23 initrd-5.16.15-1-default -rw------- 1 15306732 Jun 14 18:40 initrd-5.17.9-1-default -rw------- 1 16203164 Jul 18 01:10 initrd-5.18.9-2-default -rw------- 1 14107304 Sep 30 2020 initrd-5.7.11-1-default lrwxrwxrwx 1 23 Jul 18 01:09 initrd-cur -> initrd-5.18.9-2-default lrwxrwxrwx 1 23 Jun 14 18:40 initrd-prv -> initrd-5.17.9-1-default lrwxrwxrwx 1 24 May 17 03:22 initrd-prv2 -> initrd-5.16.15-1-default lrwxrwxrwx 1 24 Jan 18 01:20 initrd-prv3 -> initrd-5.15.12-1-default lrwxrwxrwx 1 23 Sep 30 2020 initrd-prv8 -> initrd-5.7.11-1-default The initrds are still there due to immutable flags I attach after each has proven bootable.
I also forgot to mention, I installed 5.18.9-2 using rpm -ivh, which didn't mention anything about any plan to remove anything that caught my attention. :(
I don't have any locks, but I protect the kernels differently as described above. -- Evolution as taught in public schools is, like religion, based on faith, not based on science.
Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata
On Mon 2022-07-18 02:28:31, Felix Miata wrote:
Manfred Hollstein composed on 2022-07-18 01:56 (UTC-0400):
Felix Miata wrote:
Felix Miata composed on 2022-07-15 19:20 (UTC-0400):
Is $SUBJECT what was supposed to happen?
Prior TW version was 20220630. Newest kernel installed prior to 5.18.9-2 was 5.17.9-1.
June/July portion of /var/log/zypp/history: https://paste.opensuse.org/74203728
Just happened again on dup from 20220613 to 20220714. Latest previous kernel was 5.18.2. :(
Do you know whether the older kernels were removed after "zypper dup" or after "rpm -i kernel-default-5.18.9-2 x86_64"? rpm -i should not remove any package. I guess that it was "zypper dup".
no problems here, all latest dups left all kernels installed. What does /etc/zypp/zypp.conf contain for multiversion.kernels? This is what I have here:
# grep multi /etc/zypp/zypp.conf | egrep -v '^#' multiversion = provides:multiversion(kernel) multiversion.kernels = latest,latest-1,running,oldest
Same here.
I forgot to mention, because I keep kernels locked, none added, none removed, unless and until I choose, this is/was a particularly huge surprise: # grep -B3 kernel /etc/zypp/locks type: package match_type: glob case_sensitive: on solvable_name: kernel-de* -- type: package match_type: glob case_sensitive: on solvable_name: kernel-firmwar?
I wonder if there is a bug in zypper and it ignored these locks. Does "zypper ll" still list the locks?
I know rpm doesn't pay any attention to zypp locks, but I don't recall ever seeing this kind of burn, leaving no kernel to fall back on if the fresh one fails to boot.
I also forgot to mention, I installed 5.18.9-2 using rpm -ivh, which didn't mention anything about any plan to remove anything that caught my attention. :(
Yes, this should not cause the problem. Well, I am not completely sure how zypper handle manually installed packages. I just hope that it does not make any difference. Best Regards, Petr
Hello, On Mon, Jul 18, 2022 at 01:18:52AM -0400, Felix Miata wrote:
Felix Miata composed on 2022-07-15 19:20 (UTC-0400):
Is $SUBJECT what was supposed to happen?
Prior TW version was 20220630. Newest kernel installed prior to 5.18.9-2 was 5.17.9-1.
June/July portion of /var/log/zypp/history: https://paste.opensuse.org/74203728
Just happened again on dup from 20220613 to 20220714. Latest previous kernel was 5.18.2. :(
This sounds like a zypper problem, and to analyze the problem the zypper maintainers would likely want a solver case: https://en.opensuse.org/SDB:Zypper_troubleshooting If you notice this right after an upgrade, and are using snapshots you can revert the snapshot and re-run the update, or you can try to generate a testcase for each update just in case. So far I have not seen this problem so there is probably something specific about your system that causes it. Note however that the posted zypper run does not do a kernel update. If the update is done by plain rpm it is not logged by zypper, and you need to do the logging (and snapshots) yourself. Thanks Michal
Jiri Slaby composed on 2022-07-18 10:42 (UTC+0200):
Michal Suchánek wrote:
If the update is done by plain rpm it is not logged by zypper, and you need to do the logging (and snapshots) yourself.
(Or install .rpm files using zypper. It can handle them.)
That's how I do Leap. With both Leap and TW I download kernels manually to my LAN server. But with Leap, I copy the kernels to the target's zypp cache before installing with zypper. With TW, I've been installing with rpm directly from the server, in part because of the non-scrollback kernels since 5.11, and another part the busier I/O with zypper, pausing for the kernel lock answer, and giving 3 choices instead of 2 when it announces the new kernel twice, once on the mirrors, and the other on the server, making the "remove" lock option #3, compared to #2 when using my Leap procedure. rpm doesn't even have to ask, since (thankfully) it knows nothing about zypp locks. I do what I do in part because I have >30 TW installations, and nearly as many of each supported Leap. I have a TW cache on the LAN server that I bind mount to each installation's zypp cache before duping. The zypp locks keep kernels out of that cache. I keep kernel rpms segregated from other cached rpms, and I control that cache differently. All my TWs are on EXT3 or EXT4. The only BTRFS snapshotting available here is on a laptop with Leap only. Everything else here is multi-, multiboot, between 10 and 60 partitions per disk. I repeat: the $SUBJECT behavior has occurred on two different installations, one Friday, one Sunday night. zypper dup excluded kernel on each, which is SOP here. Latest prior kernel was 5.17.9 on the Friday PC, 5.18.2 on the later. rpm -ivh caused the 5.18.9-2 installations that removed all the older kernels. Due to immutable flags, present due to absence of available configuration option to prevent automatic rebuilding of initrds proven to work, the "removed" initrds remain: # ls -1 /disks/stw/boot/initrd-5* /disks/stw/boot/initrd-5.15.12-1-default /disks/stw/boot/initrd-5.16.15-1-default /disks/stw/boot/initrd-5.17.9-1-default /disks/stw/boot/initrd-5.18.9-2-default /disks/stw/boot/initrd-5.7.11-1-default On the other PC on which this happened, without a similar freespace limitation, installed kernel versions went back to 5.4.14, which is the installed kernels situation most TW installations here still have. I can proceed with other installations in like manner if this thread encourages it, or adjust as may be suggested to try to pin down the blame. -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata
On Tue, Jul 19, 2022 at 04:33:35AM -0400, Felix Miata wrote:
Jiri Slaby composed on 2022-07-18 10:42 (UTC+0200):
Michal Suchánek wrote:
If the update is done by plain rpm it is not logged by zypper, and you need to do the logging (and snapshots) yourself.
(Or install .rpm files using zypper. It can handle them.)
That's how I do Leap. With both Leap and TW I download kernels manually to my LAN server. But with Leap, I copy the kernels to the target's zypp cache before installing with zypper. With TW, I've been installing with rpm directly from the server, in part because of the non-scrollback kernels since 5.11, and another part the busier I/O with zypper, pausing for the kernel lock answer, and giving 3 choices instead of 2 when it announces the new kernel twice, once on the mirrors, and the other on the server, making the "remove" lock option #3, compared to #2 when using my Leap procedure. rpm doesn't even have to ask, since (thankfully) it knows nothing about zypp locks.
I do what I do in part because I have >30 TW installations, and nearly as many of each supported Leap. I have a TW cache on the LAN server that I bind mount to each installation's zypp cache before duping. The zypp locks keep kernels out of that cache. I keep kernel rpms segregated from other cached rpms, and I control that cache differently.
All my TWs are on EXT3 or EXT4. The only BTRFS snapshotting available here is on a laptop with Leap only. Everything else here is multi-, multiboot, between 10 and 60 partitions per disk.
I repeat: the $SUBJECT behavior has occurred on two different installations, one Friday, one Sunday night. zypper dup excluded kernel on each, which is SOP here. Latest prior kernel was 5.17.9 on the Friday PC, 5.18.2 on the later. rpm -ivh caused the 5.18.9-2 installations that removed all the older kernels.
Was rpm updated recently causing random behavior change by any chance?
Due to immutable flags, present due to absence of available configuration option to prevent automatic rebuilding of initrds proven to work, the "removed" initrds remain:
If these are immutable they would remein indefinitely even if the kernels in question were removed much earlier. Thanks Michal
Michal Suchánek composed on 2022-07-19 10:49 (UTC+0200):
On Tue, Jul 19, 2022 at 04:33:35AM -0400, Felix Miata wrote:
I repeat: the $SUBJECT behavior has occurred on two different installations, one Friday, one Sunday night. zypper dup excluded kernel on each, which is SOP here. Latest prior kernel was 5.17.9 on the Friday PC, 5.18.2 on the later. rpm -ivh caused the 5.18.9-2 installations that removed all the older kernels.
Was rpm updated recently causing random behavior change by any chance?
A new (since previous dup) version was installed, 4.17.0-9.1, 2022-06-29 10:04. My usual dup procedure since about 2-3 months ago: 1- # zypper ref 2- # zypper -v dup -d 3- # zypstart #!/bin/sh # /usr/local/zypstart (# cat /usr/local/bin/zypstart zypper -v in --download-in-advance zypper libzypp libsolv-tools rpm libproxy1 libmodman1 curl openSUSE-release coreutils filesystem rm /etc/zypp/repos.d/repo*.repo 2> /dev/null zypper -v in --download-in-advance device-mapper glibc mdadm systemd udev aaa_base) 4- # zup (# cat /usr/local/bin/zup #!/bin/sh echo 'time zypper -v up' time zypper -v up) 5- # zypper -v dup 6- (optional) # rpm -ivh <kernel> #3 is an emulation of mandrake/mandriva/mageia's urpmi, which when doing an upgrade that includes any package management changes, upgrades the package management rpms and their deps, then restarts itself before proceeding with the rest of the upgrade processes. zypstart was written before I started caching rpms on LAN server to bind mount to /var/log/zypp/packages/ before doing dups. -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata
participants (5)
-
Felix Miata -
Jiri Slaby -
Manfred Hollstein -
Michal Suchánek -
Petr Mladek