Sometime after booting 20240605 in order to upgrade, 5 locked kernel-defaults
disappeared without any known action on my part to cause it: 6.3.9, 6.4.12, 6.5.9,
6.6.11 & 6.7.9. The only clues I can find that they were ever installed are their
existing immutable initrds, and their backups, in /boot/. I've looked in
/var/log/zypper.log, /var/log/zypp/history, journalctl, .bash_history and found
not so much as a clue what could have made them disappear. Before I started the
zypper dup process, I remember having somewhere around 85% of / in use. At the
time I discovered the absence, it had dropped to 73%, proving they really had gone
during the day, not some previous session.
# zypper ll | grep kern
23 | kernel-de* | package | (any) |
24 | kernel-lo* | package | (any) |
# grep "rpm -e" .bash_history | grep kernel
rpm -e kernel-default-5.3.12-1.1
rpm -e kernel-default-5.3.12-1.1
time rpm -e /nfs/00srv/pub/Linux/Suse/Factory/kernel-default-5.18.11-1.1.x86_64.rpm
time rpm -e /nfs/00srv/pub/Linux/Suse/Factory/kernel-default-5.18.11-1.1
time rpm -e /nfs/00srv/pub/Linux/Suse/Factory/kernel-default-5.18.11-1
time rpm -e kernel-default-5.18.11-1.1
time rpm -e /nfs/00srv/pub/Linux/Suse/Factory/kernel-default-5.18.11-1.1.x86_64.rpm
time rpm -e /nfs/00srv/pub/Linux/Suse/Factory/kernel-default-5.18.11-1.1
time rpm -e /nfs/00srv/pub/Linux/Suse/Factory/kernel-default-5.18.11-1
time rpm -e kernel-default-5.18.11-1.1
# grep rm .bash_history | grep kernel
zypper rm --clean-deps purge-kernels-service
zypper rm purge-kernels-service
chattr -i initrd-5.18.12-1-default; zypper rm kernel-default-5.18.12
chattr -i initrd-5.18.12-1-default; zypper rm kernel-default-5.18.12
zypper rm kernel-default-5.4.14
zypper rm kernel-default-5.4.14
zypper rm kernel-default-5.5.13
zypper rm kernel-default-5.6.14
zypper rm kernel-default-5.5.13
zypper rm kernel-default-5.6.14
dfh /; time zypper rm kernel-default-5.7.11 ; dfh /
dfh /; time zypper rm kernel-default-5.8.15 ; dfh /
dfh /; time zypper rm kernel-default-5.9.14 ; dfh /
dfh /; time zypper rm kernel-default-5.7.11 ; dfh /
dfh /; time zypper rm kernel-default-5.8.15 ; dfh /
dfh /; time zypper rm kernel-default-5.9.14 ; dfh /
zypper rm kernel-default-5.10.16
zypper rm kernel-default-5.10.16
time zypper rm kernel-default-6.1.8
time zypper rm kernel-default-6.1.8
time zypper rm kernel-default-6.1.8
time zypper rm kernel-default-6.1.8
zypper rm kernel-default-5.11.16
time zypper rm kernel-default-5.12.13
zypper rm kernel-firmware-sound kernel-firmware-platform kernel-firmware-media
kernel-firmware-network
zypper rm kernel-default-5.13.12
zypper rm kernel-default-5.11.16
time zypper rm kernel-default-5.12.13
zypper rm kernel-firmware-sound kernel-firmware-platform kernel-firmware-media
kernel-firmware-network
zypper rm kernel-default-5.13.12
zypper rm kernel-default-6.2.8
zypper rm kernel-default
time rpm -ivh /nfs/00srv/pub/Linux/Suse/Factory/kernel-longterm-6.6.21-1.1.x86_64.rpm
time rpm -ivh /nfs/00srv/pub/Linux/Suse/Factory/kernel-longterm-6.6.21-1.1.x86_64.rpm
time rpm -ivh /nfs/00srv/pub/Linux/Suse/Factory/kernel-longterm-6.6.25-1.1.x86_64.rpm
time rpm -ivh /nfs/00srv/pub/Linux/Suse/Factory/kernel-longterm-6.6.25-1.1.x86_64.rpm
time rpm -ivh /nfs/00srv/pub/Linux/Suse/Factory/kernel-longterm-6.6.31-1.1.x86_64.rpm
time rpm -ivh /nfs/00srv/pub/Linux/Suse/Factory/kernel-longterm-6.6.31-1.1.x86_64.rpm
# lsattr /boot/initrd-6*
----i---------e------- /boot/initrd-6.3.9-1-default
----i---------e------- /boot/initrd-6.4.12-1-default
----i---------e------- /boot/initrd-6.5.9-1-default
----i---------e------- /boot/initrd-6.6.11-1-default
----i---------e------- /boot/initrd-6.6.21-1-longterm
----i---------e------- /boot/initrd-6.6.25-1-longterm
----i---------e------- /boot/initrd-6.6.31-1-longterm
----i---------e------- /boot/initrd-6.7.9-1-default
----i---------e------- /boot/initrd-6.8.9-1-default
# ls -Ggl /boot/vmlinuz-6* /boot/initrd-6*
-rw------- 1 17586190 Aug 11 2023 /boot/initrd-6.3.9-1-default
-rw------- 1 15380244 Oct 13 2023 /boot/initrd-6.4.12-1-default
-rw------- 1 15942348 Nov 15 2023 /boot/initrd-6.5.9-1-default
-rw------- 1 16281120 Jan 24 22:22 /boot/initrd-6.6.11-1-default
-rw------- 1 16146376 Mar 9 22:17 /boot/initrd-6.6.21-1-longterm
-rw------- 1 16301820 Apr 16 00:09 /boot/initrd-6.6.25-1-longterm
-rw------- 1 16386552 May 22 22:59 /boot/initrd-6.6.31-1-longterm
-rw------- 1 16454412 Jun 13 23:31 /boot/initrd-6.7.9-1-default
-rw------- 1 16514788 Jul 10 23:45 /boot/initrd-6.8.9-1-default
lrwxrwxrwx 1 44 Mar 9 22:17 /boot/vmlinuz-6.6.21-1-longterm ->
../usr/lib/modules/6.6.21-1-longterm/vmlinuz
lrwxrwxrwx 1 44 Apr 16 00:09 /boot/vmlinuz-6.6.25-1-longterm ->
../usr/lib/modules/6.6.25-1-longterm/vmlinuz
lrwxrwxrwx 1 44 May 22 22:59 /boot/vmlinuz-6.6.31-1-longterm ->
../usr/lib/modules/6.6.31-1-longterm/vmlinuz
lrwxrwxrwx 1 42 Jul 10 23:45 /boot/vmlinuz-6.8.9-1-default ->
../usr/lib/modules/6.8.9-1-default/vmlinuz
# ls -gGld /usr/lib/modules
drwxr-xr-x 7 1024 Jul 10 23:45 /usr/lib/modules
# ls -gGl /usr/lib/modules
total 5
drwxr-xr-x 2 1024 Jan 29 2023 5.5.13-1-default
drwxr-xr-x 3 1024 Mar 9 22:17 6.6.21-1-longterm
drwxr-xr-x 3 1024 Apr 16 00:09 6.6.25-1-longterm
drwxr-xr-x 3 1024 May 22 22:59 6.6.31-1-longterm
drwxr-xr-x 3 1024 Jul 10 23:45 6.8.9-1-default
# systemctl list-unit-files | grep purge
purge-kernels.service masked enabled
# ls -gGl /etc/systemd/system/purge*
lrwxrwxrwx 1 9 May 5 2019 /etc/systemd/system/purge-kernels.service -> /dev/null
# rpm -qa | egrep 'nel-def|nel-lon'
kernel-longterm-6.6.21-1.1.x86_64
kernel-longterm-6.6.25-1.1.x86_64
kernel-longterm-6.6.31-1.1.x86_64
kernel-default-6.8.9-1.1.x86_64
# journalctl | grep 'rpm -e'
# journalctl | grep zypper
Apr 15 23:37:58 gx780 [RPM][1851]: erase zypper-1.14.68-1.3.x86_64: success
Apr 15 23:37:58 gx780 [RPM][1851]: install zypper-1.14.70-1.1.x86_64: success
Apr 15 23:37:59 gx780 [RPM][1851]: erase zypper-1.14.68-1.3.x86_64: success
Apr 15 23:37:59 gx780 [RPM][1851]: install zypper-1.14.70-1.1.x86_64: success
Apr 15 23:53:13 gx780 [RPM][24779]: erase zypper-log-1.14.68-1.3.noarch: success
Apr 15 23:53:13 gx780 [RPM][24779]: install zypper-log-1.14.70-1.1.noarch: success
Apr 15 23:53:13 gx780 [RPM][24779]: erase zypper-log-1.14.68-1.3.noarch: success
Apr 15 23:53:13 gx780 [RPM][24779]: install zypper-log-1.14.70-1.1.noarch: success
May 22 22:41:15 gx780 [RPM][1428]: erase zypper-1.14.70-1.1.x86_64: success
May 22 22:41:15 gx780 [RPM][1428]: install zypper-1.14.73-1.2.x86_64: success
May 22 22:41:16 gx780 [RPM][1428]: erase zypper-1.14.70-1.1.x86_64: success
May 22 22:41:16 gx780 [RPM][1428]: install zypper-1.14.73-1.2.x86_64: success
May 22 22:50:49 gx780 [RPM][12448]: erase zypper-log-1.14.70-1.1.noarch: success
May 22 22:50:49 gx780 [RPM][12448]: install zypper-log-1.14.73-1.2.noarch: success
May 22 22:50:49 gx780 [RPM][12448]: erase zypper-log-1.14.70-1.1.noarch: success
May 22 22:50:49 gx780 [RPM][12448]: install zypper-log-1.14.73-1.2.noarch: success
Jul 10 23:35:33 gx780 [RPM][2495]: erase zypper-log-1.14.73-1.2.noarch: success
Jul 10 23:35:33 gx780 [RPM][2495]: install zypper-log-1.14.76-1.1.noarch: success
Jul 10 23:35:33 gx780 [RPM][2495]: erase zypper-log-1.14.73-1.2.noarch: success
Jul 10 23:35:33 gx780 [RPM][2495]: install zypper-log-1.14.76-1.1.noarch: success
Jul 10 23:37:45 gx780 [RPM][2922]: erase zypper-1.14.73-1.2.x86_64: success
Jul 10 23:37:46 gx780 [RPM][2922]: install zypper-1.14.76-1.1.x86_64: success
Jul 10 23:37:46 gx780 [RPM][2922]: erase zypper-1.14.73-1.2.x86_64: success
Jul 10 23:37:46 gx780 [RPM][2922]: install zypper-1.14.76-1.1.x86_64: success
#
I've obviously self-limited data available by using rpm instead of zypper for most
TW & SR kernel installations & removals.
Digging through /var/log/zypper.log is a painful, unproductive, noisy process. I
have no idea whether what I need to know is hidden in there or not. I searched for
various strings I thought could be useful, but came up zip.
This is the second time this has happened. The first time I sort of ignored it by
reinstalling the removed kernels. Whether it was TW or SR I don't remember, nor
when, but probably since last Christmas. Eventually I can probably find it by
looking in /usr/lib/modules/ directories for a whole bunch of the same dates for
disparate kernel versions, but because I have so many, sprawled across 40+ PCs, it
would take quite some time.
Anyone else have this happen? Any ideas what could cause unintentional removal of
locked kernels?
--
Evolution as taught in public schools is, like religion,
based on faith, not based on science.
Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!
Felix Miata