[Bug 1195291] New: initrd is being rebuilt when kernel is being removed after kernel update
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291
Bug ID: 1195291 Summary: initrd is being rebuilt when kernel is being removed after kernel update Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.3 Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-bugs@opensuse.org Reporter: Ulrich.Windl@rz.uni-regensburg.de QA Contact: qa-bugs@suse.de Found By: --- Blocker: ---
Despite of the problem that initrds are needlessy rebuilt when installing kernel modules (bug 1100885, bug 1191322), there are also needless rebuilds when removing obsolete kernels: Still on the next boot when older kernels are removed, their initrds are being rebuilt before the kernels are removed (it seems at least).
Example (combining extracts from journal and /var/log/zypp/history): Jan 27 21:36:03 i7.site [RPM][7347]: install kernel-default-5.3.18-150300.59.43.1.x86_64: success Jan 27 21:36:04 i7.site [RPM][17173]: erase kernel-macros-5.3.18-59.40.1.noarch: success Jan 27 21:36:05 i7.site [RPM][17173]: install kernel-macros-5.3.18-150300.59.43.1.noarch: success Jan 27 21:36:06 i7.site [RPM][17173]: erase kernel-macros-5.3.18-59.40.1.noarch: success Jan 27 21:36:06 i7.site [RPM][17173]: install kernel-macros-5.3.18-150300.59.43.1.noarch: success Jan 27 21:37:32 i7.site [RPM][17175]: install kernel-preempt-5.3.18-150300.59.43.1.x86_64: success Jan 27 21:37:47 i7.site [RPM][27390]: install kernel-default-extra-5.3.18-150300.59.43.1.x86_64: success Jan 27 21:37:55 i7.site [RPM][27492]: install kernel-devel-5.3.18-150300.59.43.1.noarch: success Jan 27 21:37:55 i7.site [RPM][27492]: install kernel-devel-5.3.18-150300.59.43.1.noarch: success Jan 27 21:38:03 i7.site [RPM][27497]: install kernel-preempt-extra-5.3.18-150300.59.43.1.x86_64: success Jan 27 21:38:25 i7.site [RPM][27675]: install kernel-default-optional-5.3.18-150300.59.43.1.x86_64: success Jan 27 21:38:28 i7.site [RPM][27776]: install kernel-preempt-devel-5.3.18-150300.59.43.1.x86_64: success Jan 27 21:38:28 i7.site [RPM][27776]: install kernel-preempt-devel-5.3.18-150300.59.43.1.x86_64: success Jan 27 21:38:31 i7.site [RPM][27782]: install kernel-default-devel-5.3.18-150300.59.43.1.x86_64: success Jan 27 21:38:31 i7.site [RPM][27782]: install kernel-default-devel-5.3.18-150300.59.43.1.x86_64: success Jan 27 21:38:37 i7.site [RPM][27788]: install kernel-preempt-optional-5.3.18-150300.59.43.1.x86_64: success # 2022-01-27 21:43:41 Output of kernel-default-extra-5.3.18-150300.59.43.1.x86_64.rpm %posttrans script: # Creating initrd: /boot/initrd-5.3.18-150300.59.43-default # Creating initrd: /boot/initrd-5.3.18-150300.59.43-preempt # Creating initrd: /boot/initrd-5.3.18-59.40-default # Creating initrd: /boot/initrd-5.3.18-59.40-preempt
### Reboot ###
Jan 28 20:24:49 i7 zypper[1916]: The following 9 packages are going to be REMOVED: Jan 28 20:24:49 i7 zypper[1916]: kernel-default-5.3.18-59.37.2 kernel-default-devel-5.3.18-59.37.2 kernel-default-extra-5.3.18-59.37.2 kernel-default-optional-5.3.18-59.37.2 kernel-devel-5.3.18-59.37.2 kernel-preempt-5.3.18-59.37.2 kernel-preempt-devel-5.3.18-59.37.2 kernel-preempt-extra-5.3.18-59.37.2 kernel-preempt-optional-5.3.18-59.37.2 Jan 28 20:24:49 i7 zypper[1916]: 9 packages to remove. Jan 28 20:27:04 i7.site [RPM][2887]: erase kernel-default-devel-5.3.18-59.37.2.x86_64: success Jan 28 20:27:07 i7.site [RPM][2887]: erase kernel-default-devel-5.3.18-59.37.2.x86_64: success Jan 28 20:27:07 i7.site zypper[1916]: (1/9) Removing kernel-default-devel-5.3.18-59.37.2.x86_64 [.....done] Jan 28 20:27:08 i7.site [RPM][4137]: erase kernel-default-optional-5.3.18-59.37.2.x86_64: success Jan 28 20:31:10 i7.site [RPM][4137]: erase kernel-default-optional-5.3.18-59.37.2.x86_64: success Jan 28 20:31:10 i7.site zypper[1916]: (2/9) Removing kernel-default-optional-5.3.18-59.37.2.x86_64 [.....done] Jan 28 20:31:11 i7.site [RPM][11952]: erase kernel-preempt-devel-5.3.18-59.37.2.x86_64: success Jan 28 20:31:20 i7.site [RPM][11952]: erase kernel-preempt-devel-5.3.18-59.37.2.x86_64: success Jan 28 20:31:20 i7.site zypper[1916]: (3/9) Removing kernel-preempt-devel-5.3.18-59.37.2.x86_64 [.....done] Jan 28 20:31:20 i7.site [RPM][11958]: erase kernel-preempt-optional-5.3.18-59.37.2.x86_64: success Jan 28 20:32:24 i7.site [RPM][11958]: erase kernel-preempt-optional-5.3.18-59.37.2.x86_64: success Jan 28 20:32:24 i7.site zypper[1916]: (4/9) Removing kernel-preempt-optional-5.3.18-59.37.2.x86_64 [.....done] Jan 28 20:32:25 i7.site [RPM][19401]: erase kernel-default-extra-5.3.18-59.37.2.x86_64: success Jan 28 20:33:13 i7.site [RPM][19401]: erase kernel-default-extra-5.3.18-59.37.2.x86_64: success Jan 28 20:33:13 i7.site zypper[1916]: (5/9) Removing kernel-default-extra-5.3.18-59.37.2.x86_64 [.....done] Jan 28 20:33:13 i7.site zypper[1916]: Creating initrd: /boot/initrd-5.3.18-59.37-default # Creating initrd: /boot/initrd-5.3.18-59.37-default Jan 28 20:33:17 i7.site [RPM][26818]: erase kernel-devel-5.3.18-59.37.2.noarch: success Jan 28 20:33:24 i7.site [RPM][26818]: erase kernel-devel-5.3.18-59.37.2.noarch: success Jan 28 20:33:24 i7.site zypper[1916]: (6/9) Removing kernel-devel-5.3.18-59.37.2.noarch [.....done] Jan 28 20:33:24 i7.site [RPM][26819]: erase kernel-preempt-extra-5.3.18-59.37.2.x86_64: success Jan 28 20:34:11 i7.site [RPM][26819]: erase kernel-preempt-extra-5.3.18-59.37.2.x86_64: success Jan 28 20:34:11 i7.site zypper[1916]: (7/9) Removing kernel-preempt-extra-5.3.18-59.37.2.x86_64 [.....done] Jan 28 20:34:11 i7.site zypper[1916]: Creating initrd: /boot/initrd-5.3.18-59.37-preempt # Creating initrd: /boot/initrd-5.3.18-59.37-preempt Jan 28 20:34:12 i7.site [RPM][1784]: erase kernel-default-5.3.18-59.37.2.x86_64: success Jan 28 20:34:43 i7.site [RPM][1784]: erase kernel-default-5.3.18-59.37.2.x86_64: success Jan 28 20:34:43 i7.site zypper[1916]: (8/9) Removing kernel-default-5.3.18-59.37.2.x86_64 [.....done] Jan 28 20:34:44 i7.site [RPM][4725]: erase kernel-preempt-5.3.18-59.37.2.x86_64: success Jan 28 20:35:15 i7.site [RPM][4725]: erase kernel-preempt-5.3.18-59.37.2.x86_64: success Jan 28 20:35:15 i7.site zypper[1916]: (9/9) Removing kernel-preempt-5.3.18-59.37.2.x86_64 [.....done] Jan 28 20:35:15 i7.site systemd[1]: purge-kernels.service: Succeeded.
In addition to rebuiling the initrds, the bootloader was reinstalled (including the os-prober), so the removal of the old kernels took 10 minutes alltogether!
# grep run_command.*grub2 /var/log/pbl.log 2022-01-27 21:35:35 <1> update-bootloader-7882 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-27 21:36:03 <1> bootloader_entry-1449 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-27 21:37:03 <1> update-bootloader-9958 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-27 21:37:32 <1> bootloader_entry-8996 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-27 21:41:39 <1> update-bootloader-8968 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-27 21:42:20 <1> update-bootloader-0547 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-27 21:43:00 <1> update-bootloader-6949 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-27 21:43:41 <1> update-bootloader-7629 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-28 20:31:09 <1> update-bootloader-0043 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-28 20:32:23 <1> update-bootloader-4882 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-28 20:33:12 <1> update-bootloader-4790 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-28 20:34:11 <1> update-bootloader-4190 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-28 20:34:43 <1> bootloader_entry-0829 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output: 2022-01-28 20:35:14 <1> bootloader_entry-7074 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output:
Reinstalling the boot loader 15 times when updating two kernels!?
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291
Felix Miata mrmazda@earthlink.net changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |mrmazda@earthlink.net
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c7
--- Comment #7 from Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de --- Created attachment 856130 --> http://bugzilla.opensuse.org/attachment.cgi?id=856130&action=edit Output when removing an older kernel
I had removed an older kernel in one RPM transaction, but still the initrd of the kernel to remove was rebuilt twice.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c8
--- Comment #8 from Martin Wilck martin.wilck@suse.com --- I'm repeating myself here: you can't expect anything else. We (myself, at least) won't work on this before 15.4 is released. I simply have no time for it.
If you want to provide and test patches, feel very welcome.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c9
Felix Miata mrmazda@earthlink.net changed:
What |Removed |Added ---------------------------------------------------------------------------- Version|Leap 15.3 |Leap 15.5 Summary|initrd is being rebuilt |initrd is being rebuilt |when kernel is being |when kernel is being |removed after kernel update |removed
--- Comment #9 from Felix Miata mrmazda@earthlink.net --- To reproduce: 1-(e.g.) # zypper rm kernel-default-5.14.19-150400.1
Actual behavior: 1-(from /var/log/history): 2022-08-20 16:44:01|command|root@gb970|'zypper' 'rm' 'kernel-default-5.14.19-150400.1'| # 2022-08-20 16:44:40 kernel-default-extra-5.14.19-150400.1.1.x86_64 removed ok # Additional rpm output: # dracut: Executing: /usr/bin/dracut -f /boot/initrd-5.14.19-150400.1-default 5.14.19-150400.1-default ... # dracut: *** Stripping files done *** # dracut: *** Creating image file '/boot/initrd-5.14.19-150400.1-default' *** # dracut: *** Creating initramfs image file '/boot/initrd-5.14.19-150400.1-default' done *** # 2022-08-20 16:44:40|remove |kernel-default-extra|5.14.19-150400.1.1|x86_64|root@gb970| 2022-08-20 16:44:47|remove |kernel-default|5.14.19-150400.1.1|x86_64|root@gb970|
Expected behavior: 1-2022-08-20 16:44:01|command|root@gb970|'zypper' 'rm' 'kernel-default-5.14.19-150400.1'| # 2022-08-20 16:44:40 kernel-default-extra-5.14.19-150400.1.1.x86_64 removed ok 2022-08-20 16:44:40|remove |kernel-default-extra|5.14.19-150400.1.1|x86_64|root@gb970| 2022-08-20 16:44:47|remove |kernel-default|5.14.19-150400.1.1|x86_64|root@gb970| 2-no resources wasted rebuilding initrd that is to be immediately removed
Additional comments: 1-kernel-default-extra on this and several other hosts here gets installed for the exclusive reason that the required nouveau kernel module is omitted from the kernel-default package. 2-purge-kernels service is not used here. Kernels are removed completely independently of any kernel update processes, so adjusting summary accordingly, and updating version from 15.3 to 15.5 per comment #8.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c10
Michal Suchanek msuchanek@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |Ulrich.Windl@rz.uni-regensb | |urg.de Version|Leap 15.5 |Leap 15.4 Assignee|kernel-bugs@opensuse.org |martin.wilck@suse.com Flags| |needinfo?(Ulrich.Windl@rz.u | |ni-regensburg.de), | |needinfo?(mrmazda@earthlink | |.net)
--- Comment #10 from Michal Suchanek msuchanek@suse.com --- This is with suse-module-tools now.
Given that upgrade and remove run the same scripts (per rpm API) this is not trivial to fix. Patches accepted.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c11
Felix Miata mrmazda@earthlink.net changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(mrmazda@earthlink |needinfo?(msuchanek@suse.co |.net) |m)
--- Comment #11 from Felix Miata mrmazda@earthlink.net --- I have no idea what info is requested. https://bugzilla.mozilla.org/show_bug.cgi?id=187256 11 years ago contains the only patches I can recall ever managing to submit anywhere, nothing but some changes to a CSS font size table. My understanding of programming is barely
zero.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c12
Felix Miata mrmazda@earthlink.net changed:
What |Removed |Added ---------------------------------------------------------------------------- Version|Leap 15.4 |Leap 15.5
--- Comment #12 from Felix Miata mrmazda@earthlink.net --- (In reply to Martin Wilck from comment #8, 13 months ago)
We (myself, at least) won't work on this before 15.4 is released. I simply have no time for it.
How about for 15.5?
How many nouveau.ko users are there? Is nouveau.ko really still "experimental"? These users surely must represent most of the people being subjected this insane rebuilding with instant removal immediately following, besides the overhead of all the other kernel-*-extra content on disk AMD & Intel graphics users aren't forced to host. Are there enough such users to warrant a separate package be provided for it that can be installed even though the matching kernel hasn't been (yet), and dropping it out of the extra package, if it can't be restored to the company of amdgpu.ko, i915.ko, and radeon.ko in the kernel packages?
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c13
--- Comment #13 from Felix Miata mrmazda@earthlink.net --- Created attachment 864077 --> http://bugzilla.opensuse.org/attachment.cgi?id=864077&action=edit /var/log/zypp/history: kernel removal in 15.5 interposing zypper instead of using rpm directly
Couldn't something be done to have zypper remove the kernel *before* -extra when both are called for removal together?
I keep a kernel-de* lock on all installations, choosing when and if to upgrade a kernel. On NVidia installations, to block needless initrd rebuilds, I've usually been installing new kernels via this zypinknv script, which due to the lock, is always interactive:
#!/bin/sh zypper in kernel-default-extra zypper -v in kernel-default
For removal I run the reverse of the above, using rpm --nodeps on the kernel first, then remove the related -extra.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c15
--- Comment #15 from Felix Miata mrmazda@earthlink.net --- (In reply to Martin Wilck from comment #14)
I don't understand how this issue relates to nouveau. That's just one driver out of many.
Many not. Nouveau is the sore thumb drm .ko among AMD, Intel and NVidia providers in -extra: # rpm -ql kernel-default-5.14.21-150400.24.33.2 | grep drm | grep .ko | wc -l 24 # rpm -ql kernel-default-extra-5.14.21-150400.24.33.2 | grep drm | grep .ko /lib/modules/.../gpu/drm/gma500/gma500_gfx.ko.zst # 2008 Intel Atom /lib/modules/.../gpu/drm/nouveau/nouveau.ko.zst # current & past NVidia /lib/modules/.../gpu/drm/tiny/gm12u320.ko.zst # USB projector /lib/modules/.../gpu/drm/vboxvideo/vboxvideo.ko.zst # ??? #
NVidia users who wish to use exclusively FOSS are being functionally penalized by needing -extra for the singular purpose of decent GPU support, unlike AMD, (the vast majority of) Intel, & rPI graphics users.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c16
--- Comment #16 from Michal Suchanek msuchanek@suse.com --- nouveau.ko is not decent graphics support, that's why it is not included by default. It tends to crash a lot. There may be some people somewhat maintaining it but there is not enough knowledge and manpower to make it into 'decent' graphics support.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c17
Felix Miata mrmazda@earthlink.net changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |sndirsch@suse.com
--- Comment #17 from Felix Miata mrmazda@earthlink.net --- Are you sure it's nouveau.ko and not nouveau_drv.so responsible for nouveau's crashing reputation? Nouveau_drv.so isn't needed for decent graphics support. Upstream Xorg has had as standard modesetting_drv.so since 2015, relegating nouveau_drv.so to the optional/reverse-engineered/experimental status it earned. I don't recall observing any crashing of NVidia graphics using the nouveau.ko/modesetting_drv.so combination with any of my 4 NVidia cards, none of which have ever while in my possession been subjected to installation of proprietary NVidia drivers.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c19
--- Comment #19 from Felix Miata mrmazda@earthlink.net --- I found a workaround useful for picky admins. To prevent the initrd from being rebuilt between kernel-default-extra removal and kernel-default removal, make the initrd immutable. The process stats the initrd to discover immutable status, and skips the rebuild, thus making the removal process much faster and less wasteful of disk writes.
http://bugzilla.opensuse.org/show_bug.cgi?id=1195291 http://bugzilla.opensuse.org/show_bug.cgi?id=1195291#c20
Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de changed:
What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(Ulrich.Windl@rz.u | |ni-regensburg.de) |
--- Comment #20 from Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de --- Well, SUSE introduced the problem quickly, but takes a lot of time to fix it. Meanwhile users suffer: After having updated the kernel, I had to wait almost 5 minutes until I could log in after boot. That's almost as slow as booting Windows.
# systemd-analyze Startup finished in 813ms (kernel) + 13.472s (initrd) + 4min 1.948s (userspace) = 4min 16.234s graphical.target reached after 4min 1.908s in userspace # systemd-analyze blame 3min 41.865s plymouth-quit-wait.service 3min 29.156s purge-kernels.service 27.809s mandb.service ... 9ms dracut-pre-pivot.service 8ms dracut-pre-mount.service 4ms initrd-udevadm-cleanup-db.service 1ms sys-fs-fuse-connections.mount 1ms sys-kernel-config.mount
Summary from /var/log/zypp/history: 2023-01-28 17:31:44|command|root@i7g4x4a|'/usr/bin/zypper' '-n' 'purge-kernels'| 2023-01-28 17:32:30|remove |kernel-default-devel|5.14.21-150400.24.33.2|x86_64|root@pc| # 2023-01-28 17:33:38 kernel-default-optional-5.14.21-150400.24.33.2.x86_64 removed ok 2023-01-28 17:33:38|remove |kernel-default-optional|5.14.21-150400.24.33.2|x86_64|| 2023-01-28 17:33:54|remove |kernel-devel|5.14.21-150400.24.33.1|noarch|root@pc| # 2023-01-28 17:34:21 kernel-default-extra-5.14.21-150400.24.33.2.x86_64 removed ok 2023-01-28 17:34:21|remove |kernel-default-extra|5.14.21-150400.24.33.2|x86_64|| 2023-01-28 17:35:02|remove |kernel-default|5.14.21-150400.24.33.2|x86_64|root@pc|
So that's three rebuilds of the initrd when removing the packages for _one_ kernel version! (Related to comment #6: It seems bootloader was reconfigured only one, so the os-prober is not the biggest contributor) How does Fedora manage it to make only one rebuild of the initrd at the end? Maybe have a look!
In version before the bug, kernel removal took place in the background at least, and the user was able to start working while kernels were removed.
-
bugzilla_noreply＠suse.com