https://bugzilla.suse.com/show_bug.cgi?id=1198668
Bug ID: 1198668 Summary: Tumbleweed: Purge old kernels service block during boot Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem Assignee: screening-team-bugs@suse.de Reporter: petr.vorel@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: ---
Sometimes "Purge old kernels" service does not run after booting, but during boot, which slows down boot:
$ LC_ALL=C journalctl -b 0 |grep -i purge Apr 20 08:16:01 dell5510 systemd[1]: Starting Purge old kernels... Apr 20 08:16:02 dell5510 zypper[1546]: Preparing to purge obsolete kernels... Apr 20 08:17:09 dell5510 systemd[1]: purge-kernels.service: Deactivated successfully. Apr 20 08:17:09 dell5510 systemd[1]: Finished Purge old kernels. Apr 20 08:17:09 dell5510 systemd[1]: purge-kernels.service: Consumed 53.594s CPU time.
It started few weeks ago, it does not happen every boot, but maybe once a week. But bothered me enough to report the problem.
$ LC_ALL=C rpm -qi purge-kernels-service Name : purge-kernels-service Version : 0 Release : 8.4 Architecture: noarch Install Date: Tue Feb 22 11:51:00 2022 Group : Unspecified Size : 346 License : MIT Signature : RSA/SHA256, Sat Feb 19 22:50:00 2022, Key ID b88b2fd43dbdc284 Source RPM : purge-kernels-service-0-8.4.src.rpm Build Date : Sat Feb 19 22:37:41 2022 ...
$ lsb_release -a LSB Version: n/a Distributor ID: openSUSE Description: openSUSE Tumbleweed Release: 20220415 Codename: n/a
https://bugzilla.suse.com/show_bug.cgi?id=1198668
Chenzi Cao chcao@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Assignee|screening-team-bugs@suse.de |msuchanek@suse.com
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c1
--- Comment #1 from Michal Suchanek msuchanek@suse.com --- It has dependencies only on fs so it can run relatively early
Description=Purge old kernels After=local-fs.target ConditionPathExists=/boot/do_purge_kernels ConditionPathIsReadWrite=/
What does 'slows down boot' mean, specifically?
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c6
Yiannis Bonatakis ioannis.bonatakis@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |ioannis.bonatakis@suse.com
--- Comment #6 from Yiannis Bonatakis ioannis.bonatakis@suse.com --- takes <10m on my laptop. which is kinda annoying to wait for so long for the booting.
``` ��� sudo systemd-analyze blame 13min 21.087s purge-kernels.service 13min 4.007s plymouth-quit-wait.service ```
and journal reports
``` ��� sudo journalctl -u purge-kernels Apr 27 09:52:11 anais systemd[1]: Starting Purge old kernels... Apr 27 09:52:46 anais.suse.cz zypper[1442]: Reading installed packages... Apr 27 09:52:47 anais.suse.cz zypper[1442]: Preparing to purge obsolete kernels... Apr 27 09:52:47 anais.suse.cz zypper[1442]: Configuration: latest,latest-1,running Apr 27 09:52:47 anais.suse.cz zypper[1442]: Running kernel release: 5.17.3-1-default Apr 27 09:52:47 anais.suse.cz zypper[1442]: Running kernel arch: x86_64 Apr 27 09:52:47 anais.suse.cz zypper[1442]: Resolving package dependencies... Apr 27 09:52:47 anais.suse.cz zypper[1442]: The following 4 packages are going to be REMOVED: Apr 27 09:52:47 anais.suse.cz zypper[1442]: bbswitch-kmp-default-0.8_k5.16.15_1-11.60 bbswitch-kmp-default-0.8_k5.17.1_1-11.61 kernel-default-5.16.15-1.1 kernel-default-5.17.1-1.2 Apr 27 09:52:47 anais.suse.cz zypper[1442]: 4 packages to remove. Apr 27 09:52:47 anais.suse.cz zypper[1442]: After the operation, 499.7 MiB will be freed. Apr 27 09:52:47 anais.suse.cz zypper[1442]: Continue? [y/n/v/...? shows all options] (y): y Apr 27 09:53:52 anais.suse.cz [RPM][2787]: Transaction ID 6268f690 started Apr 27 09:53:59 anais.suse.cz [RPM][2787]: erase bbswitch-kmp-default-0.8_k5.16.15_1-11.60.x86_64: success Apr 27 09:59:11 anais.suse.cz [RPM][2787]: erase bbswitch-kmp-default-0.8_k5.16.15_1-11.60.x86_64: success Apr 27 09:59:11 anais.suse.cz [RPM][2787]: Transaction ID 6268f690 finished: 0 Apr 27 09:59:18 anais.suse.cz zypper[1442]: (1/4) Removing bbswitch-kmp-default-0.8_k5.16.15_1-11.60.x86_64 [.....done] Apr 27 09:59:18 anais.suse.cz [RPM][11672]: Transaction ID 6268f7d6 started Apr 27 09:59:18 anais.suse.cz [RPM][11672]: erase bbswitch-kmp-default-0.8_k5.17.1_1-11.61.x86_64: success Apr 27 10:04:18 anais.suse.cz [RPM][11672]: erase bbswitch-kmp-default-0.8_k5.17.1_1-11.61.x86_64: success Apr 27 10:04:18 anais.suse.cz [RPM][11672]: Transaction ID 6268f7d6 finished: 0 Apr 27 10:04:22 anais.suse.cz zypper[1442]: (2/4) Removing bbswitch-kmp-default-0.8_k5.17.1_1-11.61.x86_64 [.....done] Apr 27 10:04:22 anais.suse.cz [RPM][20508]: Transaction ID 6268f906 started Apr 27 10:04:25 anais.suse.cz [RPM][20508]: erase kernel-default-5.16.15-1.1.x86_64: success Apr 27 10:04:44 anais.suse.cz [RPM][20508]: erase kernel-default-5.16.15-1.1.x86_64: success Apr 27 10:04:44 anais.suse.cz [RPM][20508]: Transaction ID 6268f906 finished: 0 Apr 27 10:04:58 anais.suse.cz zypper[1442]: (3/4) Removing kernel-default-5.16.15-1.1.x86_64 [.....done] Apr 27 10:04:58 anais.suse.cz [RPM][21638]: Transaction ID 6268f92a started Apr 27 10:04:58 anais.suse.cz [RPM][21638]: erase kernel-default-5.17.1-1.2.x86_64: success Apr 27 10:05:10 anais.suse.cz macosx-prober[22483]: debug: /dev/sda2 is not an HFS+ partition: exiting Apr 27 10:05:13 anais.suse.cz [RPM][21638]: erase kernel-default-5.17.1-1.2.x86_64: success Apr 27 10:05:13 anais.suse.cz [RPM][21638]: Transaction ID 6268f92a finished: 0 Apr 27 10:05:21 anais.suse.cz zypper[1442]: (4/4) Removing kernel-default-5.17.1-1.2.x86_64 [.....done] Apr 27 10:05:30 anais.suse.cz zypper[1442]: There are running programs which still use files and libraries deleted or updated by recent upgrades. They should be restarted to benefit from the latest updates. Run 'zypper ps -s' to list th> Apr 27 10:05:30 anais.suse.cz zypper[1442]: Apr 27 10:05:30 anais.suse.cz systemd[1]: purge-kernels.service: Deactivated successfully. Apr 27 10:05:30 anais.suse.cz systemd[1]: Finished Purge old kernels. Apr 27 10:05:30 anais.suse.cz systemd[1]: purge-kernels.service: Consumed 2min 12.374s CPU time.
```
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c7
--- Comment #7 from Petr Vorel petr.vorel@suse.com --- (In reply to Michal Suchanek from comment #4)
Why is it waiting?
I'll try to debug whether starting purge-kernels.service somehow depend on the delay between user input for decrypting /home. Obviously it'd be great if it depend on GUI already started (if GUI enabled).
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c8
--- Comment #8 from Michal Suchanek msuchanek@suse.com --- It reports that purge-kernels and some 'waiting for boot to finish' is running.
So probably that 'waiting for boot to finish' waits for purge-kernels but I have no idea how that dependency is created.
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c9
--- Comment #9 from Michal Suchanek msuchanek@suse.com --- the other running service is likely
/usr/lib/systemd/system/plymouth-quit-wait.service
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c10
Michal Suchanek msuchanek@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |systemd-maintainers@suse.de Flags| |needinfo?(systemd-maintaine | |rs@suse.de)
--- Comment #10 from Michal Suchanek msuchanek@suse.com --- Please advise how to diagnose why the desktop cannot be displyed while purge-kernels.service is running.
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c11
Michal Koutn� mkoutny@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |mkoutny@suse.com
--- Comment #11 from Michal Koutn� mkoutny@suse.com --- Quick check would be
systemctl show -p Before purge-kernels.service
(possibly traverse further until you reach systemd-user-sessions.service or default.target (graphical.target))
Alternatively, you may find the ordering path(s) in a plot
systemd-analyze dot --order | dot -Tsvg >systemd.svg
(warning: kind of firehose drinking)
Thirdly, I can see the purge-kernels.service is of Type=oneshot, which means it'll block subsequent jobs until the ExecStart= command finishes. Not sure if there's actually necessity to wait after kernel removal is done, the Type=simple or Type=exec would only wait for the command start and the removal may run in parallel (even with jobs that specify After=purge-kernels.service).
Additionally, there may be extra-systemd implicit dependency (such as the zypper lock) that prevents boot progression.
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c12
--- Comment #12 from Michal Suchanek msuchanek@suse.com --- So let's start with the simple part: https://build.opensuse.org/request/show/977061
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c13
Franck Bui fbui@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |fbui@suse.com Flags|needinfo?(systemd-maintaine | |rs@suse.de) |
--- Comment #13 from Franck Bui fbui@suse.com --- I don't think Type=exec is the correct fix here. If there's anything wrong with purge-kernels.service ordering constraints then they should be fixed instead of tweaking the type of the service itself.
You can start by inspecting the ordering constraints of purge-kernels.service with `systemctl list-dependencies --before --all purge-kernels.service` for example.
Or maybe add an additional "ExecStartPre=/usr/bin/sleep 60" in the service to see if the problem is really an ordering issue. Maybe it's more related to slow IOs or such. If that's the case, you might consider purging the kernel regularly during runtime (with a timer unit and with limited IO resources for the service) instead of doing that during the boot process.
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c15
--- Comment #15 from Michal Koutn� mkoutny@suse.com --- (In reply to Tejas Guruswamy from comment #14)
systemctl show -p Before purge-kernels.service
Before=shutdown.target YaST2-Second-Stage.service multi-user.target YaST2-Firstboot.service
I'm not sure purge-kernels.service:Before=multi-user.target would cause the 'waiting for boot to finish' (actually is it a verbatim message? I can't grep it in sources). (The YaST2* service may cause it but it's not what the original report was about AFAICT.)
Because of the '[Install] WantedBy=multi-user.target', it is getting an implicit Before=multi-user.target.
Is After=multi-user.target ok to fix this? Does it even need an [Install] section at all?
You need some other unit to pull-in purge-kernels.service into the boot transaction.
(In reply to Franck Bui from comment #13)
I don't think Type=exec is the correct fix here. If there's anything wrong with purge-kernels.service ordering constraints then they should be fixed instead of tweaking the type of the service itself.
It IMO depends whether the mere launch of the service is important or someone relies on old kernels being truly gone (i.e. wait for purge-kernels to finish).
Maybe it's more related to slow IOs or such.
Yes, in that case the Type=exec workaround would prove ineffective. So I wonder when we can get feedback on the maint-request.
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c16
--- Comment #16 from Michal Suchanek msuchanek@suse.com --- (In reply to Michal Koutn� from comment #15)
(In reply to Tejas Guruswamy from comment #14)
systemctl show -p Before purge-kernels.service
Before=shutdown.target YaST2-Second-Stage.service multi-user.target YaST2-Firstboot.service
I'm not sure purge-kernels.service:Before=multi-user.target would cause the 'waiting for boot to finish' (actually is it a verbatim message? I can't grep it in sources).
AFAICT it's /usr/lib/systemd/system/plymouth-quit-wait.service
https://bugzilla.suse.com/show_bug.cgi?id=1198668
Michal Suchanek msuchanek@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |petr.vorel@suse.com Flags| |needinfo?(petr.vorel@suse.c | |om)
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c19
Petr Vorel petr.vorel@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED Flags|needinfo?(petr.vorel@suse.c | |om) |
--- Comment #19 from Petr Vorel petr.vorel@suse.com --- I haven't noticed any problem for quite long time. I guess the change service type to exec solved the problem, thus closing https://build.opensuse.org/request/show/977061
https://bugzilla.suse.com/show_bug.cgi?id=1198668 https://bugzilla.suse.com/show_bug.cgi?id=1198668#c22
--- Comment #22 from Maintenance Automation maint-coord+maintenance-robot@suse.de --- SUSE-RU-2023:0793-1: An update that has one recommended fix can now be installed.
Category: recommended (moderate) Bug References: 1198668 Sources used: openSUSE Leap 15.4 (src): purge-kernels-service-0-150200.8.6.1 Basesystem Module 15-SP4 (src): purge-kernels-service-0-150200.8.6.1 SUSE Linux Enterprise Real Time 15 SP3 (src): purge-kernels-service-0-150200.8.6.1
NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
-
bugzilla_noreply＠suse.com