[opensuse-arm] Re: [opensuse-project] Raspi3 boot imes
Hi Tobias, Thanks for looking into that. adding opensuse-arm mailinglist, that's where further discussion should take place. Comments below. On 25/01/2019 20:54, admins wrote:
On 25.01.19 19:59, Bruno Friedmann wrote:
On jeudi, 24 janvier 2019 11.53:34 h CET Axel Braun wrote:
Gesendet: Donnerstag, 24. Januar 2019 um 08:53 Uhr Von: "Matthias Brugger" <mbrugger@suse.com> An: "Basil Chupin" <blchupin@iinet.net.au>, opensuse-project <opensuse-project@opensuse.org> Cc: "nicolas saenz julienne" <nicolassaenzj@gmail.com> Betreff: Re: [opensuse-project] No info. re Leap 15.1 in Wikipedia
I think we read about people complaining on the performance of openSUSE this week already. I haven't done any measurements but heard that on Raspberry Pi 3, Debian is significantly faster in booting then openSUSE. Raspi are heavily depending on the kind of 'hard disk' you are using, whether it is a SD card (connected via USB2) or an internal SSD Here is the result of a Raspi using a Leap 15 LXQT image:
/home/test # systemd-analyze blame 1min 30.071s display-manager.service 1min 18.481s backup-rpmdb.service 14.815s ca-certificates.service 13.788s postfix.service 12.885s btrfsmaintenance-refresh.service 11.001s apparmor.service 10.121s logrotate.service 9.277s backup-sysconfig.service 7.925s ModemManager.service 7.755s lvm2-monitor.service 5.889s NetworkManager.service 5.086s postgresql.service 4.790s initrd-switch-root.service 3.074s systemd-journal-flush.service 2.868s nscd.service 2.721s kbdsettings.service 2.672s sshd.service 2.316s ntpd.service 2.191s avahi-daemon.service 2.153s systemd-udevd.service 1.967s polkit.service 1.959s user@1000.service 1.299s systemd-tmpfiles-setup-dev.service 1.171s systemd-remount-fs.service 1.095s sys-kernel-debug.mount 943ms systemd-logind.service 920ms upower.service 803ms wpa_supplicant.service 789ms dev-mqueue.mount 736ms dev-hugepages.mount 715ms boot-efi.mount 598ms udisks2.service 493ms systemd-journald.service 482ms initrd-parse-etc.service 440ms systemd-udev-trigger.service 436ms systemd-modules-load.service 435ms auditd.service 403ms dracut-cmdline.service 348ms systemd-random-seed.service 251ms initrd-cleanup.service 250ms systemd-tmpfiles-setup.service 247ms systemd-sysctl.service 210ms systemd-update-utmp-runlevel.service 207ms dev-disk-by\x2duuid-d90960ec\x2df373\x2d472d\x2d8b3f\x2d27a48d232d7a.swap 193ms systemd-fsck-root.service 184ms plymouth-switch-root.service 170ms sysroot.mount 166ms sys-fs-fuse-connections.mount 159ms systemd-user-sessions.service 151ms kmod-static-nodes.service 112ms systemd-rfkill.service 98ms plymouth-quit.service 97ms plymouth-read-write.service 85ms systemd-vconsole-setup.service 76ms rtkit-daemon.service 60ms dracut-shutdown.service 51ms check-battery.service 51ms systemd-update-utmp.service 21ms initrd-udevadm-cleanup-db.service 18ms sys-kernel-config.mount 16ms plymouth-quit-wait.service
Quite strange, btrfsmaintenance and lvm2-monitor enabled, although both are not used. Clearly most time is burned on display-manager and backup of the rpm-db (could this not be moved to a systemd-timer triggered event at a later point in time?)
Cheers Axel Most of the service you see there aren't they systemd timers that run because it was not running during their normal schedule. Did you try to reboot half an hour after this boot how it behaves ?
Ahrrr tests :-)
I can confirm that the openSUSE boot times on a Raspi 3 are horrendous for several reasons. I tested and tweaked the unstable tumbleweed images, so take it with precaution, Leap might behave different:
* It takes quite some time until even grub2 is loaded (u-boot uefi chainload), but it brings a full grub2 with options to load different kernels or even distributions. (So for me this is worth the wait)
With which u-boot did you test? I think there were some changes to the distroboot scripts which might add more device probing before we actually find and run the grub binary.
* It takes quite along time until the handover (uboot, uefi -> kernel) is done:
+ u-boot slow at reading the actual files (kernel+initrd)
You mean reading the grub binary. Grub is in charge to load kernel+initrd.
+ a not compressed kernel image (~24MB uncompressed vs ~8MB compressed)
+ a initrd which ships (even after first installation and "mkinitrd") several kiwi components
Which components do you refer to?
* Always triggered backup-*.timer(s) and mandb-timer
So to cut boot times idid several things:
* remove all kiwi packages -> smaller initrd file
* disable the above mentioned triggers -> this might not be applicable on a intended stable system
* create and use a kernel with a compressed instead of an uncompressed kernel image [1].
I'm not sure if we should add this to the kernel build infrastructure, as it is a RPi3 thing. Thoughts?
With all this done boot times on my raspi 3 from a slow SD card were cut by around a half (to multi-user console login screen).
[1]: https://build.opensuse.org/package/show/home:tobijk:kernel/kernel-default
Greetings,
Tobias
-- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
On 28.01.19 11:38, Matthias Brugger wrote:
Hi Tobias,
Thanks for looking into that.
adding opensuse-arm mailinglist, that's where further discussion should take place.
Comments below.
On 25/01/2019 20:54, admins wrote:
On 25.01.19 19:59, Bruno Friedmann wrote:
On jeudi, 24 janvier 2019 11.53:34 h CET Axel Braun wrote:
Gesendet: Donnerstag, 24. Januar 2019 um 08:53 Uhr Von: "Matthias Brugger" <mbrugger@suse.com> An: "Basil Chupin" <blchupin@iinet.net.au>, opensuse-project <opensuse-project@opensuse.org> Cc: "nicolas saenz julienne" <nicolassaenzj@gmail.com> Betreff: Re: [opensuse-project] No info. re Leap 15.1 in Wikipedia
I think we read about people complaining on the performance of openSUSE this week already. I haven't done any measurements but heard that on Raspberry Pi 3, Debian is significantly faster in booting then openSUSE. Raspi are heavily depending on the kind of 'hard disk' you are using, whether it is a SD card (connected via USB2) or an internal SSD Here is the result of a Raspi using a Leap 15 LXQT image:
/home/test # systemd-analyze blame 1min 30.071s display-manager.service 1min 18.481s backup-rpmdb.service 14.815s ca-certificates.service 13.788s postfix.service 12.885s btrfsmaintenance-refresh.service 11.001s apparmor.service 10.121s logrotate.service 9.277s backup-sysconfig.service 7.925s ModemManager.service 7.755s lvm2-monitor.service 5.889s NetworkManager.service 5.086s postgresql.service 4.790s initrd-switch-root.service 3.074s systemd-journal-flush.service 2.868s nscd.service 2.721s kbdsettings.service 2.672s sshd.service 2.316s ntpd.service 2.191s avahi-daemon.service 2.153s systemd-udevd.service 1.967s polkit.service 1.959s user@1000.service 1.299s systemd-tmpfiles-setup-dev.service 1.171s systemd-remount-fs.service 1.095s sys-kernel-debug.mount 943ms systemd-logind.service 920ms upower.service 803ms wpa_supplicant.service 789ms dev-mqueue.mount 736ms dev-hugepages.mount 715ms boot-efi.mount 598ms udisks2.service 493ms systemd-journald.service 482ms initrd-parse-etc.service 440ms systemd-udev-trigger.service 436ms systemd-modules-load.service 435ms auditd.service 403ms dracut-cmdline.service 348ms systemd-random-seed.service 251ms initrd-cleanup.service 250ms systemd-tmpfiles-setup.service 247ms systemd-sysctl.service 210ms systemd-update-utmp-runlevel.service 207ms dev-disk-by\x2duuid-d90960ec\x2df373\x2d472d\x2d8b3f\x2d27a48d232d7a.swap 193ms systemd-fsck-root.service 184ms plymouth-switch-root.service 170ms sysroot.mount 166ms sys-fs-fuse-connections.mount 159ms systemd-user-sessions.service 151ms kmod-static-nodes.service 112ms systemd-rfkill.service 98ms plymouth-quit.service 97ms plymouth-read-write.service 85ms systemd-vconsole-setup.service 76ms rtkit-daemon.service 60ms dracut-shutdown.service 51ms check-battery.service 51ms systemd-update-utmp.service 21ms initrd-udevadm-cleanup-db.service 18ms sys-kernel-config.mount 16ms plymouth-quit-wait.service
Quite strange, btrfsmaintenance and lvm2-monitor enabled, although both are not used. Clearly most time is burned on display-manager and backup of the rpm-db (could this not be moved to a systemd-timer triggered event at a later point in time?)
Cheers Axel Most of the service you see there aren't they systemd timers that run because it was not running during their normal schedule. Did you try to reboot half an hour after this boot how it behaves ?
Ahrrr tests :-)
I can confirm that the openSUSE boot times on a Raspi 3 are horrendous for several reasons. I tested and tweaked the unstable tumbleweed images, so take it with precaution, Leap might behave different:
* It takes quite some time until even grub2 is loaded (u-boot uefi chainload), but it brings a full grub2 with options to load different kernels or even distributions. (So for me this is worth the wait)
With which u-boot did you test? I think there were some changes to the distroboot scripts which might add more device probing before we actually find and run the grub binary.
We have a U-Boot timeout of 2 seconds, that adds time. Then U-Boot's USB initialization also takes a while. On top of that, we search for overlay DTs as well, which again may take a few ms. Either way, feel free to identify big chunks here and optimize them :).
* It takes quite along time until the handover (uboot, uefi -> kernel) is done:
+ u-boot slow at reading the actual files (kernel+initrd)
You mean reading the grub binary. Grub is in charge to load kernel+initrd.
+ a not compressed kernel image (~24MB uncompressed vs ~8MB compressed)
+ a initrd which ships (even after first installation and "mkinitrd") several kiwi components
Which components do you refer to?
* Always triggered backup-*.timer(s) and mandb-timer
So to cut boot times idid several things:
* remove all kiwi packages -> smaller initrd file
* disable the above mentioned triggers -> this might not be applicable on a intended stable system
* create and use a kernel with a compressed instead of an uncompressed kernel image [1].
I'm not sure if we should add this to the kernel build infrastructure, as it is a RPi3 thing. Thoughts?
It doesn't have to be an RPi3 thing. Grub can extract files before booting them. We could probably just start shipping Image.gz (or some other compression method) files instead of plain Image ones in kernel-default.rpm. Obviously, this will need a bit of auxiliary adjustments as well. Someone will have to double-check that grub really does extract it before running it. There are also a few scripts that check for the file name and may not execute on an Image.gz file. Once all that is identifies, we should set respective dependency rules in the kernel rpm. But overall, this would be a *big* improvement for everyone. It means we'd waste much less space on /boot too! Could you please open a FATE for this? Alex -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Hi, this time with the real mail address and not the spam mail address of last time (oops), replays down below On 28.01.19 12:02, Alexander Graf wrote:
On 28.01.19 11:38, Matthias Brugger wrote:
Hi Tobias,
Thanks for looking into that.
adding opensuse-arm mailinglist, that's where further discussion should take place.
Comments below.
On 25/01/2019 20:54, admins wrote:
On 25.01.19 19:59, Bruno Friedmann wrote:
On jeudi, 24 janvier 2019 11.53:34 h CET Axel Braun wrote:
Gesendet: Donnerstag, 24. Januar 2019 um 08:53 Uhr Von: "Matthias Brugger" <mbrugger@suse.com> An: "Basil Chupin" <blchupin@iinet.net.au>, opensuse-project <opensuse-project@opensuse.org> Cc: "nicolas saenz julienne" <nicolassaenzj@gmail.com> Betreff: Re: [opensuse-project] No info. re Leap 15.1 in Wikipedia
I think we read about people complaining on the performance of openSUSE this week already. I haven't done any measurements but heard that on Raspberry Pi 3, Debian is significantly faster in booting then openSUSE. Raspi are heavily depending on the kind of 'hard disk' you are using, whether it is a SD card (connected via USB2) or an internal SSD Here is the result of a Raspi using a Leap 15 LXQT image:
/home/test # systemd-analyze blame 1min 30.071s display-manager.service 1min 18.481s backup-rpmdb.service 14.815s ca-certificates.service 13.788s postfix.service 12.885s btrfsmaintenance-refresh.service 11.001s apparmor.service 10.121s logrotate.service 9.277s backup-sysconfig.service 7.925s ModemManager.service 7.755s lvm2-monitor.service 5.889s NetworkManager.service 5.086s postgresql.service 4.790s initrd-switch-root.service 3.074s systemd-journal-flush.service 2.868s nscd.service 2.721s kbdsettings.service 2.672s sshd.service 2.316s ntpd.service 2.191s avahi-daemon.service 2.153s systemd-udevd.service 1.967s polkit.service 1.959s user@1000.service 1.299s systemd-tmpfiles-setup-dev.service 1.171s systemd-remount-fs.service 1.095s sys-kernel-debug.mount 943ms systemd-logind.service 920ms upower.service 803ms wpa_supplicant.service 789ms dev-mqueue.mount 736ms dev-hugepages.mount 715ms boot-efi.mount 598ms udisks2.service 493ms systemd-journald.service 482ms initrd-parse-etc.service 440ms systemd-udev-trigger.service 436ms systemd-modules-load.service 435ms auditd.service 403ms dracut-cmdline.service 348ms systemd-random-seed.service 251ms initrd-cleanup.service 250ms systemd-tmpfiles-setup.service 247ms systemd-sysctl.service 210ms systemd-update-utmp-runlevel.service 207ms dev-disk-by\x2duuid-d90960ec\x2df373\x2d472d\x2d8b3f\x2d27a48d232d7a.swap 193ms systemd-fsck-root.service 184ms plymouth-switch-root.service 170ms sysroot.mount 166ms sys-fs-fuse-connections.mount 159ms systemd-user-sessions.service 151ms kmod-static-nodes.service 112ms systemd-rfkill.service 98ms plymouth-quit.service 97ms plymouth-read-write.service 85ms systemd-vconsole-setup.service 76ms rtkit-daemon.service 60ms dracut-shutdown.service 51ms check-battery.service 51ms systemd-update-utmp.service 21ms initrd-udevadm-cleanup-db.service 18ms sys-kernel-config.mount 16ms plymouth-quit-wait.service
Quite strange, btrfsmaintenance and lvm2-monitor enabled, although both are not used. Clearly most time is burned on display-manager and backup of the rpm-db (could this not be moved to a systemd-timer triggered event at a later point in time?)
Cheers Axel Most of the service you see there aren't they systemd timers that run because it was not running during their normal schedule. Did you try to reboot half an hour after this boot how it behaves ?
Ahrrr tests :-)
I can confirm that the openSUSE boot times on a Raspi 3 are horrendous for several reasons. I tested and tweaked the unstable tumbleweed images, so take it with precaution, Leap might behave different:
* It takes quite some time until even grub2 is loaded (u-boot uefi chainload), but it brings a full grub2 with options to load different kernels or even distributions. (So for me this is worth the wait)
With which u-boot did you test? I think there were some changes to the distroboot scripts which might add more device probing before we actually find and run the grub binary.
I did test with u-boot 2018.x, which comes with the latest Tumbleweed appliance, and uboot 2019.01, forked from hardware:boot [1], no difference.
We have a U-Boot timeout of 2 seconds, that adds time. Then U-Boot's USB initialization also takes a while. On top of that, we search for overlay DTs as well, which again may take a few ms.
Either way, feel free to identify big chunks here and optimize them :).
* It takes quite along time until the handover (uboot, uefi -> kernel) is done:
+ u-boot slow at reading the actual files (kernel+initrd)
You mean reading the grub binary. Grub is in charge to load kernel+initrd.
No i mean the linux kernel and initrd, i thought u-boot would read those files for grub2, anyway, reading those two takes ages for me. Using a tumbleweed image (and my slow sd card) i have a total boot time of around 1 min 45 sec _after_ hitting enter in grub, where 1min 15 sec is for reading those two files, the rest is for the kernel to boot to the multiuser login screen.
+ a not compressed kernel image (~24MB uncompressed vs ~8MB compressed)
+ a initrd which ships (even after first installation and "mkinitrd") several kiwi components
Which components do you refer to?
Ok, not several, but one only necessary for firstboot: dracut-kiwi-lib dracut-kiwi-oem-repart after removing those compressed initrd size goes down from ~14MB to ~8MB (of course a mkinitrd run inbetween)
* Always triggered backup-*.timer(s) and mandb-timer
So to cut boot times idid several things:
* remove all kiwi packages -> smaller initrd file
* disable the above mentioned triggers -> this might not be applicable on a intended stable system
* create and use a kernel with a compressed instead of an uncompressed kernel image [1].
I'm not sure if we should add this to the kernel build infrastructure, as it is a RPi3 thing. Thoughts?
It doesn't have to be an RPi3 thing. Grub can extract files before booting them. We could probably just start shipping Image.gz (or some other compression method) files instead of plain Image ones in kernel-default.rpm.
Obviously, this will need a bit of auxiliary adjustments as well. Someone will have to double-check that grub really does extract it before running it. There are also a few scripts that check for the file name and may not execute on an Image.gz file. Once all that is identifies, we should set respective dependency rules in the kernel rpm.
But overall, this would be a *big* improvement for everyone. It means we'd waste much less space on /boot too!
Its certainly not a Raspi thing, as shipping compressed kernel for other architectures is not either, it is all available upstream in the kernel. Concerning arm64, right now we can ship the uncompressed "Image" or the compressed "Image.gz". As noted in my first email i have gone ahead and created a kernel-default package which ships a compressed image [2]. For some reason, i did not find the time to investigate (yet), shipping Image.gz does not work out (no grub entry + no initrd), yet installing and renaming the Image.gz to Image (still compressed), works and boots fine on my raspi 3. [1]: https://build.opensuse.org/project/monitor/home:tobijk:basesystem [2] https://build.opensuse.org/package/show/home:tobijk:kernel/kernel-source Greetings, Tobias
Could you please open a FATE for this?
Alex -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
participants (3)
-
Alexander Graf -
Matthias Brugger -
Tobias Klausmann