http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c1 --- Comment #1 from Antonio Larrosa --- Created attachment 772559 --> http://bugzilla.opensuse.org/attachment.cgi?id=772559&action=edit systemd-analyze blame The output of systemd-analyze blame -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c3 Antonio Larrosa changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(alarrosa@suse.com | |) | --- Comment #3 from Antonio Larrosa --- Created attachment 774572 --> http://bugzilla.opensuse.org/attachment.cgi?id=774572&action=edit journalctl -b Sure, this is the output of journalctl -b -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c5 --- Comment #5 from Antonio Larrosa --- Thanks for the idea. I didn't know about those generators nor that they could be masked. I tried running "systemctl daemon-reload" but that returns inmediately, so I have to reboot each time I want to test the generators. After trying to mask a few of them, having no luck, masking a few more, having no luck, and then going one by one... I noticed something strange. I masked all generators and the delay was gone. The system booted up fine. Then I removed the mask for systemd-gpt-auto-generator (or said another way, I only enabled the systemd-gpt-auto-generator) and the delay returned. I masked it again and the delay was gone. Then I removed the mask for systemd-debug-generator and the delay returned again. I masked it again and the delay was fixed once more. Then I removed the mask for systemd-cryptsetup-generator and that one didn't influence the delay. The system boots up fine if only systemd-cryptsetup-generator is enabled. I removed the mask for systemd-hibernate-resume-generator (so only systemd-cryptsetup-generator and this one are enabled) and the delay returned... Does that behaviour say anything to you? I'll continue testing one by one and try to find out the set of generators that seem to introduce the delay if any of them is enabled, but the process is slow. I guess there's no way to get a shell at that point of the boot process, is it? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c9 --- Comment #9 from Antonio Larrosa --- Created attachment 774878 --> http://bugzilla.opensuse.org/attachment.cgi?id=774878&action=edit backtrace of gpt generator stuck waiting for available random data -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c11 --- Comment #11 from Franck Bui --- Talked to Antonio privately and it appears that gcrypt run in fips mode for some *unknown* reasons. This has the side effect to run some self tests on each program linked to libgcrypt (I think) and those self tests exhaust /dev/urandom somehow. The gcrypt fips mode is activated by the presence of /etc/gcrypt/fips_enabled file. And removing this file solves this issue as well as bug #1096050. So I guess the question is now how was this file created at all. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c13 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lnussel@suse.com Flags| |needinfo?(lnussel@suse.com) --- Comment #13 from Franck Bui --- Since this started happening after upgrading 42.3 to 15.0, I guess this file was created during the upgrade process. Ludwig any idea ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c15 --- Comment #15 from Dr. Werner Fink --- (In reply to Franck Bui from comment #10)

hum, what's this thing:

#8 0x00007ffff64dd751 in _gcry_random_selftest (report=report@entry=0x7ffff642b2a0 <reporter>) at random.c:581 #9 0x00007ffff642c1ea in run_random_selftests () at fips.c:589 #10 _gcry_fips_run_selftests (extended=extended@entry=0) at fips.c:736

Werner any idea ?

Ahh .. yes, the FIPS tests do drain the current entropy of the system. This is a known problem of FIPS I was not aware that FIPS had been enabled on Leap 15. Also I've never understood why those FIPS tests done at first usage of FIPS do drain the entropy of a system without restoring or refilling the entropy as not existin entropy make the system unusable ... no program can use real random numbers anymore upto the point where the entropy gets filled again. Now on switch root every program wich had not been rebased to the new root (by using chroot(2)) and protected with a leading `@' byte in its argv[0] will be killed by systemd and/or do removed name space. To get this final solved the kernel has to collect entropy from any source it can use in initrd, e.g. using the jitter on the CPUs https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c17 --- Comment #17 from Antonio Larrosa --- Just (In reply to Franck Bui from comment #13)

Since this started happening after upgrading 42.3 to 15.0, I guess this file was created during the upgrade process.

Yes, the problems (both this and the sudo issue) started happening after upgrading to 15.0, but note that it seems the file wasn't created during the upgrade process. At least, it has an mtime of 2017-10-16 at 13:42:00 and the upgrade was done on 2018-06-03. I checked old zypper logs and it seems on 2017-10-16 I ran "zypper in libgcrypt20-hmac" at 13:41:42 for some reason I can't remember. So it matches with probably the %post script of the package creating it, can it be? At least, the package description mentions FIPS. Just in case it matters, libgcrypt20-hmac-1.6.1-39.1 was the package installed at that time, from the openSUSE 42.3 Update repository, while I currently have libgcrypt20-hmac-1.8.2-lp150.4.2 . -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c19 --- Comment #19 from Dr. Werner Fink --- Found this in my current Tumbleweed zgrep -i jitter /proc/config.gz CONFIG_CRYPTO_JITTERENTROPY=y hmm ... is this also available on Leap 15 / SLES 15 ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c22 --- Comment #22 from Dr. Werner Fink --- I'm not able to install the rng-tools for Tumbleweed ... the source pacakge seems to be there but the package is not installable ... also this package seems to miss a dracut part to be usable in initrd -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c24 --- Comment #24 from Dr. Werner Fink --- (In reply to Franck Bui from comment #23)

(In reply to Dr. Werner Fink from comment #18).

...

I don't have a /etc/gcrypt/fips_enabled nor /etc/gcrypt/ around here ... to which package does those belogn to?

Perhaps /proc/sys/crypto/fips_enabled exists and contains a value different from 0 ?

Otherwise your issue is probably different.

noether:~ # cat /proc/sys/crypto/fips_enabled cat: /proc/sys/crypto/fips_enabled: No such file or directory noether:~ # cat /proc/sys/crypto/ cat: /proc/sys/crypto/: No such file or directory -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c26 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(pmonrealgonzalez@ | |suse.com) | --- Comment #26 from Franck Bui --- Ok I'm re-assigning this bug to Pedro as he's the maintainer of libgcrypt. So basically there're 2 issues here: - FIPS mode has been unexpectedly activated (via the creation of /etc/gcrypt/fips_enabled). Would be interesting to figure out how IMHO. - When the FIPS mode is activated, it exhausts /dev/urandom pool which may lead to such issues during the boot process. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c27 --- Comment #27 from Dr. Werner Fink --- rpm -qf /etc/gcrypt/fips_enabled -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c29 --- Comment #29 from Franck Bui --- (In reply to Dr. Werner Fink from comment #28)

IMHO this may lead to similar problems as I guess that fips test might also drain the entropy on a SLES 15

Agreed although in practice the HWs targeted by SLES might have their entropy pool initialized before it is needed. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c32 Dr. Werner Fink changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(alarrosa@suse.com | |) --- Comment #32 from Dr. Werner Fink --- As the OBS has finished the build my hacked version of haveged for Leap 15.0 x86_64 I'd like to ask if this makes a difference. For this the binaries can be downloaded by osc getbinaries home:WernerFink:branches:security/haveged openSUSE_Leap_15.0 x86_64 haveged-1.9.2-lp150.134.1.x86_64.rpm and installed with sudo rpm -Uhv binaries/haveged-1.9.2-lp150.134.1.x86_64.rpm sudo mkinitrd the library package is not required as the change is only in the daemon its self. Does this version of haveged makes a difference at (re)boot? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c33 Antonio Larrosa changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(alarrosa@suse.com | |) | --- Comment #33 from Antonio Larrosa --- I enabled fips again and rebooted to be sure the problem was reproduced again. Then I installed the haveged package from your home, ran mkinitrd and rebooted. With your package, I can confirm the boot delay doesn't happen anymore. Btw, the problem with sudo (bsc#1096050) still continues happening as I guess that's related to fips being enabled and not the the entropy pool being empty. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 http://bugzilla.opensuse.org/show_bug.cgi?id=1096072#c35 --- Comment #35 from Dr. Werner Fink --- (In reply to Antonio Larrosa from comment #33)

Btw, the problem with sudo (bsc#1096050) still continues happening as I guess that's related to fips being enabled and not the the entropy pool being empty.

Then bsc#1096050 is not duplicate, isn't it? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1096072 Dr. Werner Fink changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |systemd-maintainers@suse.de -- You are receiving this mail because: You are on the CC list for the bug.