http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c2 --- Comment #2 from Stefan Dirsch --- Created attachment 744337 --> http://bugzilla.suse.com/attachment.cgi?id=744337&action=edit rdsosreport.txt -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 Stefan Dirsch changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fbui@suse.com Assignee|bnc-team-screening@forge.pr |systemd-maintainers@suse.de |ovo.novell.com | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c8 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS --- Comment #8 from Franck Bui --- Hi Stefan, (In reply to Stefan Dirsch from comment #0)

After updating systemd, the system no longer asks for the LVM encryption password during boot. Instead it runs in a timeout and an emergency shell. :-(

I made sure, that neither a kernel update nor a dracut update is responsible for this. Both updates regenerate the initrds and I rebooted afterwards successfully.

Hm the update of systemd came with the following change: * Di Jul 11 2017 fbui@suse.com - Make sure dracut (if installed) will embed the new compat rule The new compat rules (as well as the compat generation number) must be embedded in the initramfs so make sure that the installed dracut supports it. This is implemented with the following conflict directive in the spec file: Conflicts: dracut < 044.1 So you're supposed to update dracut to its latest version as well, which is dracut-044.1-26.1.x86_64 but according to your logs your version is only at 044-26.1 ... How is this possible ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c10 --- Comment #10 from Stefan Dirsch --- Created attachment 745343 --> http://bugzilla.suse.com/attachment.cgi?id=745343&action=edit Photo -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c12 --- Comment #12 from Stefan Dirsch --- (In reply to Stefan Dirsch from comment #11)

Created attachment 745344 [details] rdsosreport.txt

Again this looks like 044-26.1 instead of 044.1-26.1, but I double checked after the online update that dracut is on 0.44.1 level with latest changelog entry ------------------------------------------------------------------- Tue Aug 29 13:46:38 UTC 2017 - daniel.molkentin@suse.com - Don't detect crc32.ko as built-in (bsc#1054538) * adds 0537-dracut-init.sh-ignore-crc32.ko-in-builtin-test.patch - Enable systemd-based core dumps for initrd (bsc#1054809) * adds 0538-Enable-core-dumps-with-systemd-from-initrd.patch Also I even manually ran mkinitrd afterwards. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c14 --- Comment #14 from Stefan Dirsch --- Created attachment 745346 --> http://bugzilla.suse.com/attachment.cgi?id=745346&action=edit journalctl.log -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c16 Stefan Dirsch changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sndirsch@suse.com Flags| |needinfo?(sndirsch@suse.com | |) --- Comment #16 from Stefan Dirsch --- (In reply to Franck Bui from comment #15)

sorry for asking again but could you show the same debug logs but with a working systemd ?

Sure, can do this on monday. Be aware, that this means reinstalling the system. So always takes some time. Actually, this is the bugreport (+ the data loss you have, when you've observed this bug the first time). ;-) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c18 --- Comment #18 from Stefan Dirsch --- Created attachment 745451 --> http://bugzilla.suse.com/attachment.cgi?id=745451&action=edit /etc/crypttab -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c20 Stefan Dirsch changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(sndirsch@suse.com | |) | --- Comment #20 from Stefan Dirsch --- Created attachment 745493 --> http://bugzilla.suse.com/attachment.cgi?id=745493&action=edit journalctl.log.gz journalctl with debug options enabled (with still working systemd) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c22 --- Comment #22 from Stefan Dirsch --- (In reply to Franck Bui from comment #21)

Thanks.

Just to make sure the installation was done from the Leap 42.3 ISO, right ?

Yes, yes, yes. ISO written on a USB-stick.

IOW udev-228-25.6.1.x86_64 was the version used during the installation until you updated and broke your system.

In the reinstalled system I see udev-228-27.2.x86_64 (without any Online updates)

So basically during the installation, /dev/disk/by-id/cr_-LENSE20512GMSP34MEAT2TA_1142267006586-part2 was used for initializing /etc/crypttab.

No, there is no /dev/disk/by-id/cr_-LENSE20512GMSP34MEAT2TA_1142267006586-part2, it is /dev/disk/by-id/-LENSE20512GMSP34MEAT2TA_1142267006586-part2 (without cr_).

If so, I currently don't see how this could have happened and I would be interested in seeing the output of "udevadm test /block/nvme0n1p2". Also "udevadm info -e" might help.

Before I do this wrong. Should I do this after breaking system, i.e. in the emergency shell or in the working system? Currently I have a working system (before online update).

I'm afraid you need to reinstall your system, sorry.

This is no problem, if this can be of any help. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c24 --- Comment #24 from Franck Bui --- (In reply to Stefan Dirsch from comment #22)

Before I do this wrong. Should I do this after breaking system, i.e. in the emergency shell or in the working system? Currently I have a working system (before online update).

in the working system please ! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c26 --- Comment #26 from Stefan Dirsch --- Created attachment 745518 --> http://bugzilla.suse.com/attachment.cgi?id=745518&action=edit udevadm_test_block_nvme0n1_nvme0n1p2.txt udevadm test /block/nvme0n1/nvme0n1p2 (in working system) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c28 --- Comment #28 from Franck Bui --- Thanks Stefan. So apparently in the following rules:

KERNEL=="nvme*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}!="?*", IMPORT{program}="scsi_id --export --whitelisted -d $tempnode", ENV{ID_BUS}="nvme" KERNEL=="sd*|sr*|cciss*|nvme*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="?*", SYMLINK+="disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}" KERNEL=="sd*|cciss*|nvme*", ENV{DEVTYPE}=="partition", ENV{ID_SERIAL}=="?*", SYMLINK+="disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}-part%n"

ID_BUS is empty although it should be set to "nvme" by the first rule. I don't see how this can happen... Could you show the output of "/usr/lib/udev/scsi_id --export --whitelisted -d /dev/nvme0n1" ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c30 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lnussel@suse.com Flags| |needinfo?(lnussel@suse.com) --- Comment #30 from Franck Bui --- Thanks Stefan. Actually it was easy: in 60-persistent-storage.rules line 23:

KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ATTRS{model}=="?*", ENV{ID_SERIAL_SHORT}=="?*", ENV{ID_SERIAL}="$attr{model}_$env{ID_SERIAL_SHORT}", ...

so 'ID_SERIAL' is set and therefore rule line 46 is skipped.

KERNEL=="nvme*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}!="?*", IMPORT{program}="scsi_id --export --whitelisted -d $tempnode", ENV{ID_BUS}="nvme"

and that explains why brand new systems still continue to use such broken by-id symlinks (where the "nvme-" prefix is missing. Ludwig, couldn't we make the latest version of systemd/udev part of the ISO instead of shipping a broken version ? Don't the ISOs regularly updated ? Thanks. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c32 --- Comment #32 from Stefan Dirsch --- Wow! Indeed this change fixes the issue for me! Thanks a lot! About updating the ISO. Sure it would be appreciated. Unfortunately we've given out promotion DVDs, which we obviously cannot update. Also ISOs are meanwhile on media, which we do not control. Since this issue means complete data loss for the customer (I was happy that I've updated my system that early), I would like to know if you can think of any workaround, which the customer can apply in the emergency shell (or by other means). Then we could describe this below "Known issues for Leap 42.3". And now I'm wondering, whether we see this issue also on sle12-sp3. I haven't tested this yet. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c34 --- Comment #34 from Franck Bui --- (In reply to Stefan Dirsch from comment #32)

Wow! Indeed this change fixes the issue for me! Thanks a lot!

Thanks a lot for helping me debugging this.

About updating the ISO. Sure it would be appreciated. Unfortunately we've given out promotion DVDs, which we obviously cannot update. Also ISOs are meanwhile on media, which we do not control.

Indeed but at least we could limit the number of impacted users.

Since this issue means complete data loss for the customer (I was happy that I've updated my system that early), I would like to know if you can think of any workaround, which the customer can apply in the emergency shell (or by other means). Then we could describe this below "Known issues for Leap 42.3".

Maybe from the emergency shell: /etc/crypptab may be fixed to use the correct symlink, then "systemctl daemon-reload" and then exit from the shell. Could you try that ?

And now I'm wondering, whether we see this issue also on sle12-sp3. I haven't tested this yet.

It depends on the version of udev shipped by the ISO... do you know where I could find such information ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 Stefan Dirsch changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aschnell@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c37 --- Comment #37 from Stefan Dirsch --- (In reply to Stefan Dirsch from comment #36)

I could successfully change /etc/crypttab from emergency. Unfortunately this alone didn't help. Most likely one also needs to run dracut/mkinitrd to recreate initrd.

Tried that by using cryptsetup, lvm, mount, etc. but failed in the end. I'm afraid that's something, we cannot describe below "Known issues". We should really provide another systemd update, which fixes this issue. Seriously. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c39 --- Comment #39 from Franck Bui --- (In reply to Stefan Dirsch from comment #37)

Tried that by using cryptsetup, lvm, mount, etc. but failed in the end. I'm afraid that's something, we cannot describe below "Known issues". We should really provide another systemd update, which fixes this issue. Seriously.

hmm this sounds like you tried to update /etc/cryptsetup located in your rootfs. If so I was suggesting to update the one in the initramfs which should be directly accessible at /etc/crypttab from the emergency shell. The recipe is quite simple: - edit /etc/crypttab and and the "nvme" prefix if it's missing. you can figure out it's missing if the by-id path starts with '-' - systemctl daemon-reload - exit from the emergency shell If that works you still need to update /etc/crypttab in your rootfs and run "mkinitrd". -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c41 --- Comment #41 from Stefan Dirsch --- (In reply to Franck Bui from comment #39)

hmm this sounds like you tried to update /etc/cryptsetup located in your rootfs.

That's correct.

If so I was suggesting to update the one in the initramfs which should be directly accessible at /etc/crypttab from the emergency shell.

The recipe is quite simple:

- edit /etc/crypttab and and the "nvme" prefix if it's missing. you can figure out it's missing if the by-id path starts with '-'

- systemctl daemon-reload

- exit from the emergency shell

Did this. Then I see two lines dracut-initqueue Warning: Not all disks have been found. dracut-initqueue Warning: You might want to regenerate the initramfs Nothing more happens then. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c43 --- Comment #43 from Franck Bui --- (In reply to Stefan Dirsch from comment #41)

dracut-initqueue Warning: Not all disks have been found. dracut-initqueue Warning: You might want to regenerate the initramfs

Nothing more happens then.

Hmm you probably need to start a couple services by hand before exiting the shell then... Maybe "systemctl restart cryptsetup.target" ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c45 --- Comment #45 from Franck Bui --- BTW, AFAIK only leap 42.3 is affected (not SLE12). The fix will be released but it would be really nice if we could update the version of udev shipped by the ISO. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c47 Ludwig Nussel changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(lnussel@suse.com) | --- Comment #47 from Ludwig Nussel --- We cannot really update the ISO at this point. The issue needs to be documented in the release notes https://github.com/openSUSE/release-notes-openSUSE/tree/Leap_42.3 Feel free to file a pull request yourself or open bug here for the release notes component. Would it help to install with online updates? If so documenting that would also help. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c50 --- Comment #50 from Stefan Dirsch --- (In reply to Franck Bui from comment #49)

*** Bug 1060226 has been marked as a duplicate of this bug. ***

OMG! This issue is known since 2017-09-25 and the culprit found just a few days later by Jonathan Cottrill . And the bugowner of that bug is in Cc of this bug ... (JFYI, the workaround in this bugreport is not really a workaround; it can only be applied after a new installation, i.e. after you had the complete data loss) Ludwig, I'm not sure what another entry in our release notes would help here. One of the systemd update was the culprit here. What we need here is another systemd Online update. ASAP. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c53 --- Comment #53 from Franck Bui --- OK I've submitted a new update which contains the fix. I still need to document this shortcoming in the release notes so Leap 42.3 users are aware that /etc/crypttab uses a boggus symlink and therefore the file should be updated. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c56 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nrochard@gmail.com --- Comment #56 from Franck Bui --- *** Bug 1060156 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c60 --- Comment #60 from Franck Bui --- BTW I think I'll add a sanity check in systemd package to detect if the broken symlink is in used in /etc/crypttab and warns if it's the case. We'll also ask to replace it before upgrading the system so hopefully once the support for 42.3 will be ended, we could get rid of it. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Whiteboard| |ibs:running:5978:moderate -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 Marcus Meissner changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(fbui@suse.com) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Whiteboard|ibs:running:5978:moderate | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c67 --- Comment #67 from Swamp Workflow Management --- SUSE-RU-2017:3163-1: An update that has 11 recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1004995,1035386,1039099,1040800,1045472,1048605,1050152,1053137,1053595,1055641,1063249 CVE References: Sources used: SUSE Linux Enterprise Software Development Kit 12-SP3 (src): systemd-228-150.22.1 SUSE Linux Enterprise Software Development Kit 12-SP2 (src): systemd-228-150.22.1 SUSE Linux Enterprise Server for Raspberry Pi 12-SP2 (src): systemd-228-150.22.1 SUSE Linux Enterprise Server 12-SP3 (src): systemd-228-150.22.1 SUSE Linux Enterprise Server 12-SP2 (src): systemd-228-150.22.1 SUSE Linux Enterprise Desktop 12-SP3 (src): systemd-228-150.22.1 SUSE Linux Enterprise Desktop 12-SP2 (src): systemd-228-150.22.1 SUSE Container as a Service Platform ALL (src): systemd-228-150.22.1 OpenStack Cloud Magnum Orchestration 7 (src): systemd-228-150.22.1 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 Swamp Workflow Management changed: What |Removed |Added ---------------------------------------------------------------------------- Whiteboard|obs:running:7545:moderate |ibs:running:6053:moderate |ibs:running:6053:moderate | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c69 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Status|IN_PROGRESS |RESOLVED Resolution|--- |FIXED --- Comment #69 from Franck Bui --- The fix has been released, it's time to close this bug. Thanks all for helping me sort this out. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1063249 http://bugzilla.suse.com/show_bug.cgi?id=1063249#c70 --- Comment #70 from Swamp Workflow Management --- SUSE-SU-2018:0053-1: An update that solves 29 vulnerabilities and has 57 fixes is now available. Category: security (moderate) Bug References: 1003846,1004995,1009966,1022404,1025282,1025891,1026567,1029907,1029908,1029909,1029995,1030623,1035386,1036619,1039099,1039276,1039513,1040800,1040968,1041090,1043059,1043590,1043883,1043966,1044016,1045472,1045522,1045732,1047178,1047233,1048605,1048861,1050152,1050258,1050487,1052503,1052507,1052509,1052511,1052514,1052518,1053137,1053347,1053595,1053671,1055446,1055641,1055825,1056058,1056312,1056381,1057007,1057139,1057144,1057149,1057188,1057634,1057721,1057724,1058480,1058695,1058783,1059050,1059065,1059075,1059292,1059723,1060599,1060621,1061241,1061384,1062561,1063249,1063269,1064571,1064999,1065363,1066242,1066371,1066500,1066611,1067891,1070878,1070958,1071905,1071906 CVE References: CVE-2014-3710,CVE-2014-8116,CVE-2014-8117,CVE-2014-9620,CVE-2014-9621,CVE-2014-9653,CVE-2017-12448,CVE-2017-12450,CVE-2017-12452,CVE-2017-12453,CVE-2017-12454,CVE-2017-12456,CVE-2017-12799,CVE-2017-12837,CVE-2017-12883,CVE-2017-13757,CVE-2017-14128,CVE-2017-14129,CVE-2017-14130,CVE-2017-14333,CVE-2017-14529,CVE-2017-14729,CVE-2017-14745,CVE-2017-14974,CVE-2017-3735,CVE-2017-3736,CVE-2017-3737,CVE-2017-3738,CVE-2017-6512 Sources used: SUSE CaaS Platform ALL (src): sles12-caasp-dex-image-2.0.0-3.3.11, sles12-dnsmasq-nanny-image-2.0.1-2.3.15, sles12-haproxy-image-2.0.1-2.3.16, sles12-kubedns-image-2.0.1-2.3.11, sles12-mariadb-image-2.0.1-2.3.15, sles12-openldap-image-2.0.0-2.3.11, sles12-pause-image-2.0.1-2.3.9, sles12-pv-recycler-node-image-2.0.1-2.3.10, sles12-salt-api-image-2.0.1-2.3.10, sles12-salt-master-image-2.0.1-2.3.10, sles12-salt-minion-image-2.0.1-2.3.14, sles12-sidecar-image-2.0.1-2.3.11, sles12-tiller-image-2.0.0-2.3.11, sles12-velum-image-2.0.1-2.3.13 -- You are receiving this mail because: You are on the CC list for the bug.