http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c1 --- Comment #1 from andreas bittner --- I dug through the history file of zypp a bit: dpg-ppb-smu-tux01:/var/log/zypp # grep "systemd-" history 2018-05-25 21:30:28|install|systemd-32bit|228-50.1|x86_64||repo-update|d27b22c8259c737403600ad9d87f742807a8f54937ae3b315c65b0f6b5d3cbe4| 2018-05-25 21:30:28|install|systemd-bash-completion|228-50.1|noarch||repo-update|59c306156952bb2a4337ee8ce03634c3df91e859cfd1eebcc59f3606e292c395| 2018-05-25 21:30:35|install|systemd-sysvinit|228-50.1|x86_64||repo-update|4eefb782fba0b870e43ff2fb3075619e24ff802434c47e7c5b2799f24c5a2ac9| # dracut: *** Including module: systemd-initrd *** # dracut: *** Including module: systemd-initrd *** 2018-05-26 22:47:28|install|systemd-bash-completion|234-lp150.19.1|noarch||repo-oss|1617c4e81881daaf105771889e81800a224e4138c983dd5e36277964208eaf34| 2018-05-26 22:48:22|install|systemd-rpm-macros|3-lp150.5.20|noarch||repo-oss|48cc7e002620e75010516608301cd3d44527135939b6e0be6872616b9ffd0779| 2018-05-26 22:48:22|install|systemd-presets-common-SUSE|15-lp150.1.1|noarch||repo-oss|49a5fafc828e2d80520ddbb58a46fd47e12daa1767e4ab1fcd161be10c5531ca| 2018-05-26 22:48:33|install|systemd-presets-branding-openSUSE|12.2-lp150.12.1|noarch||repo-oss|4818e7f3f9cf8e919724c7cc290c48fcafd29ebf5c63e2384e448a98caac1ea3| # 2018-05-26 22:50:07 systemd-234-lp150.19.1.x86_64.rpm installed ok # 2018-05-26 22:51:48 util-linux-systemd-2.31.1-lp150.6.1.x86_64.rpm installed ok # 2018-05-26 22:51:48 systemd-sysvinit-234-lp150.19.1.x86_64.rpm installed ok 2018-05-26 22:51:48|install|systemd-sysvinit|234-lp150.19.1|x86_64||repo-oss|6b478bd245ff3ce947733dc94d3d956061f94accd5ca94a61d287687f7a3c06c| # Failed to try-restart systemd-udevd.service: Connection timed out # See system logs and 'systemctl status systemd-udevd.service' for details. The hinted command there in the error line doesnt help at all, systemd or some basic part of it is never reacting or I am unaware and too noob to be able to make it start over again or know how to talk to it: dpg-ppb-smu-tux01:~ # systemctl status systemd-udevd.service Failed to get properties: Activation of org.freedesktop.systemd1 timed out -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c3 --- Comment #3 from andreas bittner --- ... (1564/3403) Installing: samba-winbind-4.7.6+git.54.6e3276c9872-lp150.2.20.x86_64 ......................................................................................[done] Additional rpm output: Failed to reload daemon: Activation of org.freedesktop.systemd1 timed out Updating /etc/sysconfig/samba ... Failed to reload daemon: Activation of org.freedesktop.systemd1 timed out Failed to try-restart winbind.service: Activation of org.freedesktop.systemd1 timed out See system logs and 'systemctl status winbind.service' for details. ... (1570/3403) Installing: polkit-0.114-lp150.1.6.x86_64 .................................................................................................................[done] Additional rpm output: Failed to reload daemon: Activation of org.freedesktop.systemd1 timed out Failed to reload daemon: Activation of org.freedesktop.systemd1 timed out Failed to try-restart polkit.service: Connection timed out See system logs and 'systemctl status polkit.service' for details. ... (1578/3403) Installing: udisks2-2.6.5-lp150.1.13.x86_64 ...............................................................................................................[done] Additional rpm output: Failed to reload daemon: Activation of org.freedesktop.systemd1 timed out Failed to reload daemon: Activation of org.freedesktop.systemd1 timed out Failed to try-restart udisks2.service: Connection timed out See system logs and 'systemctl status udisks2.service' for details. .... as more examples, still ongoing.... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c5 --- Comment #5 from andreas bittner --- I am rarely doing this but how is a later bug and more importantly less extensive bug a duplicate of an earlier reported bug referencing all the relevant places and discussions. sigh. seriously. never mind so continue the bug over there. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c7 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |carlos.e.r@opensuse.org --- Comment #7 from Franck Bui --- *** Bug 1094771 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c9 --- Comment #9 from andreas bittner --- the affected system is still up and in the not-yet rebooted state. I can grab journalctl from around that timeframe: May 26 22:30:01 dpg-ppb-smu-tux01 cron[11597]: pam_unix(crond:session): session opened for user root by (uid=0) May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Created slice User Slice of root. -- Subject: Unit user-0.slice has finished start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit user-0.slice has finished starting up. -- -- The start-up result is done. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Starting User Manager for UID 0... -- Subject: Unit user@0.service has begun start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit user@0.service has begun starting up. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: pam_unix(systemd-user:session): session opened for user root by (uid=0) May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Started Session 920 of user root. -- Subject: Unit session-920.scope has finished start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit session-920.scope has finished starting up. -- -- The start-up result is done. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Reached target Timers. -- Subject: Unit UNIT has finished start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit user@0.service has begun shutting down. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Reached target Shutdown. -- Subject: Unit UNIT has finished start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit UNIT has finished starting up. -- -- The start-up result is done. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Default. -- Subject: Unit UNIT has finished shutting down -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit UNIT has finished shutting down. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Basic System. -- Subject: Unit UNIT has finished shutting down -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit UNIT has finished shutting down. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Timers. -- Subject: Unit UNIT has finished shutting down -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit UNIT has finished shutting down. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Paths. -- Subject: Unit UNIT has finished shutting down -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit UNIT has finished shutting down. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Sockets. -- Subject: Unit UNIT has finished shutting down -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit UNIT has finished shutting down. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Starting Exit the Session... -- Subject: Unit UNIT has begun start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit UNIT has begun starting up. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Received SIGRTMIN+24 from PID 11643 (kill). May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11600]: pam_unix(systemd-user:session): session closed for user root May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Stopped User Manager for UID 0. -- Subject: Unit user@0.service has finished shutting down .... Any more info needed? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c11 --- Comment #11 from andreas bittner --- some minutes later another entry: May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Starting Exit the Session... -- Subject: Unit UNIT has begun start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit UNIT has begun starting up. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Stopped target Sockets. -- Subject: Unit UNIT has finished shutting down -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit UNIT has finished shutting down. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Received SIGRTMIN+24 from PID 12397 (kill). May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12343]: pam_unix(systemd-user:session): session closed for user root May 26 22:45:01 dpg-ppb-smu-tux01 systemd[1]: Stopped User Manager for UID 0. -- Subject: Unit user@0.service has finished shutting down -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit user@0.service has finished shutting down. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[1]: Removed slice User Slice of root. -- Subject: Unit user-0.slice has finished shutting down -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit user-0.slice has finished shutting down. ... -- Unit user-0.slice has finished shutting down. May 26 22:45:37 dpg-ppb-smu-tux01 dns-resolver[12513]: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... May 26 22:45:37 dpg-ppb-smu-tux01 dns-resolver[12515]: You can find my version in /etc/resolv.conf.netconfig May 26 22:45:37 dpg-ppb-smu-tux01 wickedd[1185]: ni_process_reap: process 12419 has not exited yet; now doing a blocking waitpid() May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/icewm-session.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/icewm.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/kde-plasma.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/plasma5.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/twm.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/xsession.desktop" May 26 22:46:57 dpg-ppb-smu-tux01 systemd[1]: Reexecuting. May 26 22:46:57 dpg-ppb-smu-tux01 systemd[1]: systemd 228 running in system mode. (+PAM -AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS +> May 26 22:46:57 dpg-ppb-smu-tux01 systemd[1]: Detected architecture x86-64. May 26 22:46:57 dpg-ppb-smu-tux01 systemd[1]: nss-lookup.target: Dependency Before=nss-lookup.target dropped .... after that then lot of rpm detailed logs follow of all or many of the updated upgraded packages and their output.... ... May 26 23:12:24 dpg-ppb-smu-tux01 [RPM][16338]: Transaction ID 5b09cdb8 finished: 0 May 26 23:12:24 dpg-ppb-smu-tux01 [RPM][16341]: Transaction ID 5b09cdb8 started May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16341]: erase libsamplerate0-0.1.8-16.15.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16341]: install libsamplerate0-0.1.9-lp150.3.4.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16341]: erase libsamplerate0-0.1.8-16.15.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16341]: install libsamplerate0-0.1.9-lp150.3.4.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16341]: Transaction ID 5b09cdb8 finished: 0 May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16344]: Transaction ID 5b09cdb9 started May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16344]: install libruby2_5-2_5-2.5.0-lp150.2.11.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16344]: install libruby2_5-2_5-2.5.0-lp150.2.11.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16344]: Transaction ID 5b09cdb9 finished: 0 May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16346]: Transaction ID 5b09cdb9 started May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16346]: erase libreiserfscore0-3.6.24-10.3.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16346]: install libreiserfscore0-3.6.27-lp150.1.10.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16346]: erase libreiserfscore0-3.6.24-10.3.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16346]: install libreiserfscore0-3.6.27-lp150.1.10.x86_64: success May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16346]: Transaction ID 5b09cdb9 finished: 0 May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16349]: Transaction ID 5b09cdb9 started May 26 23:12:25 dpg-ppb-smu-tux01 [RPM][16349]: erase libreiserfs-0_3-0-0.3.0.5-171.3.x86_64: success May 26 23:12:26 dpg-ppb-smu-tux01 [RPM][16349]: install libreiserfs-0_3-0-0.3.0.5-lp150.1.14.x86_64: success May 26 23:12:26 dpg-ppb-smu-tux01 [RPM][16349]: erase libreiserfs-0_3-0-0.3.0.5-171.3.x86_64: success May 26 23:12:26 dpg-ppb-smu-tux01 [RPM][16349]: install libreiserfs-0_3-0-0.3.0.5-lp150.1.14.x86_64: success May 26 23:12:26 dpg-ppb-smu-tux01 [RPM][16349]: Transaction ID 5b09cdb9 finished: 0 May 26 23:12:26 dpg-ppb-smu-tux01 [RPM][16352]: Transaction ID 5b09cdba started May 26 23:12:26 dpg-ppb-smu-tux01 [RPM][16352]: erase libre2-0-20180301-15.1.x86_64: success .... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c13 --- Comment #13 from andreas bittner --- doing only -b and scrolling to that timestamp: ... May 26 22:29:39 dpg-ppb-smu-tux01 sshd[11590]: Disconnected from 121.18.238.115 port 39264 [preauth] May 26 22:30:01 dpg-ppb-smu-tux01 cron[11597]: pam_unix(crond:session): session opened for user root by (uid=0) May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Created slice User Slice of root. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Starting User Manager for UID 0... May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: pam_unix(systemd-user:session): session opened for user root by (uid=0) May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Started Session 920 of user root. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Reached target Timers. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Reached target Sockets. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Reached target Paths. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Reached target Basic System. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Reached target Default. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Startup finished in 22ms. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Started User Manager for UID 0. May 26 22:30:01 dpg-ppb-smu-tux01 CRON[11597]: pam_unix(crond:session): session closed for user root May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Stopping User Manager for UID 0... May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Reached target Shutdown. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Default. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Basic System. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Timers. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Paths. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Stopped target Sockets. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Starting Exit the Session... May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11598]: Received SIGRTMIN+24 from PID 11643 (kill). May 26 22:30:01 dpg-ppb-smu-tux01 systemd[11600]: pam_unix(systemd-user:session): session closed for user root May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Stopped User Manager for UID 0. May 26 22:30:01 dpg-ppb-smu-tux01 systemd[1]: Removed slice User Slice of root. May 26 22:30:13 dpg-ppb-smu-tux01 dns-resolver[11753]: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... May 26 22:30:13 dpg-ppb-smu-tux01 dns-resolver[11755]: You can find my version in /etc/resolv.conf.netconfig May 26 22:33:43 dpg-ppb-smu-tux01 squid[2826]: Logfile: opening log stdio:/var/log/squid/netdb.state May 26 22:33:43 dpg-ppb-smu-tux01 squid[2826]: Logfile: closing log stdio:/var/log/squid/netdb.state May 26 22:33:43 dpg-ppb-smu-tux01 squid[2826]: NETDB state saved; 174 entries, 1 msec May 26 22:33:54 dpg-ppb-smu-tux01 sshd[11916]: Received disconnect from 221.194.44.232 port 44321:11: [preauth] May 26 22:33:54 dpg-ppb-smu-tux01 sshd[11916]: Disconnected from 221.194.44.232 port 44321 [preauth] May 26 22:35:59 dpg-ppb-smu-tux01 dns-resolver[12051]: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... May 26 22:35:59 dpg-ppb-smu-tux01 dns-resolver[12053]: You can find my version in /etc/resolv.conf.netconfig May 26 22:35:59 dpg-ppb-smu-tux01 wickedd[1185]: ni_process_reap: process 11957 has not exited yet; now doing a blocking waitpid() May 26 22:38:08 dpg-ppb-smu-tux01 sshd[12200]: Received disconnect from 122.226.181.165 port 42238:11: [preauth] May 26 22:38:08 dpg-ppb-smu-tux01 sshd[12200]: Disconnected from 122.226.181.165 port 42238 [preauth] May 26 22:43:47 dpg-ppb-smu-tux01 dbus[1011]: [system] Activating service name='org.opensuse.Snapper' (using servicehelper) May 26 22:43:47 dpg-ppb-smu-tux01 dbus[1011]: [system] Successfully activated service 'org.opensuse.Snapper' May 26 22:45:01 dpg-ppb-smu-tux01 cron[12338]: pam_unix(crond:session): session opened for user root by (uid=0) May 26 22:45:01 dpg-ppb-smu-tux01 systemd[1]: Created slice User Slice of root. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[1]: Starting User Manager for UID 0... May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: pam_unix(systemd-user:session): session opened for user root by (uid=0) May 26 22:45:01 dpg-ppb-smu-tux01 systemd[1]: Started Session 921 of user root. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Reached target Paths. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Reached target Timers. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Reached target Sockets. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Reached target Basic System. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Reached target Default. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Startup finished in 19ms. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[1]: Started User Manager for UID 0. May 26 22:45:01 dpg-ppb-smu-tux01 run-crons[12385]: suse.de-snapper: OK May 26 22:45:01 dpg-ppb-smu-tux01 CRON[12338]: pam_unix(crond:session): session closed for user root May 26 22:45:01 dpg-ppb-smu-tux01 systemd[1]: Stopping User Manager for UID 0... May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Stopped target Default. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Stopped target Basic System. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Stopped target Timers. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Stopped target Paths. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Reached target Shutdown. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Starting Exit the Session... May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Stopped target Sockets. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12339]: Received SIGRTMIN+24 from PID 12397 (kill). May 26 22:45:01 dpg-ppb-smu-tux01 systemd[12343]: pam_unix(systemd-user:session): session closed for user root May 26 22:45:01 dpg-ppb-smu-tux01 systemd[1]: Stopped User Manager for UID 0. May 26 22:45:01 dpg-ppb-smu-tux01 systemd[1]: Removed slice User Slice of root. May 26 22:45:37 dpg-ppb-smu-tux01 dns-resolver[12513]: ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... May 26 22:45:37 dpg-ppb-smu-tux01 dns-resolver[12515]: You can find my version in /etc/resolv.conf.netconfig May 26 22:45:37 dpg-ppb-smu-tux01 wickedd[1185]: ni_process_reap: process 12419 has not exited yet; now doing a blocking waitpid() May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/icewm-session.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/icewm.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/kde-plasma.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/plasma5.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/twm.desktop" May 26 22:46:55 dpg-ppb-smu-tux01 sddm-greeter[21871]: Reading from "/usr/share/xsessions/xsession.desktop" ..... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c15 --- Comment #15 from Franck Bui --- Could you also please describe the steps you used to upgrade your system from 42.3 to 15.0 ? So I can try to reproduce quickly... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c17 --- Comment #17 from andreas bittner --- There were no really noteworthy situations or things on this system. a simple ssh to the machine (normal user), then su - to gain root, in there screen -d -R zypper dup --download-in-advance that was all. Also I have several machines, and only this select one was affected by this bug, but the mailing list discussion that is mentioned here in this bugreport speaks about even users having this problem from going leap 15 beta to leap 15 rtm. This machine has maybe come a longer way from previous opensuse releases, but that is all I can say, but then again other people said having this problem with even only having as little history as beta to rtm. Cant really say more nothing coming to my mind. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c19 --- Comment #19 from Carlos Robinson --- Created attachment 771606 --> http://bugzilla.opensuse.org/attachment.cgi?id=771606&action=edit sesion journal log, virtualbox machine -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c21 --- Comment #21 from Franck Bui --- (In reply to Carlos Robinson from comment #18)

If you wish, I can send you the entire VM. I have taken a snapshot while this was happening.

Yes please. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c23 --- Comment #23 from Carlos Robinson --- I have it uploaded on Google Drive. Now I need a gmail address of someone to pick it up from there. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c25 --- Comment #25 from Carlos Robinson --- But I'm not prepared to make it public. It is only for the person that works on this issue. So as soon as I know the gmail address, I will share the file with him, and tell the root and user password of the VM to him. I don't think there is any private information, but just in case. The .tgz is already uploaded to google drive and waiting. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c27 --- Comment #27 from Carlos Robinson --- Sorry, google drive sharing only works with G+ users. The correct method would be to upload the file to bugzilla and mark it private, but it is way too big. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c29 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mkoutny@suse.com Flags| |needinfo?(mkoutny@suse.com) --- Comment #29 from Franck Bui --- Michal perhaps you can spot something in the meantime. To sum up, right after the upgrade of systemd package, the dbus daemon is no more able to use 'org.freedesktop.systemd1' service:

May 27 11:39:24 linux-phe7 dbus-daemon[693]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

PID1 doesn't seem to have crashed. It might be worth noting that polkit and dbus are reloading their configuration while systemd is reexecuted (due to the package upgrade). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c31 --- Comment #31 from Carlos Robinson --- (In reply to Franck Bui from comment #28)

(In reply to Carlos Robinson from comment #27)

...

Sorry, google drive sharing only works with G+ users.

Ok, to make it simple I sent you my gmail address privately.

Ok, done. It is not the snapshot only, it is the whole virtual box machine directory. Disk image, machine definition, suspend data, one snapshot, logs: all in tar bz2 archive. The machine is suspended (Vbox halt with saved state) with zypper dup running and stalling. Unfortunately, I don't have a snapshot made *before* the upgrade :-( Perhaps a fresh install of the beta would exhibit the symptoms? I have the beta ISO still saved. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c33 --- Comment #33 from Carlos Robinson --- Well... I can create a new virtual machine, install the Beta, do a snapshot, try zypper dup, see if it stalls, and if it does, revert the snapshot. You mean that? It will take me some time to do it. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c35 --- Comment #35 from Carlos Robinson --- I have the initial ISO used to create that machine stored, but it is possible I did an intermediate zypper dup, too. I used openSUSE-Leap-15.0-DVD-x86_64-Build206.1-Media.iso, perhaps on 2018-04-24 zypper.log starts 2018-05-27. Either it rotated or this is the first zypper dup of this machine. This is good news. Confirm, I did no "zypper dup" of this particular machine during Beta phase. So I would need to recreate with similar choices from beta iso. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c37 --- Comment #37 from Franck Bui --- (In reply to Carlos Robinson from comment #36)

I could try to install minimal X instead, or Gnome, with network disabled. :-?

Yes you could try with a minimal installation (without Xorg) and see if you can still reproduce. If so I'll try to get the same ISO. Thanks. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c40 --- Comment #40 from andreas bittner --- browsing around a bit on: https://en.opensuse.org/SDB:Systemd I wonder what settings I really need to set into /etc/systemd/system.conf LogLevel=debug LogTarget=syslog-or-kmsg SysVConsole=yes I understand the LogLevel line, but the defaults in that file apparently (uncommented for reference) show: #LogTarget=journal-or-kmsg and I read nothing about SysVConsole so I wonder what I should really put there and when to revert back. Also it is not even clear yet if this system will be affected by this bug or not at all. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 Tomáš Chvátal changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bnc-team-screening@forge.pr |systemd-maintainers@suse.de |ovo.novell.com | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c43 --- Comment #43 from andreas bittner --- If you give me detailed instructions on how to set loglevels that will help you, I have still that one unupgraded machine here, I need to do something about it. I dont know though if the machine will be affected though. What comes to my mind maybe is, is it possible that non-rebooted machines that e.g. had updates on 42.3 level (systemd and other stuff lately, not yet replaced and activated all files in use, e.g. non rebooted yet) and then going for a zypper dup with repositories swapped over to 15.0 URL addresses maybe can cause this bug? just a gut feeling, maybe I am totally wrong. So pending systemd (and the processes you speak about) updates and then in addition to that zypper dup to 15.0 and then maybe restarts of services fail differently than on a cleanly booted 42.3 system with nothing pending? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c45 --- Comment #45 from Franck Bui --- (In reply to Carlos Robinson from comment #31)

Unfortunately, I don't have a snapshot made *before* the upgrade :-(

Unfortunately it's going to be hard to find out what was going on if we are not able to reproduce. I'm afraid that trying to do so with the debug logs enabled will change the timings and will prevent the crash to happen. The only idea I have currently would be to reproduce with the kernel coredump handling enabled (echo core >/proc/sys/kernel/core_pattern). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c47 --- Comment #47 from Franck Bui --- (In reply to andreas bittner from comment #46)

Do i need to restart systemd for the debug level to take effect or anything?

You needn't. But beware that the journal is not persistent by default on Leap. That means that if for any reason you need to reboot, the journal contents from the previous boot will be lost. You can still make the journal peristent by doing: 1. mkdir /var/log/journal 2. systemctl restart systemd-journald.service -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c49 --- Comment #49 from andreas bittner --- that one file... is a directory ;) there is plenty of stuff in there... we shall see. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c51 --- Comment #51 from andreas bittner --- this is currently on another screen page, journalctl -b -f output: May 30 15:19:54 schreinux [RPM][19573]: erase util-linux-systemd-2.29.2-5.1.x86_64: success May 30 15:20:19 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:20:19 schreinux [RPM][19573]: install util-linux-systemd-2.31.1-lp150.6.1.x86_64: success May 30 15:20:44 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:21:09 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:21:34 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:21:34 schreinux [RPM][19573]: erase util-linux-systemd-2.29.2-5.1.x86_64: success May 30 15:21:34 schreinux [RPM][19573]: install util-linux-systemd-2.31.1-lp150.6.1.x86_64: success May 30 15:21:34 schreinux [RPM][19573]: Transaction ID 5b0ea4fa finished: 0 May 30 15:21:34 schreinux [RPM][19585]: Transaction ID 5b0ea55e started May 30 15:21:34 schreinux [RPM][19585]: erase systemd-logger-228-50.1.x86_64: success May 30 15:21:35 schreinux [RPM][19585]: install systemd-logger-234-lp150.19.1.x86_64: success May 30 15:21:35 schreinux [RPM][19585]: erase systemd-logger-228-50.1.x86_64: success May 30 15:21:35 schreinux [RPM][19585]: install systemd-logger-234-lp150.19.1.x86_64: success May 30 15:21:35 schreinux [RPM][19585]: Transaction ID 5b0ea55e finished: 0 May 30 15:21:35 schreinux [RPM][19588]: Transaction ID 5b0ea55f started May 30 15:21:35 schreinux [RPM][19588]: erase libwicked-0-6-0.6.40-9.1.x86_64: success May 30 15:21:35 schreinux [RPM][19588]: install libwicked-0-6-0.6.47-lp150.1.1.x86_64: success May 30 15:21:35 schreinux [RPM][19588]: erase libwicked-0-6-0.6.40-9.1.x86_64: success May 30 15:21:35 schreinux [RPM][19588]: install libwicked-0-6-0.6.47-lp150.1.1.x86_64: success May 30 15:21:35 schreinux [RPM][19588]: Transaction ID 5b0ea55f finished: 0 May 30 15:21:35 schreinux [RPM][19593]: Transaction ID 5b0ea55f started May 30 15:21:35 schreinux [RPM][19593]: erase wicked-0.6.40-9.1.x86_64: success May 30 15:21:35 schreinux dbus[923]: [system] Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:21:35 schreinux dbus[923]: [system] Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:21:35 schreinux dbus-daemon[923]: Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:21:35 schreinux dbus-daemon[923]: Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:21:59 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:21:59 schreinux [RPM][19593]: install wicked-0.6.47-lp150.1.1.x86_64: success May 30 15:21:59 schreinux [RPM][19593]: erase wicked-0.6.40-9.1.x86_64: success May 30 15:21:59 schreinux [RPM][19593]: install wicked-0.6.47-lp150.1.1.x86_64: success May 30 15:21:59 schreinux [RPM][19593]: Transaction ID 5b0ea55f finished: 0 May 30 15:21:59 schreinux [RPM][19604]: Transaction ID 5b0ea577 started May 30 15:21:59 schreinux [RPM][19604]: erase wicked-service-0.6.40-9.1.x86_64: success May 30 15:22:24 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:22:49 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:19:30 schreinux [RPM][19526]: install systemd-234-lp150.19.1.x86_64: success May 30 15:19:31 schreinux dbus[923]: [system] Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:19:31 schreinux dbus[923]: [system] Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:19:31 schreinux dbus-daemon[923]: Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:19:31 schreinux dbus-daemon[923]: Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:19:31 schreinux dbus[923]: [system] Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:19:31 schreinux dbus[923]: [system] Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:19:31 schreinux dbus-daemon[923]: Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:19:31 schreinux dbus-daemon[923]: Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:19:54 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:19:54 schreinux [RPM][19526]: erase systemd-228-50.1.x86_64: success May 30 15:19:54 schreinux [RPM][19526]: install systemd-234-lp150.19.1.x86_64: success May 30 15:19:54 schreinux [RPM][19526]: Transaction ID 5b0ea4e0 finished: 0 May 30 15:19:54 schreinux [RPM][19573]: Transaction ID 5b0ea4fa started May 30 15:19:54 schreinux [RPM][19573]: erase util-linux-systemd-2.29.2-5.1.x86_64: success May 30 15:20:19 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:20:19 schreinux [RPM][19573]: install util-linux-systemd-2.31.1-lp150.6.1.x86_64: success May 30 15:20:44 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:21:09 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:21:34 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:21:34 schreinux [RPM][19573]: erase util-linux-systemd-2.29.2-5.1.x86_64: success May 30 15:21:34 schreinux [RPM][19573]: install util-linux-systemd-2.31.1-lp150.6.1.x86_64: success May 30 15:21:34 schreinux [RPM][19573]: Transaction ID 5b0ea4fa finished: 0 May 30 15:21:34 schreinux [RPM][19585]: Transaction ID 5b0ea55e started May 30 15:21:34 schreinux [RPM][19585]: erase systemd-logger-228-50.1.x86_64: success May 30 15:21:35 schreinux [RPM][19585]: install systemd-logger-234-lp150.19.1.x86_64: success May 30 15:21:35 schreinux [RPM][19585]: erase systemd-logger-228-50.1.x86_64: success May 30 15:21:35 schreinux [RPM][19585]: install systemd-logger-234-lp150.19.1.x86_64: success May 30 15:21:35 schreinux [RPM][19585]: Transaction ID 5b0ea55e finished: 0 May 30 15:21:35 schreinux [RPM][19588]: Transaction ID 5b0ea55f started May 30 15:21:35 schreinux [RPM][19588]: erase libwicked-0-6-0.6.40-9.1.x86_64: success May 30 15:21:35 schreinux [RPM][19588]: install libwicked-0-6-0.6.47-lp150.1.1.x86_64: success May 30 15:21:35 schreinux [RPM][19588]: erase libwicked-0-6-0.6.40-9.1.x86_64: success May 30 15:21:35 schreinux [RPM][19588]: install libwicked-0-6-0.6.47-lp150.1.1.x86_64: success May 30 15:21:35 schreinux [RPM][19588]: Transaction ID 5b0ea55f finished: 0 May 30 15:21:35 schreinux [RPM][19593]: Transaction ID 5b0ea55f started May 30 15:21:35 schreinux [RPM][19593]: erase wicked-0.6.40-9.1.x86_64: success May 30 15:21:35 schreinux dbus[923]: [system] Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:21:35 schreinux dbus[923]: [system] Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:21:35 schreinux dbus-daemon[923]: Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:21:35 schreinux dbus-daemon[923]: Unable to reload configuration: Configuration file needs one or more <listen> elements giving addresses May 30 15:21:59 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:21:59 schreinux [RPM][19593]: install wicked-0.6.47-lp150.1.1.x86_64: success May 30 15:21:59 schreinux [RPM][19593]: erase wicked-0.6.40-9.1.x86_64: success May 30 15:21:59 schreinux [RPM][19593]: install wicked-0.6.47-lp150.1.1.x86_64: success May 30 15:21:59 schreinux [RPM][19593]: Transaction ID 5b0ea55f finished: 0 May 30 15:21:59 schreinux [RPM][19604]: Transaction ID 5b0ea577 started May 30 15:21:59 schreinux [RPM][19604]: erase wicked-service-0.6.40-9.1.x86_64: success May 30 15:22:24 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:22:49 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:23:14 schreinux dbus[923]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out May 30 15:23:14 schreinux [RPM][19604]: install wicked-service-0.6.47-lp150.1.1.x86_64: success -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c53 --- Comment #53 from andreas bittner --- only the systemd debug stuff you initially wrote, have seen that core dump variable too late. but isnt the journalctl -b output I pastes the stuff you wanted to see? isnt there anything in there? which lines do you need or are those lines I copy pasted only after the crash of systemd? what file or lines do I submit now? zypper dup is at 598/3080 and it will take a long while now with those lot of timeouts -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c55 --- Comment #55 from Franck Bui --- (In reply to andreas bittner from comment #53)

only the systemd debug stuff you initially wrote, have seen that core dump variable too late.

Bad luck...

but isnt the journalctl -b output I pastes the stuff you wanted to see? isnt there anything in there?

It just seems that there're no debug logs in there. Could you attach the output of "journalctl -b" ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c58 --- Comment #58 from andreas bittner --- Michael: how would I find out about this? These machines are pretty simple setups. I dont know much about snapshots. Does snapper or btrfs come into this somehow as these machines are only ext4 filesystems if I recally. If you tell me how I can find out if some snapshot might exist, I can check. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c60 --- Comment #60 from Franck Bui --- (In reply to andreas bittner from comment #56)

Created attachment 771834 [details] journalctl -b output while still running zypper dup, systemd trouble around 15:17+ timestamp onward

journalctl -b output while still running zypper dup, systemd package became installed and then troubled around 2018-05-30 15:17+ timestamp onward methinks.

Sadly the log level is not kept/saved by PID1 when it is re-executed. IOW your logs don't provide more info than the ones already provided. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c63 --- Comment #63 from andreas bittner --- Something I thought about: if btrfs is default and clean install if btrfs is used inside that virtualmachine image wouldnt you be able to make use of that kernel variable and systemd debug stuff you asked for finding the desired crash reports? then again: maybe this bug only happens when not using btrfs? i have never used btrfs in production. most my machines are in use over several generations of opensuse versions. maybe i can still do a clean 42.2, then dup it to 42.3 and dup it to 15.0 and see if i will eventually find this bug happening again all using btrfs from scratch 42.2 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c67 --- Comment #67 from Franck Bui --- (In reply to andreas bittner from comment #65)

i am doing the zypper dup from 42.2 to 42.3 atm, will only zypper up 42.3 afterwards and stop there, awaiting some more hints or requests i could help with for this bug.

Please before upgrading to 15.0 make sure to re-enable coredumps with: echo core >/proc/sys/kernel/core_pattern therefore if systemd crashes again you should find /core* coredump file. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c73 --- Comment #73 from andreas bittner --- special package could be put into a special repository on the net or in a local directory or place that would be available to the system during zypper dup? thats my layman understanding of repos. they all get considered during zypper activity if their names and vendors and all that fits into place. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c76 --- Comment #76 from Carlos Robinson --- "--allow-vendor-change" can be used on test system, but not on typical real iron that use extra repos such as packman. Unfortunately the two test systems I created failed to reproduce. Heisenbug? I could, however, clone a 42.3 install (virtual machine) and try upgrading it. But those machines I can not share, they contain private data. I could share the coredump. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c81 --- Comment #81 from Franck Bui --- (In reply to andreas bittner from comment #80)

okay so with this test repo, what additional or fundamental steps for the best debug results should I follow?

if have the 42.2 -> 42.3 system here, can add (modify) all the 15. repos (oss, non-oss, update-oss, update-non-oss) and this special systemd repo

and set the systemd special repo to highest prio

and then zypper dup --download-in-advance

right?

Correct.

what kernel variables, scripts or debug stuff should i follow additonally? or is the special debug repo already including all that is needed?

The special repo should be enough, we'll generate a (more useful) core dump later if you manage to reproduce the issue. But I strongly advice you to try to reproduce on a *testing* system only. Thanks ! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c83 --- Comment #83 from Franck Bui --- (In reply to andreas bittner from comment #82)

yes it is a testing system.

side note: my other production system that was in the unrebooted state, i had the idea to reboot it eventually via reboot -f (reboot alone did nothing) but it never came back since :( so I need to visit on site and see where it hangs or what went wrong.

Possibly some services weren't enabled/disabled during the upgrade... it would help if you could get the logs.

my other production site has some mdraid and data that i dont dare yet to reboot maybe until we find out more about this bug.

At least let's see what the status of your other system that doesn't boot anymore. That said for production systems I wouldn't recommend to make such major upgrade "online" but use the ISO instead. Isn't that the official way to do such upgrades BTW ?

or would it be safe to manually fire up the systemd main (which exactly?) process somehow and re-execute all the processes from the zypper dup stage that wanted to interact with systemd?

Unfortunately I can't think of a way to restart systemd when it exited or crashed. Maybe some people on the systemd mailing list might have some idea...

it is totally unclear to me what I can do about my hanging pending production system where this bug happened during dup.

what is a possible workaround even now that the situation happened?

I'm afraid there're no workarounds at that points :( -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c85 --- Comment #85 from andreas bittner --- No luck here with the 42.2->42.3 system, added the systemd testrepo you created and only activated (edited) the 15.0 oss repo and the 15.0 non-oss repo, but no update repos (situation like on release day with no updates i figured). anyways it is now at around 17xx/27xx packages installing and didnt show timeouts or services and systemd problems I could notice. or would there be more inside some logs no matter what? but it is rather speedyly installing upgrading so i figure it doesnt have systemd problems. this is btrfs system now, which i never use in my normal machines. what about that virtual machine image, did that use btrfs as well? maybe this is only on ext filesystems? wondering of what to do next and how to somehow reproduce the bug situation? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c87 --- Comment #87 from Carlos Robinson --- (In reply to Franck Bui from comment #84)

Carlos, Andreas any chance you can try to reproduce with the test package provided in comment #79 ?

I'm trying. I was somewhere else most of yesterday. The first step is cloning a VM of 42.3 that I have on Vmware Player and creating a copy on VBox instead (VB has snapshots, VmwP does not). This is half done, I have to start the new machine and adjust things till it runs. Then I'll do a snapshot and finally try the zypper dup to 15.0

I tried to do an upgrade from 42.3 to 15.0 but it worked just fine.

Carlos perhaps you can share the ISO you used to reproduce the issue ?

Sure. I'm uploading it to my google drive account and will share it publicly, for a limited time (till I need the space, probably). It says one minute left since minutes. [...] Ok! https://drive.google.com/open?id=1SrYm55VppdehlfYylWyAHh0GHmRDrl4_ -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c89 --- Comment #89 from Franck Bui --- (In reply to Carlos Robinson from comment #87)

Sure. I'm uploading it to my google drive account and will share it publicly, for a limited time (till I need the space, probably). It says one minute left since minutes. [...] Ok!

https://drive.google.com/open?id=1SrYm55VppdehlfYylWyAHh0GHmRDrl4_

Thanks Carlos ! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c91 --- Comment #91 from Carlos Robinson --- It took me some hours to migrate my vmware 42.3 machine to Vbox. Like do something, then wait for completion while doing something else while the VM chugs along, then forgetting. I struggled and failed to get the guest tools installed. Thus I can not copy-paste text. Finally, I took a snapshot and proceeded to upgrade. I forgot to add your modified repo. It did not crash, but there was a curious message somewhere along saying, when some lvm thing crashed, that due to a previous systemd crash coredumps were disabled. But I could not find clues to that previous systemd crash. I will now try to extract that data. Huh, the upgrade changed the session login manager, and this changed default session from xfce to plasma. Possibly another bug. Ok, got the text: 2018-06-06T16:27:28.431706+02:00 Eleanor-423 [RPM][29041]: Transaction ID 5b17ef4a finished: 0 2018-06-06T16:27:28.752736+02:00 Eleanor-423 [RPM][29042]: Transaction ID 5b17ef50 started 2018-06-06T16:27:30.362599+02:00 Eleanor-423 [RPM][29042]: erase lvm2-2.02.120-80.1.x86_64: success 2018-06-06T16:27:32.441801+02:00 Eleanor-423 systemd[1]: Reloading. 2018-06-06T16:27:33.192040+02:00 Eleanor-423 kernel: [14171.371792] lvm2-activation[29061]: segfault at e0 ip 00007fed2151d006 sp 00007ffe20b2e460 err or 4 in liblvm2app.so.2.2[7fed2150b000+ef000] 2018-06-06T16:27:33.680334+02:00 Eleanor-423 systemd-coredump[29070]: Due to PID 1 having crashed coredump collection will now be turned off. 2018-06-06T16:27:33.775189+02:00 Eleanor-423 systemd-coredump[29070]: Failed to parse resource limit: lvm2-activation 2018-06-06T16:27:33.815383+02:00 Eleanor-423 systemd-coredump[29070]: Process 29061 (lvm2-activation) of user 0 dumped core. 2018-06-06T16:27:34.509932+02:00 Eleanor-423 systemd[29059]: /usr/lib/systemd/system-generators/lvm2-activation-generator terminated by signal SEGV. 2018-06-06T16:27:34.522098+02:00 Eleanor-423 systemd[1]: nss-lookup.target: Dependency Before=nss-lookup.target dropped 2018-06-06T16:27:34.871947+02:00 Eleanor-423 systemd[1]: Reloading. 2018-06-06T16:27:35.663657+02:00 Eleanor-423 kernel: [14174.053591] lvm2-activation[29080]: segfault at e0 ip 00007fc6d1d71006 sp 00007ffc285fab50 err or 4 in liblvm2app.so.2.2[7fc6d1d5f000+ef000] 2018-06-06T16:27:36.120172+02:00 Eleanor-423 systemd[29078]: /usr/lib/systemd/system-generators/lvm2-activation-generator terminated by signal SEGV. 2018-06-06T16:27:36.471958+02:00 Eleanor-423 systemd[1]: nss-lookup.target: Dependency Before=nss-lookup.target dropped 2018-06-06T16:27:37.035799+02:00 Eleanor-423 systemd[1]: Reloading. 2018-06-06T16:27:38.038115+02:00 Eleanor-423 kernel: [14176.428105] lvm2-activation[29099]: segfault at e0 ip 00007fb12bffc006 sp 00007ffc07546b00 err or 4 in liblvm2app.so.2.2[7fb12bfea000+ef000] 2018-06-06T16:27:38.316009+02:00 Eleanor-423 systemd[29097]: /usr/lib/systemd/system-generators/lvm2-activation-generator terminated by signal SEGV. 2018-06-06T16:27:38.515374+02:00 Eleanor-423 systemd[1]: nss-lookup.target: Dependency Before=nss-lookup.target dropped 2018-06-06T16:27:39.258534+02:00 Eleanor-423 [RPM][29042]: install lvm2-2.02.177-lp150.5.1.x86_64: success 2018-06-06T16:27:39.974385+02:00 Eleanor-423 systemd[1]: Reloading. 2018-06-06T16:27:41.467045+02:00 Eleanor-423 systemd[1]: nss-lookup.target: Dependency Before=nss-lookup.target dropped 2018-06-06T16:27:42.158789+02:00 Eleanor-423 [RPM][29042]: erase lvm2-2.02.120-80.1.x86_64: success 2018-06-06T16:27:42.166037+02:00 Eleanor-423 [RPM][29042]: Transaction ID 5b17ef50 finished: 0 2018-06-06T16:27:42.395444+02:00 Eleanor-423 [RPM][29139]: Transaction ID 5b17ef5e started I think I started zypper dup (download part) at 12:21:54 (looking at zypper.log): 2018-06-06 12:21:54 <1> Eleanor-423(4354) [zypper] main.cc(main):75 ===== 'zypper' 'dup' ===== 2018-06-06 13:43:01 <1> Eleanor-423(4354) [libsolv++] PoolImpl.cc(logSat):119 file conflict detection took 118916 ms Even the download was slow, and I have 300 MB fibre internet connection! 2018-06-06T13:47:51.642450+02:00 Eleanor-423 systemd[1]: Reexecuting. 2018-06-06T13:47:54.071990+02:00 Eleanor-423 systemd[1]: systemd 228 running in system mode. (+PAM -AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD 2018-06-06T13:47:54.205487+02:00 Eleanor-423 systemd[1]: Detected virtualization kvm. 2018-06-06T13:47:54.255644+02:00 Eleanor-423 systemd[1]: Detected architecture x86-64. 2018-06-06T13:47:56.439330+02:00 Eleanor-423 systemd[1]: nss-lookup.target: Dependency Before=nss-lookup.target dropped 2018-06-06T14:00:02.123843+02:00 Eleanor-423 cron[7479]: PAM unable to dlopen(/lib64/security/pam_unix.so): /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /lib64/libcrypt.so.1) 2018-06-06T14:00:04.776654+02:00 Eleanor-423 cron[7479]: PAM adding faulty module: /lib64/security/pam_unix.so 2018-06-06T14:00:04.915096+02:00 Eleanor-423 cron[7479]: PAM unable to dlopen(/lib64/security/pam_systemd.so): /lib64/libresolv.so.2: symbol __resolv_context_get_preinit, version GLIBC_PRIVATE not defined in file libc.so.6 with li 2018-06-06T14:00:04.921174+02:00 Eleanor-423 cron[7479]: PAM adding faulty module: /lib64/security/pam_systemd.so 2018-06-06T14:00:05.465933+02:00 Eleanor-423 cron[7479]: (root) PAM ERROR (Module is unknown) 2018-06-06T14:00:05.613932+02:00 Eleanor-423 cron[7479]: (root) FAILED to open PAM security session (Module is unknown) 2018-06-06T14:06:20.363433+02:00 Eleanor-423 systemd[1]: Reloading. zypper dup started the rpm transactions about 2018-06-06T14:15:12, finished 2018-06-06T17:01:25 As you can see, about five hours, very slow even without crashes. I have stored the logs safely (yast save logs and some others), in case you want them. I will try, this afternoon, to apply (undo?) the snapshot and redo dup with the test repo added. Even if it runs slow, I don't have to tend to it much :-) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c93 --- Comment #93 from Carlos Robinson --- It does not seem to reproduce without. As it takes so long to run, I thought better to try with it. :-? I'm hoping it is random. (zypper dup started downloads at 12:21, finished transactions 17:01, so 4 hours 40 minutes ;-) Virtual box seems slower than vmware here) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c95 --- Comment #95 from Carlos Robinson --- I forgot about this: (In reply to Franck Bui from comment #44)

(In reply to andreas bittner from comment #43)

...

...

What comes to my mind maybe is, is it possible that non-rebooted machines that e.g. had updates on 42.3 level (systemd and other stuff lately, not yet replaced and activated all files in use, e.g. non rebooted yet) and then going for a zypper dup with repositories swapped over to 15.0 URL addresses

Hmm is this scenario supported ?

I mean should such upgrade are supposed to be done offline ?

Yes, IMHO, it is better to do them offline, but yes, that scenario is supported, and I think that it is more popular in fact. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c97 --- Comment #97 from andreas bittner --- Woot ;) i have a repro! Sigh... with that iso image comment 87 / 94 #boo1094762#c94 #boo1094762#c87 first to write down my thoughts briefly: timestamp: around exactly 13:59.00 CEST to be able to find this later in the logs package (install) number 760 was systemd. took long and gave immediately afterwards those 250000000ms timeout... ... few packages later package number 774 was some other package trying to talk to systemd and brought the same timeouts 2500000000000000ms ;) it is still ongoing. Details: simple seasobed (dualcore) amd64 based machine, single ata disk (think it is even parallel ata even) what i did: install this iso: sha256sum openSUSE-Leap-15.0-DVD-x86_64-Build206.1-Media.iso c38df426959dc69475090f0567ce6661826266f7095d3127e30aac19fdc859c3 openSUSE-Leap-15.0-DVD-x86_64-Build206.1-Media.iso boot this .iso from usb stick. select install. have no network (ethernet cable unplugged) connected. configure network for ethernet interface (during install time) to later be dhcp both ipv4 and ipv6 cable not connected while installing this 206.1.iso install on whole disk, partition /swap (2gig), / = 40gig, /home (rest, some hundred gigs) go for ext4 filesystem for / and /home i unselected some bloat stuff like huge packages libreoffice, k3b and whatnot. nothing special. kde plasma type of install (desktop). added one single testuser/test123 during install, and same password for root. let it install and complete the whole process. no troubles seen here. boot the installed 206.1 machine (still no network cable). takes a while as wicked tries to start the network and waits for dhcp probably or something. log into the kde desktop with the testuser/test123 open a single Konsole there (via alt+f2, konsole) plug in the ethernet cable now, can ping6 and ping various stuff, networking works. all the default repos that come via the default install, so 15-oss, 15-non-oss, 15-update-oss, 15-update-nonoss or these things leave these default repos. su - I then applied that systemd-analyze set-log-level debug command and then continued. I didnt set kernel variables for crashes or something, as you guys said it wasnt about crashes. then: switch to root screen -d -R there create some subterminals or whatever they are called (ctrl+a, c, several times) inside the screen, one of the created subterminals (whatever the naming) do zypper ref --force zypper dup --download-in-advance I created a second tab in Konsole, by clicking next to the single existing tab (where the su - session is being shown inside), and stayed as default testuser, and only went for a watch -n 1 date, to be able to tell when select packages and stuff happens inside the other Konsole tab in the zypper progress...... that was all... first it fetched all the packages (around 16xx), then it did the file check, then it started the dup process.... at package 760, systemd main package was being handled.... immediately after that one (already this one), the timeouts about 250xxxxxxxxms were shown.... some few packages later, another affected package (udevd or something).... was showing the same effects..... --------------- now what? i hope someone can very much the same way repro this stuff again. in fact, i will just try to do this very same procedure over once more to see if the outcome is the same....... any more hints? requests? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c99 --- Comment #99 from andreas bittner --- I see an removal/uninstall at 13:59:09 of the old systemd package, and a successful install/update of the new systemd rpm package at 13:59:11 with the help of journalctl -b there is nothing unusual... after at 14:00:00 that there are entries from e.g. systemd dbusdaemon.... failed to activate service.... org.freedesktop.systemd1..... timeout...... and so on. I dont seem to be able to see errors about systemd unclean exits or something. should I wait for this zypper dup to finish... will take long or is it sufficient if I dump the journalctl -b > somewhere ? I only have this single system/disk at disposal so I can only try again by wiping this very same machine. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c101 --- Comment #101 from Franck Bui --- (In reply to andreas bittner from comment #97)

any more hints? requests?

Great you seem to reproduce this quite easily. Can you retry to reproduce again but with the test repo enabled (see comment #79) so the "special" systemd package is used during the upgrade ? Thanks ! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c103 --- Comment #103 from andreas bittner --- No luck here. With btrfs... 776 = systemd and 791 = udev went through smoothly and zypper dup is still ongoing but smoothly and speedily doing its business. Theory: systemd install/restart is tied to some snapshot/btrfs or similar stuff in a way it can not properly execute or work with nonbtrfs fs? Only thing I can now try is to recreate the ext4 style of the situation. If I can absolutely reproduce this bug with ext4 over and over again, that is the path we need to follow and look for the bug. Maybe its even some kernel or other related bug that comes into play somewhere. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c105 --- Comment #105 from Carlos Robinson --- Failed to reproduce on my side. I'll redo, disabling update repo perhaps :-? Yes, I think it is possibly some race condition or some trigger that we do not know what it is. Meanwhile, my previous run detected a bug on btrfsmaintenance, because it should not run like it did on my test system, which I'll report later. As curiosity, the upgrade took 4,8333 hours ;-) time zypper dup... real 290m35.710s user 92m2.886s sys 132m7.424s I have a Virtualbox snapshot placed just before running zypper dup, so it is really easy for me to run it. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c113 --- Comment #113 from Carlos Robinson --- Sorry, chaps, I have been busy with other things, like filling my yearly tax forms, something I hate because they don't talk a human language, but some form of legalesse. I did one zypper dup on real hardware on a test partition of my desktop machine. I was going to do an offline upgrade with the DVD-USB_stick, but it crashed (Bug 1094963), so contrary to my tradition I did a zypper dup instead which worked perfectly. I will try again on my virtual platform for this issue hopefully this week. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 http://bugzilla.opensuse.org/show_bug.cgi?id=1094762#c116 Franck Bui changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution|--- |WORKSFORME --- Comment #116 from Franck Bui --- OK I'm closing this bug for now, feel free to re-open in case you have a reproducer (and make sure to back it up). Again, thanks Andreas and Carlos for trying hard to track this one down. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1094762 Nick Dordea changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ndordea@gmail.com -- You are receiving this mail because: You are on the CC list for the bug.