https://bugzilla.suse.com/show_bug.cgi?id=1215837 Bug ID: 1215837 Summary: [Virtualization: KVM] systemd[1]: "Started Process Core Dump" in journals on localhost Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Tumbleweed Status: NEW Severity: Normal Priority: P5 - None Component: KVM Assignee: kvm-bugs@suse.de Reporter: varun.ojha@suse.com QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: --- Blocker: --- Created attachment 869843 --> https://bugzilla.suse.com/attachment.cgi?id=869843&action=edit Support Config KVM ## Description [Virtualization: KVM] systemd[1]: "Started Process Core Dump" in journals on localhost In the logs, there's a recurring issue with NetworkManager, which is triggering a "Started Process Core Dump" error in the system journals on the localhost. This issue is causing the NetworkManager service to fail with a core dump. The issue seems to be affecting the stability of the network management service on the system. NetworkManager Version: 1.44.0-1.1 Architecture: x86_64 Operating System: openSUSE-20230927-0 ## Information blackbauhinia:~ # zypper info NetworkManager Loading repository data... Reading installed packages... Information for package NetworkManager: --------------------------------------- Repository : openSUSE-20230927-0 Name : NetworkManager Version : 1.44.0-1.1 Arch : x86_64 Vendor : openSUSE Installed Size : 5.9 MiB Installed : Yes Status : up-to-date Source package : NetworkManager-1.44.0-1.1.src Upstream URL : https://networkmanager.dev/ Summary : Standard Linux network configuration tool suite Description : NetworkManager attempts to keep an active network connection available at all times. The point of NetworkManager is to make networking configuration and setup as painless and automatic as possible. If using DHCP, NetworkManager is intended to replace default routes, obtain IP addresses from a DHCP server, and change name servers whenever it sees fit. blackbauhinia:~ # coredumpctl --all TIME PID UID GID SIG COREFILE EXE SIZE Fri 2023-09-29 09:40:48 EDT 1787 0 0 SIGABRT present /usr/sbin/NetworkManager 522.1K blackbauhinia:~ # ## Reproduce steps The exact steps to reproduce the issue are not provided in the bug report. However, the issue has been observed in the logs of the system. Possible steps 1. Install openSUSE Tumbleweed with Latest build 2. After Autoyast installation, file: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/data/... 3. During virt_autotest/login_console, after KVM modules are loaded and NetworkManager is seen with Core dump in journalctl Can be seen here: https://openqa.opensuse.org/tests/3609023#step/login_console/17 https://openqa.opensuse.org/tests/3608243#step/login_console/34 https://openqa.opensuse.org/tests/3603405#step/login_console/19 Expected result: The NetworkManager service should run without errors and manage network connections as expected. Real result: The NetworkManager service is encountering a critical error and failing with a core dump. ## Reproducibility seen multiple times, frequently: Not 100% Was seen in build 20230927 ## Last good build was 20230922, 20230925 Now Seen at recent builds, 20230927 20230926 ## Environment(optional) Linux blackbauhinia 6.5.4-1-default #1 SMP PREEMPT_DYNAMIC Wed Sep 20 05:07:04 UTC 2023 (fdd7e9e) x86_64 x86_64 x86_64 GNU/Linux Linux Kernel Version: 6.5.4-1-default Hostname: blackbauhinia ## Testsuite description(optional, apply for complex scenarios) Test suit: virt-guest-installation-kvm@64bit-ipmi & virt-guest-installation-xen@64bit-ipmi ## Error https://openqa.opensuse.org/tests/3597532#step/login_console/19 Sep 25 00:03:57 blackbauhinia systemd[1]: Started Process Core Dump (PID 2366/UID 0). Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager-wait-online.service: Main process exited, code=exited, status=> Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager-wait-online.service: Failed with result 'exit-code'. Sep 25 00:03:57 blackbauhinia systemd[1]: Failed to start Network Manager Wait Online. Sep 25 00:03:57 blackbauhinia systemd[1]: Reached target Network is Online. Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Login and scanning of iSCSI devices... Sep 25 00:03:57 blackbauhinia iscsiadm[2369]: iscsiadm: No records found Sep 25 00:03:57 blackbauhinia iscsiadm[2371]: iscsiadm: No records found Sep 25 00:03:57 blackbauhinia iscsiadm[2373]: iscsiadm: Could not get list of targets from firmware. (err 21) Sep 25 00:03:57 blackbauhinia systemd-coredump[2367]: [🡕] Process 1781 (NetworkManager) of user 0 dumped core. Stack trace of thread 1781: #0 0x00007f42f2691e0c __pthread_kill_implementation (libc.so.6 +> #1 0x00007f42f263f0e6 raise (libc.so.6 + 0x3f0e6) #2 0x00007f42f26268d7 abort (libc.so.6 + 0x268d7) #3 0x00007f42f2e58b1d g_mutex_clear (libglib-2.0.so.0 + 0xb7b1d) #4 0x00007f42f30094a2 n/a (libgio-2.0.so.0 + 0xbe4a2) #5 0x00007f42f3005b4c n/a (libgio-2.0.so.0 + 0xbab4c) #6 0x00007f42f2f05970 g_object_unref (libgobject-2.0.so.0 + 0x1c> #7 0x00007f42f2dfaf89 n/a (libglib-2.0.so.0 + 0x59f89) #8 0x00007f42f2dfcb46 n/a (libglib-2.0.so.0 + 0x5bb46) #9 0x00007f42f2dfcf6f n/a (libglib-2.0.so.0 + 0x5bf6f) #10 0x00007f42f2dfeb18 n/a (libglib-2.0.so.0 + 0x5db18) #11 0x00007f42f2dff3ef g_main_loop_run (libglib-2.0.so.0 + 0x5e3e> #12 0x00005643adfd6493 n/a (NetworkManager + 0x3f493) #13 0x00007f42f26281b0 __libc_start_call_main (libc.so.6 + 0x281b> #14 0x00007f42f2628279 __libc_start_main@@GLIBC_2.34 (libc.so.6 +> #15 0x00005643adfd6915 n/a (NetworkManager + 0x3f915) Stack trace of thread 1791: #0 0x00007f42f2709d2f __poll (libc.so.6 + 0x109d2f) #1 0x00007f42f2dfeabf n/a (libglib-2.0.so.0 + 0x5dabf) #2 0x00007f42f2dff1cc g_main_context_iteration (libglib-2.0.so.0> #3 0x00007f42f2dff211 n/a (libglib-2.0.so.0 + 0x5e211) #4 0x00007f42f2e2b44e n/a (libglib-2.0.so.0 + 0x8a44e) #5 0x00007f42f268ff64 start_thread (libc.so.6 + 0x8ff64) #6 0x00007f42f271847c __clone3 (libc.so.6 + 0x11847c) Stack trace of thread 1792: #0 0x00007f42f271616d syscall (libc.so.6 + 0x11616d) #1 0x00007f42f2e58cb0 g_cond_wait (libglib-2.0.so.0 + 0xb7cb0) #2 0x00007f42f2dc902b n/a (libglib-2.0.so.0 + 0x2802b) #3 0x00007f42f2e2ba92 n/a (libglib-2.0.so.0 + 0x8aa92) #4 0x00007f42f2e2b44e n/a (libglib-2.0.so.0 + 0x8a44e) #5 0x00007f42f268ff64 start_thread (libc.so.6 + 0x8ff64) #6 0x00007f42f271847c __clone3 (libc.so.6 + 0x11847c) Stack trace of thread 1793: #0 0x00007f42f2709d2f __poll (libc.so.6 + 0x109d2f) #1 0x00007f42f2dfeabf n/a (libglib-2.0.so.0 + 0x5dabf) #2 0x00007f42f2dff3ef g_main_loop_run (libglib-2.0.so.0 + 0x5e3e> #3 0x00007f42f30708e6 n/a (libgio-2.0.so.0 + 0x1258e6) #4 0x00007f42f2e2b44e n/a (libglib-2.0.so.0 + 0x8a44e) #5 0x00007f42f268ff64 start_thread (libc.so.6 + 0x8ff64) #6 0x00007f42f271847c __clone3 (libc.so.6 + 0x11847c) Stack trace of thread 2365: #0 0x00007f42f271616d syscall (libc.so.6 + 0x11616d) #1 0x00007f42f2e58e5c g_cond_wait_until (libglib-2.0.so.0 + 0xb7> #2 0x00007f42f2dc9003 n/a (libglib-2.0.so.0 + 0x28003) #3 0x00007f42f2e2be0a n/a (libglib-2.0.so.0 + 0x8ae0a) #4 0x00007f42f2e2b44e n/a (libglib-2.0.so.0 + 0x8a44e) #5 0x00007f42f268ff64 start_thread (libc.so.6 + 0x8ff64) #6 0x00007f42f271847c __clone3 (libc.so.6 + 0x11847c) ELF object binary architecture: AMD x86-64 Sep 25 00:03:57 blackbauhinia systemd[1]: Finished Login and scanning of iSCSI devices. Sep 25 00:03:57 blackbauhinia systemd[1]: systemd-coredump@0-2366-0.service: Deactivated successfully. Sep 25 00:03:57 blackbauhinia systemd[1]: Reached target Remote File Systems. Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Autoyast2 Init Scripts... Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Virtualization qemu daemon... Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Virtualization storage daemon... Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager.service: Main process exited, code=dumped, status=6/ABRT Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager.service: Failed with result 'core-dump'. ## Logs/ crash # /bin/coredumpctl info 1787 PID: 1787 (NetworkManager) UID: 0 (root) GID: 0 (root) Signal: 6 (ABRT) Timestamp: Fri 2023-09-29 09:40:48 EDT (53min ago) Command Line: /usr/sbin/NetworkManager --no-daemon Executable: /usr/sbin/NetworkManager Control Group: /system.slice/NetworkManager.service Unit: NetworkManager.service Slice: system.slice Boot ID: 752a88f1ac2b4381b0fc7591961e743a Machine ID: 629cb90a95bb451e9026683e7cb65723 Hostname: blackbauhinia Storage: /var/lib/systemd/coredump/core.NetworkManager.0.752a88f1ac2b4381b0fc7591961e743a.1787.1695994848000000.zst (present) Size on Disk: 522.1K Message: Process 1787 (NetworkManager) of user 0 dumped core. Stack trace of thread 1787: #0 0x00007f4ed8491e0c __pthread_kill_implementation (libc.so.6 + 0x91e0c) #1 0x00007f4ed843f0e6 raise (libc.so.6 + 0x3f0e6) #2 0x00007f4ed84268d7 abort (libc.so.6 + 0x268d7) #3 0x00007f4ed8bcdb1d g_mutex_clear (libglib-2.0.so.0 + 0xb7b1d) #4 0x00007f4ed8d7e4a2 n/a (libgio-2.0.so.0 + 0xbe4a2) #5 0x00007f4ed8d7ab4c n/a (libgio-2.0.so.0 + 0xbab4c) #6 0x00007f4ed8c7a970 g_object_unref (libgobject-2.0.so.0 + 0x1c970) #7 0x00007f4ed8b6ff89 n/a (libglib-2.0.so.0 + 0x59f89) #8 0x00007f4ed8b71b46 n/a (libglib-2.0.so.0 + 0x5bb46) #9 0x00007f4ed8b71f6f n/a (libglib-2.0.so.0 + 0x5bf6f) #10 0x00007f4ed8b73b18 n/a (libglib-2.0.so.0 + 0x5db18) #11 0x00007f4ed8b743ef g_main_loop_run (libglib-2.0.so.0 + 0x5e3ef) #12 0x00005652d1963493 n/a (NetworkManager + 0x3f493) #13 0x00007f4ed84281b0 __libc_start_call_main (libc.so.6 + 0x281b0) #14 0x00007f4ed8428279 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x28279) #15 0x00005652d1963915 n/a (NetworkManager + 0x3f915) Stack trace of thread 1796: #0 0x00007f4ed8509d2f __poll (libc.so.6 + 0x109d2f) #1 0x00007f4ed8b73abf n/a (libglib-2.0.so.0 + 0x5dabf) #2 0x00007f4ed8b741cc g_main_context_iteration (libglib-2.0.so.0 + 0x5e1cc) #3 0x00007f4ed8b74211 n/a (libglib-2.0.so.0 + 0x5e211) #4 0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e) #5 0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64) #6 0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c) Stack trace of thread 1797: #0 0x00007f4ed851616d syscall (libc.so.6 + 0x11616d) #1 0x00007f4ed8bcdcb0 g_cond_wait (libglib-2.0.so.0 + 0xb7cb0) #2 0x00007f4ed8b3e02b n/a (libglib-2.0.so.0 + 0x2802b) #3 0x00007f4ed8ba0a92 n/a (libglib-2.0.so.0 + 0x8aa92) #4 0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e) #5 0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64) #6 0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c) Stack trace of thread 2369: #0 0x00007f4ed851616d syscall (libc.so.6 + 0x11616d) #1 0x00007f4ed8bcde5c g_cond_wait_until (libglib-2.0.so.0 + 0xb7e5c) #2 0x00007f4ed8b3e003 n/a (libglib-2.0.so.0 + 0x28003) #3 0x00007f4ed8ba0e0a n/a (libglib-2.0.so.0 + 0x8ae0a) #4 0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e) #5 0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64) #6 0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c) Stack trace of thread 1798: #0 0x00007f4ed8509d2f __poll (libc.so.6 + 0x109d2f) #1 0x00007f4ed8b73abf n/a (libglib-2.0.so.0 + 0x5dabf) #2 0x00007f4ed8b743ef g_main_loop_run (libglib-2.0.so.0 + 0x5e3ef) #3 0x00007f4ed8de58e6 n/a (libgio-2.0.so.0 + 0x1258e6) #4 0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e) #5 0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64) #6 0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c) ELF object binary architecture: AMD x86-64 The stack trace of the core dump is also provided in the logs. ## Workaround No Please note that resolving this issue may require further investigation, debugging, and potentially collaboration with the NetworkManager development team. -- You are receiving this mail because: You are on the CC list for the bug.