Created attachment 869843 [details]
Support Config KVM
## Description
[Virtualization: KVM] systemd[1]: "Started Process Core Dump" in journals on
localhost
In the logs, there's a recurring issue with NetworkManager, which is triggering
a "Started Process Core Dump" error in the system journals on the localhost.
This issue is causing the NetworkManager service to fail with a core dump. The
issue seems to be affecting the stability of the network management service on
the system.
NetworkManager Version: 1.44.0-1.1
Architecture: x86_64
Operating System: openSUSE-20230927-0
## Information
blackbauhinia:~ # zypper info NetworkManager
Loading repository data...
Reading installed packages...
Information for package NetworkManager:
---------------------------------------
Repository : openSUSE-20230927-0
Name : NetworkManager
Version : 1.44.0-1.1
Arch : x86_64
Vendor : openSUSE
Installed Size : 5.9 MiB
Installed : Yes
Status : up-to-date
Source package : NetworkManager-1.44.0-1.1.src
Upstream URL : https://networkmanager.dev/
Summary : Standard Linux network configuration tool suite
Description :
NetworkManager attempts to keep an active network connection available
at all times. The point of NetworkManager is to make networking
configuration and setup as painless and automatic as possible. If
using DHCP, NetworkManager is intended to replace default routes,
obtain IP addresses from a DHCP server, and change name servers
whenever it sees fit.
blackbauhinia:~ # coredumpctl --all
TIME PID UID GID SIG COREFILE EXE
SIZE
Fri 2023-09-29 09:40:48 EDT 1787 0 0 SIGABRT present
/usr/sbin/NetworkManager 522.1K
blackbauhinia:~ #
## Reproduce steps
The exact steps to reproduce the issue are not provided in the bug report.
However, the issue has been observed in the logs of the system.
Possible steps
1. Install openSUSE Tumbleweed with Latest build
2. After Autoyast installation, file:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/data/virt_autotest/host_unattended_installation_files/autoyast_opensuse_kvm_two_disk.xml
3. During virt_autotest/login_console, after KVM modules are loaded and
NetworkManager is seen with Core dump in journalctl
Can be seen here:
https://openqa.opensuse.org/tests/3609023#step/login_console/17
https://openqa.opensuse.org/tests/3608243#step/login_console/34
https://openqa.opensuse.org/tests/3603405#step/login_console/19
Expected result: The NetworkManager service should run without errors and
manage network connections as expected.
Real result: The NetworkManager service is encountering a critical error and
failing with a core dump.
## Reproducibility
seen multiple times, frequently: Not 100%
Was seen in build 20230927
## Last good build
was
20230922, 20230925
Now Seen at recent builds,
20230927
20230926
## Environment(optional)
Linux blackbauhinia 6.5.4-1-default #1 SMP PREEMPT_DYNAMIC Wed Sep 20 05:07:04
UTC 2023 (fdd7e9e) x86_64 x86_64 x86_64 GNU/Linux
Linux Kernel Version: 6.5.4-1-default
Hostname: blackbauhinia
## Testsuite description(optional, apply for complex scenarios)
Test suit: virt-guest-installation-kvm@64bit-ipmi &
virt-guest-installation-xen@64bit-ipmi
## Error
https://openqa.opensuse.org/tests/3597532#step/login_console/19
Sep 25 00:03:57 blackbauhinia systemd[1]: Started Process Core Dump (PID
2366/UID 0).
Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager-wait-online.service:
Main process exited, code=exited, status=>
Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager-wait-online.service:
Failed with result 'exit-code'.
Sep 25 00:03:57 blackbauhinia systemd[1]: Failed to start Network Manager Wait
Online.
Sep 25 00:03:57 blackbauhinia systemd[1]: Reached target Network is Online.
Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Login and scanning of iSCSI
devices...
Sep 25 00:03:57 blackbauhinia iscsiadm[2369]: iscsiadm: No records found
Sep 25 00:03:57 blackbauhinia iscsiadm[2371]: iscsiadm: No records found
Sep 25 00:03:57 blackbauhinia iscsiadm[2373]: iscsiadm: Could not get list of
targets from firmware. (err 21)
Sep 25 00:03:57 blackbauhinia systemd-coredump[2367]: [🡕] Process 1781
(NetworkManager) of user 0 dumped core.
Stack trace of thread
1781:
#0 0x00007f42f2691e0c
__pthread_kill_implementation (libc.so.6 +>
#1 0x00007f42f263f0e6
raise (libc.so.6 + 0x3f0e6)
#2 0x00007f42f26268d7
abort (libc.so.6 + 0x268d7)
#3 0x00007f42f2e58b1d
g_mutex_clear (libglib-2.0.so.0 + 0xb7b1d)
#4 0x00007f42f30094a2
n/a (libgio-2.0.so.0 + 0xbe4a2)
#5 0x00007f42f3005b4c
n/a (libgio-2.0.so.0 + 0xbab4c)
#6 0x00007f42f2f05970
g_object_unref (libgobject-2.0.so.0 + 0x1c>
#7 0x00007f42f2dfaf89
n/a (libglib-2.0.so.0 + 0x59f89)
#8 0x00007f42f2dfcb46
n/a (libglib-2.0.so.0 + 0x5bb46)
#9 0x00007f42f2dfcf6f
n/a (libglib-2.0.so.0 + 0x5bf6f)
#10 0x00007f42f2dfeb18
n/a (libglib-2.0.so.0 + 0x5db18)
#11 0x00007f42f2dff3ef
g_main_loop_run (libglib-2.0.so.0 + 0x5e3e>
#12 0x00005643adfd6493
n/a (NetworkManager + 0x3f493)
#13 0x00007f42f26281b0
__libc_start_call_main (libc.so.6 + 0x281b>
#14 0x00007f42f2628279
__libc_start_main@@GLIBC_2.34 (libc.so.6 +>
#15 0x00005643adfd6915
n/a (NetworkManager + 0x3f915)
Stack trace of thread
1791:
#0 0x00007f42f2709d2f
__poll (libc.so.6 + 0x109d2f)
#1 0x00007f42f2dfeabf
n/a (libglib-2.0.so.0 + 0x5dabf)
#2 0x00007f42f2dff1cc
g_main_context_iteration (libglib-2.0.so.0>
#3 0x00007f42f2dff211
n/a (libglib-2.0.so.0 + 0x5e211)
#4 0x00007f42f2e2b44e
n/a (libglib-2.0.so.0 + 0x8a44e)
#5 0x00007f42f268ff64
start_thread (libc.so.6 + 0x8ff64)
#6 0x00007f42f271847c
__clone3 (libc.so.6 + 0x11847c)
Stack trace of thread
1792:
#0 0x00007f42f271616d
syscall (libc.so.6 + 0x11616d)
#1 0x00007f42f2e58cb0
g_cond_wait (libglib-2.0.so.0 + 0xb7cb0)
#2 0x00007f42f2dc902b
n/a (libglib-2.0.so.0 + 0x2802b)
#3 0x00007f42f2e2ba92
n/a (libglib-2.0.so.0 + 0x8aa92)
#4 0x00007f42f2e2b44e
n/a (libglib-2.0.so.0 + 0x8a44e)
#5 0x00007f42f268ff64
start_thread (libc.so.6 + 0x8ff64)
#6 0x00007f42f271847c
__clone3 (libc.so.6 + 0x11847c)
Stack trace of thread
1793:
#0 0x00007f42f2709d2f
__poll (libc.so.6 + 0x109d2f)
#1 0x00007f42f2dfeabf
n/a (libglib-2.0.so.0 + 0x5dabf)
#2 0x00007f42f2dff3ef
g_main_loop_run (libglib-2.0.so.0 + 0x5e3e>
#3 0x00007f42f30708e6
n/a (libgio-2.0.so.0 + 0x1258e6)
#4 0x00007f42f2e2b44e
n/a (libglib-2.0.so.0 + 0x8a44e)
#5 0x00007f42f268ff64
start_thread (libc.so.6 + 0x8ff64)
#6 0x00007f42f271847c
__clone3 (libc.so.6 + 0x11847c)
Stack trace of thread
2365:
#0 0x00007f42f271616d
syscall (libc.so.6 + 0x11616d)
#1 0x00007f42f2e58e5c
g_cond_wait_until (libglib-2.0.so.0 + 0xb7>
#2 0x00007f42f2dc9003
n/a (libglib-2.0.so.0 + 0x28003)
#3 0x00007f42f2e2be0a
n/a (libglib-2.0.so.0 + 0x8ae0a)
#4 0x00007f42f2e2b44e
n/a (libglib-2.0.so.0 + 0x8a44e)
#5 0x00007f42f268ff64
start_thread (libc.so.6 + 0x8ff64)
#6 0x00007f42f271847c
__clone3 (libc.so.6 + 0x11847c)
ELF object binary
architecture: AMD x86-64
Sep 25 00:03:57 blackbauhinia systemd[1]: Finished Login and scanning of iSCSI
devices.
Sep 25 00:03:57 blackbauhinia systemd[1]: systemd-coredump@0-2366-0.service:
Deactivated successfully.
Sep 25 00:03:57 blackbauhinia systemd[1]: Reached target Remote File Systems.
Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Autoyast2 Init Scripts...
Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Virtualization qemu
daemon...
Sep 25 00:03:57 blackbauhinia systemd[1]: Starting Virtualization storage
daemon...
Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager.service: Main process
exited, code=dumped, status=6/ABRT
Sep 25 00:03:57 blackbauhinia systemd[1]: NetworkManager.service: Failed with
result 'core-dump'.
## Logs/ crash
# /bin/coredumpctl info 1787
PID: 1787 (NetworkManager)
UID: 0 (root)
GID: 0 (root)
Signal: 6 (ABRT)
Timestamp: Fri 2023-09-29 09:40:48 EDT (53min ago)
Command Line: /usr/sbin/NetworkManager --no-daemon
Executable: /usr/sbin/NetworkManager
Control Group: /system.slice/NetworkManager.service
Unit: NetworkManager.service
Slice: system.slice
Boot ID: 752a88f1ac2b4381b0fc7591961e743a
Machine ID: 629cb90a95bb451e9026683e7cb65723
Hostname: blackbauhinia
Storage:
/var/lib/systemd/coredump/core.NetworkManager.0.752a88f1ac2b4381b0fc7591961e743a.1787.1695994848000000.zst
(present)
Size on Disk: 522.1K
Message: Process 1787 (NetworkManager) of user 0 dumped core.
Stack trace of thread 1787:
#0 0x00007f4ed8491e0c __pthread_kill_implementation (libc.so.6
+ 0x91e0c)
#1 0x00007f4ed843f0e6 raise (libc.so.6 + 0x3f0e6)
#2 0x00007f4ed84268d7 abort (libc.so.6 + 0x268d7)
#3 0x00007f4ed8bcdb1d g_mutex_clear (libglib-2.0.so.0 +
0xb7b1d)
#4 0x00007f4ed8d7e4a2 n/a (libgio-2.0.so.0 + 0xbe4a2)
#5 0x00007f4ed8d7ab4c n/a (libgio-2.0.so.0 + 0xbab4c)
#6 0x00007f4ed8c7a970 g_object_unref (libgobject-2.0.so.0 +
0x1c970)
#7 0x00007f4ed8b6ff89 n/a (libglib-2.0.so.0 + 0x59f89)
#8 0x00007f4ed8b71b46 n/a (libglib-2.0.so.0 + 0x5bb46)
#9 0x00007f4ed8b71f6f n/a (libglib-2.0.so.0 + 0x5bf6f)
#10 0x00007f4ed8b73b18 n/a (libglib-2.0.so.0 + 0x5db18)
#11 0x00007f4ed8b743ef g_main_loop_run (libglib-2.0.so.0 +
0x5e3ef)
#12 0x00005652d1963493 n/a (NetworkManager + 0x3f493)
#13 0x00007f4ed84281b0 __libc_start_call_main (libc.so.6 +
0x281b0)
#14 0x00007f4ed8428279 __libc_start_main@@GLIBC_2.34 (libc.so.6
+ 0x28279)
#15 0x00005652d1963915 n/a (NetworkManager + 0x3f915)
Stack trace of thread 1796:
#0 0x00007f4ed8509d2f __poll (libc.so.6 + 0x109d2f)
#1 0x00007f4ed8b73abf n/a (libglib-2.0.so.0 + 0x5dabf)
#2 0x00007f4ed8b741cc g_main_context_iteration
(libglib-2.0.so.0 + 0x5e1cc)
#3 0x00007f4ed8b74211 n/a (libglib-2.0.so.0 + 0x5e211)
#4 0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e)
#5 0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64)
#6 0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c)
Stack trace of thread 1797:
#0 0x00007f4ed851616d syscall (libc.so.6 + 0x11616d)
#1 0x00007f4ed8bcdcb0 g_cond_wait (libglib-2.0.so.0 + 0xb7cb0)
#2 0x00007f4ed8b3e02b n/a (libglib-2.0.so.0 + 0x2802b)
#3 0x00007f4ed8ba0a92 n/a (libglib-2.0.so.0 + 0x8aa92)
#4 0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e)
#5 0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64)
#6 0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c)
Stack trace of thread 2369:
#0 0x00007f4ed851616d syscall (libc.so.6 + 0x11616d)
#1 0x00007f4ed8bcde5c g_cond_wait_until (libglib-2.0.so.0 +
0xb7e5c)
#2 0x00007f4ed8b3e003 n/a (libglib-2.0.so.0 + 0x28003)
#3 0x00007f4ed8ba0e0a n/a (libglib-2.0.so.0 + 0x8ae0a)
#4 0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e)
#5 0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64)
#6 0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c)
Stack trace of thread 1798:
#0 0x00007f4ed8509d2f __poll (libc.so.6 + 0x109d2f)
#1 0x00007f4ed8b73abf n/a (libglib-2.0.so.0 + 0x5dabf)
#2 0x00007f4ed8b743ef g_main_loop_run (libglib-2.0.so.0 +
0x5e3ef)
#3 0x00007f4ed8de58e6 n/a (libgio-2.0.so.0 + 0x1258e6)
#4 0x00007f4ed8ba044e n/a (libglib-2.0.so.0 + 0x8a44e)
#5 0x00007f4ed848ff64 start_thread (libc.so.6 + 0x8ff64)
#6 0x00007f4ed851847c __clone3 (libc.so.6 + 0x11847c)
ELF object binary architecture: AMD x86-64
The stack trace of the core dump is also provided in the logs.
## Workaround
No
Please note that resolving this issue may require further investigation,
debugging, and potentially collaboration with the NetworkManager development
team.