[Bug 1107197] New: NetworkManager coredump in _con_curl_check_connectivity
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 Bug ID: 1107197 Summary: NetworkManager coredump in _con_curl_check_connectivity Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Major Priority: P5 - None Component: Network Assignee: bnc-team-screening@forge.provo.novell.com Reporter: adam@mizerski.pl QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- NetworkManager randomly crashed when network connectivity check fails to connect. (gdb) bt #0 0x00007f5a91eb94bb in _int_free () from /lib64/libc.so.6 #1 0x00007f5a9263b901 in ?? () from /usr/lib64/libcurl.so.4 #2 0x00007f5a92649cad in curl_easy_cleanup () from /usr/lib64/libcurl.so.4 #3 0x0000560ecadfc07c in cb_data_free (cb_data=0x560ecc079800, state=NM_CONNECTIVITY_LIMITED, error=0x0, log_message=0x560ecc082bb0 "check failed with curl status 7") at src/nm-connectivity.c:209 #4 0x0000560ecadfc285 in _con_curl_check_connectivity (mhandle=0x560ecbff7c40, sockfd=<optimized out>, ev_bitmask=<optimized out>) at src/nm-connectivity.c:275 #5 0x0000560ecadfc39d in _con_curl_socketevent_cb (ch=0x560ecc09ef00, condition=<optimized out>, user_data=0x560ecc0dc960) at src/nm-connectivity.c:359 #6 0x00007f5a92ba3ea8 in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0 #7 0x00007f5a92ba4298 in ?? () from /usr/lib64/libglib-2.0.so.0 #8 0x00007f5a92ba4592 in g_main_loop_run () from /usr/lib64/libglib-2.0.so.0 #9 0x0000560ecacaf072 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:438 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 Karl Cheng <qantas94heavy@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High CC| |qantas94heavy@gmail.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c1 Vojtech Zeisek <Vojtech.Zeisek@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Vojtech.Zeisek@opensuse.org Hardware|Other |x86-64 --- Comment #1 from Vojtech Zeisek <Vojtech.Zeisek@opensuse.org> --- I have same problem, I think. Every login or connecting to WiFi, NetworkManager randomly crashes. It is then usually restarted (sometimes not), and then fails to connect to KWallet (KDE). Result is no WiFi connection. It doesn't happen every time, sometimes everything works. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c2 Stefan Brüns <stefan.bruens@rwth-aachen.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |stefan.bruens@rwth-aachen.d | |e --- Comment #2 from Stefan Brüns <stefan.bruens@rwth-aachen.de> --- (In reply to Adam Mizerski from comment #0)
NetworkManager randomly crashed when network connectivity check fails to connect.
(gdb) bt #0 0x00007f5a91eb94bb in _int_free () from /lib64/libc.so.6 #1 0x00007f5a9263b901 in ?? () from /usr/lib64/libcurl.so.4 #2 0x00007f5a92649cad in curl_easy_cleanup () from /usr/lib64/libcurl.so.4
Did you update using zypper dup? Does the problem persist? If yes, please install the libcurl debuginfo package, and recreate the backtrace. If you use "up" in the gdb prompt, it will also show the function parameters, when available. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c3 --- Comment #3 from Adam Mizerski <adam@mizerski.pl> ---
Did you update using zypper dup?
I always do.
Does the problem persist?
No. This happened only two times, when my internet connection was unstable.
If yes, please install the libcurl debuginfo package, and recreate the backtrace. If you use "up" in the gdb prompt, it will also show the function parameters, when available.
Unfortunately my coredumps were removed from journal. If this happens again, I'll get more info. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c4 Martin Wilck <martin.wilck@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.wilck@suse.com --- Comment #4 from Martin Wilck <martin.wilck@suse.com> --- Can you correlate this to the update of NM to 1.12.2? If yes, I am having (albeit different) trouble with that version, too (bug 1106993). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c5 --- Comment #5 from Vojtech Zeisek <Vojtech.Zeisek@opensuse.org> --- (In reply to Martin Wilck from comment #4)
Can you correlate this to the update of NM to 1.12.2?
Yes, exactly. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c6 Luca Beltrame <lbeltrame@kde.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lbeltrame@kde.org --- Comment #6 from Luca Beltrame <lbeltrame@kde.org> --- #1 0x00007f3c36ca72b2 in calloc () from /lib64/libc.so.6 (gdb) bt #0 0x00007f3c36ca5228 in _int_malloc () at /lib64/libc.so.6 #1 0x00007f3c36ca72b2 in calloc () at /lib64/libc.so.6 #2 0x00007f3c37425c4a in Curl_open (curl=curl@entry=0x7fffba3b5110) at url.c:558 #3 0x00007f3c374339d4 in curl_easy_init () at easy.c:370 #4 0x000055902d383a75 in () #5 0x000055902d3190ed in () #6 0x000055902d319223 in () #7 0x000055902d31937e in () #8 0x00007f3c379b4933 in () at /usr/lib64/libglib-2.0.so.0 #9 0x00007f3c379b3ea8 in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0 #10 0x00007f3c379b4298 in () at /usr/lib64/libglib-2.0.so.0 #11 0x00007f3c379b4592 in g_main_loop_run () at /usr/lib64/libglib-2.0.so.0 #12 0x000055902d236072 in () #13 0x00007f3c36c43feb in __libc_start_main () at /lib64/libc.so.6 #14 0x000055902d23668a in () I generated the bt with the debug symbols above. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c7 --- Comment #7 from Adam Mizerski <adam@mizerski.pl> --- Happened again to me. #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007ff9037a14e9 in __GI_abort () at abort.c:79 #2 0x00007ff9037fa9a7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ff9039044c2 "%s\n") at ../sysdeps/posix/libc_fatal.c:181 #3 0x00007ff903800e9c in malloc_printerr (str=str@entry=0x7ff90390262b "corrupted double-linked list") at malloc.c:5350 #4 0x00007ff903804338 in _int_malloc (av=av@entry=0x7ff903b3ac40 <main_arena>, bytes=bytes@entry=21336) at malloc.c:3926 #5 0x00007ff9038062b2 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3436 #6 0x00007ff903f84c4a in ?? () from /usr/lib64/libcurl.so.4 #7 0x00007ff903f929d4 in curl_easy_init () from /usr/lib64/libcurl.so.4 #8 0x000055936b3bca75 in nm_connectivity_check_start (self=0x55936c2dfb60, iface=iface@entry=0x55936c347af0 "virbr0", callback=callback@entry=0x55936b35aeb0 <concheck_cb>, user_data=user_data@entry=0x55936c28f740) at src/nm-connectivity.c:510 #9 0x000055936b3520ed in concheck_start (self=self@entry=0x55936c35f6e0, callback=callback@entry=0x0, user_data=user_data@entry=0x0, is_periodic=is_periodic@entry=1) at src/devices/nm-device.c:2772 #10 0x000055936b352223 in concheck_periodic_schedule_set (self=0x55936c35f6e0, self@entry=<error reading variable: value has been optimized out>, mode=mode@entry=CONCHECK_SCHEDULE_CHECK_PERIODIC) at src/devices/nm-device.c:2439 #11 0x000055936b35237e in concheck_periodic_timeout_cb (user_data=<error reading variable: value has been optimized out>) at src/devices/nm-device.c:2265 #12 0x00007ff9044ed933 in g_timeout_dispatch (source=0x55936c3776f0, callback=<optimized out>, user_data=<optimized out>) at gmain.c:4649 #13 0x00007ff9044ecea8 in g_main_dispatch (context=0x55936c28fc50) at gmain.c:3176 #14 g_main_context_dispatch (context=context@entry=0x55936c28fc50) at gmain.c:3829 #15 0x00007ff9044ed298 in g_main_context_iterate (context=0x55936c28fc50, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3902 #16 0x00007ff9044ed592 in g_main_loop_run (loop=0x55936c28fd90) at gmain.c:4098 #17 0x000055936b26f072 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:438 Unfortunately curl was updated in repo, so debuginfo I've installed is not matching library I currently have installed. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c8 --- Comment #8 from Martin Wilck <martin.wilck@suse.com> --- It'd be important (In reply to Vojtech Zeisek from comment #5)
(In reply to Martin Wilck from comment #4)
Can you correlate this to the update of NM to 1.12.2?
Well, there was also a recent libcurl4 update. Both could be the culprit, although I find libcurl4 highly unlikely (very minor update, and what could go wrong in curl_easy_init())? Likely candidate is NM commit d8a31794c8b9 "connectivity: rework async connectivity check requests". -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c9 --- Comment #9 from Martin Wilck <martin.wilck@suse.com> --- Upstream doesn't seem to have this reported yet. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c10 --- Comment #10 from Adam Mizerski <adam@mizerski.pl> --- New one: #0 0x00007f95dd7c7232 in _int_malloc (av=av@entry=0x7f95ddafdc40 <main_arena>, bytes=bytes@entry=32768) at malloc.c:4028 #1 0x00007f95dd7c87ba in __GI___libc_malloc (bytes=bytes@entry=32768) at malloc.c:3065 #2 0x00007f95de4b58d1 in g_malloc (n_bytes=n_bytes@entry=32768) at gmem.c:99 #3 0x000055aefa54a013 in nl_recv (sk=sk@entry=0x55aefacbe980, nla=nla@entry=0x7ffc10e6f554, buf=buf@entry=0x7ffc10e6f538, creds=creds@entry=0x7ffc10e6f530) at src/platform/nm-netlink.c:1403 #4 0x000055aefa53c633 in event_handler_recvmsgs (platform=platform@entry=0x55aefaccd120, handle_events=handle_events@entry=1) at src/platform/nm-linux-platform.c:6584 #5 0x000055aefa54007d in event_handler_read_netlink (platform=platform@entry=0x55aefaccd120, wait_for_acks=wait_for_acks@entry=0) at src/platform/nm-linux-platform.c:6790 #6 0x000055aefa543d46 in do_add_addrroute (platform=platform@entry=0x55aefaccd120, obj_id=obj_id@entry=0x7ffc10e6fbf0, nlmsg=nlmsg@entry=0x55aefacce590, suppress_netlink_failure=suppress_netlink_failure@entry=0) at src/platform/nm-linux-platform.c:4566 #7 0x000055aefa54416a in ip6_address_add (platform=0x55aefaccd120, ifindex=3, addr=..., plen=<optimized out>, peer_addr=..., lifetime=<optimized out>, preferred=4294967295, flags=640) at src/platform/nm-linux-platform.c:6273 #8 0x000055aefa557098 in nm_platform_ip6_address_add (self=self@entry=0x55aefaccd120, ifindex=ifindex@entry=3, address=..., plen=<optimized out>, peer_address=..., lifetime=4294967295, preferred=4294967295, flags=640) at src/platform/nm-platform.c:3181 #9 0x000055aefa55b1d2 in nm_platform_ip6_address_sync () at src/platform/nm-platform.c:3798 #10 0x000055aefa53383c in nm_ip6_config_commit (self=self@entry=0x55aefad616e0, platform=0x55aefaccd120, route_table_sync=route_table_sync@entry=NM_IP_ROUTE_TABLE_SYNC_MODE_MAIN, out_temporary_not_available=out_temporary_not_available@entry=0x7ffc10e700a0) at src/nm-ip6-config.c:549 #11 0x000055aefa5d1c78 in nm_device_set_ip_config (self=self@entry=0x55aefad6b590, addr_family=addr_family@entry=10, new_config=new_config@entry=0x55aefad616e0, commit=commit@entry=1, ip4_dev_route_blacklist=0x0) at src/devices/nm-device.c:11397 #12 0x000055aefa5d3319 in ip_config_merge_and_apply (self=self@entry=0x55aefad6b590, addr_family=addr_family@entry=10, commit=commit@entry=1) at src/devices/nm-device.c:6785 #13 0x000055aefa5e65d5 in activate_stage5_ip6_config_commit (self=0x55aefad6b590) at src/devices/nm-device.c:9964 #14 0x000055aefa5c8e85 in activation_source_handle_cb (self=0x55aefad6b590, addr_family=addr_family@entry=10) at src/devices/nm-device.c:5684 #15 0x000055aefa5c902e in activation_source_handle_cb6 (user_data=<optimized out>) at src/devices/nm-device.c:5621 #16 0x00007f95de4afea8 in g_main_dispatch (context=0x55aefacbf3b0) at gmain.c:3176 #17 g_main_context_dispatch (context=context@entry=0x55aefacbf3b0) at gmain.c:3829 #18 0x00007f95de4b0298 in g_main_context_iterate (context=0x55aefacbf3b0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3902 #19 0x00007f95de4b0592 in g_main_loop_run (loop=0x55aefacbf4f0) at gmain.c:4098 #20 0x000055aefa4e8072 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:438 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c11 --- Comment #11 from Martin Wilck <martin.wilck@suse.com> --- Hm, another crash in malloc(). This usually means that the heap is corrupted by some prior double-free or similar. Could you try running NM under valgrind? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c12 --- Comment #12 from Adam Mizerski <adam@mizerski.pl> ---
Could you try running NM under valgrind?
Done. Waiting for results. There is a second issue though (maybe it should be as a separate bug): After NM restart, it's no longer being able to obtain passwords from wallet. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c13 --- Comment #13 from Adam Mizerski <adam@mizerski.pl> --- Invalid write of size 8 at 0x52267BE: curl_multi_remove_handle (in /usr/lib64/libcurl.so.4.5.0) by 0x283072: cb_data_free (nm-connectivity.c:208) by 0x2832EA: _con_curl_check_connectivity (nm-connectivity.c:270) by 0x28339C: _con_curl_socketevent_cb (nm-connectivity.c:359) by 0x4C72EA7: g_main_dispatch (gmain.c:3176) by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) by 0x4C73591: g_main_loop_run (gmain.c:4098) by 0x136071: main (main.c:438) Address 0x8d10420 is 0 bytes inside a block of size 21,336 free'd at 0x40319AB: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x5210900: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521ECAC: curl_easy_cleanup (in /usr/lib64/libcurl.so.4.5.0) by 0x28307B: cb_data_free (nm-connectivity.c:209) by 0x283C71: nm_connectivity_check_cancel (nm-connectivity.c:574) by 0x214AE8: concheck_handle_complete (nm-device.c:2601) by 0x222157: concheck_cb (nm-device.c:2725) by 0x28361F: easy_header_cb (nm-connectivity.c:424) by 0x520A1BB: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x520864F: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521CB87: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x5226D7A: ??? (in /usr/lib64/libcurl.so.4.5.0) Block was alloc'd at at 0x40327A5: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x5210C49: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521E9D3: curl_easy_init (in /usr/lib64/libcurl.so.4.5.0) by 0x283A74: nm_connectivity_check_start (nm-connectivity.c:528) by 0x2190EC: concheck_start (nm-device.c:2772) by 0x219222: concheck_periodic_schedule_set (nm-device.c:2439) by 0x21937D: concheck_periodic_timeout_cb (nm-device.c:2265) by 0x4C73932: g_timeout_dispatch (gmain.c:4649) by 0x4C72EA7: g_main_dispatch (gmain.c:3176) by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) by 0x4C73591: g_main_loop_run (gmain.c:4098) by 0x136071: main (main.c:438) Invalid read of size 4 at 0x5228471: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x52287C4: curl_multi_socket_action (in /usr/lib64/libcurl.so.4.5.0) by 0x283110: _con_curl_check_connectivity (nm-connectivity.c:249) by 0x28339C: _con_curl_socketevent_cb (nm-connectivity.c:359) by 0x4C72EA7: g_main_dispatch (gmain.c:3176) by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) by 0x4C73591: g_main_loop_run (gmain.c:4098) by 0x136071: main (main.c:438) Address 0x937f610 is 21,328 bytes inside a block of size 21,336 free'd at 0x40319AB: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x5210900: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521ECAC: curl_easy_cleanup (in /usr/lib64/libcurl.so.4.5.0) by 0x28307B: cb_data_free (nm-connectivity.c:209) by 0x283C71: nm_connectivity_check_cancel (nm-connectivity.c:574) by 0x214AE8: concheck_handle_complete (nm-device.c:2601) by 0x222157: concheck_cb (nm-device.c:2725) by 0x28361F: easy_header_cb (nm-connectivity.c:424) by 0x520A1BB: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x520864F: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521CB87: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x5226D7A: ??? (in /usr/lib64/libcurl.so.4.5.0) Block was alloc'd at at 0x40327A5: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x5210C49: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521E9D3: curl_easy_init (in /usr/lib64/libcurl.so.4.5.0) by 0x283A74: nm_connectivity_check_start (nm-connectivity.c:528) by 0x2190EC: concheck_start (nm-device.c:2772) by 0x219222: concheck_periodic_schedule_set (nm-device.c:2439) by 0x21937D: concheck_periodic_timeout_cb (nm-device.c:2265) by 0x4C73932: g_timeout_dispatch (gmain.c:4649) by 0x4C72EA7: g_main_dispatch (gmain.c:3176) by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) by 0x4C73591: g_main_loop_run (gmain.c:4098) by 0x136071: main (main.c:438) Invalid write of size 8 at 0x5225E76: curl_multi_add_handle (in /usr/lib64/libcurl.so.4.5.0) by 0x283B4A: nm_connectivity_check_start (nm-connectivity.c:541) by 0x2190EC: concheck_start (nm-device.c:2772) by 0x219222: concheck_periodic_schedule_set (nm-device.c:2439) by 0x21937D: concheck_periodic_timeout_cb (nm-device.c:2265) by 0x4C73932: g_timeout_dispatch (gmain.c:4649) by 0x4C72EA7: g_main_dispatch (gmain.c:3176) by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) by 0x4C73591: g_main_loop_run (gmain.c:4098) by 0x136071: main (main.c:438) Address 0x8d10420 is 0 bytes inside a block of size 21,336 free'd at 0x40319AB: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x5210900: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521ECAC: curl_easy_cleanup (in /usr/lib64/libcurl.so.4.5.0) by 0x28307B: cb_data_free (nm-connectivity.c:209) by 0x283C71: nm_connectivity_check_cancel (nm-connectivity.c:574) by 0x214AE8: concheck_handle_complete (nm-device.c:2601) by 0x222157: concheck_cb (nm-device.c:2725) by 0x28361F: easy_header_cb (nm-connectivity.c:424) by 0x520A1BB: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x520864F: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521CB87: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x5226D7A: ??? (in /usr/lib64/libcurl.so.4.5.0) Block was alloc'd at at 0x40327A5: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x5210C49: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521E9D3: curl_easy_init (in /usr/lib64/libcurl.so.4.5.0) by 0x283A74: nm_connectivity_check_start (nm-connectivity.c:528) by 0x2190EC: concheck_start (nm-device.c:2772) by 0x219222: concheck_periodic_schedule_set (nm-device.c:2439) by 0x21937D: concheck_periodic_timeout_cb (nm-device.c:2265) by 0x4C73932: g_timeout_dispatch (gmain.c:4649) by 0x4C72EA7: g_main_dispatch (gmain.c:3176) by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) by 0x4C73591: g_main_loop_run (gmain.c:4098) by 0x136071: main (main.c:438) Invalid write of size 8 at 0x52267BE: curl_multi_remove_handle (in /usr/lib64/libcurl.so.4.5.0) by 0x283072: cb_data_free (nm-connectivity.c:208) by 0x2835C6: _timeout_cb (nm-connectivity.c:473) by 0x4C73932: g_timeout_dispatch (gmain.c:4649) by 0x4C72EA7: g_main_dispatch (gmain.c:3176) by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) by 0x4C73591: g_main_loop_run (gmain.c:4098) by 0x136071: main (main.c:438) Address 0x8d10420 is 0 bytes inside a block of size 21,336 free'd at 0x40319AB: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x5210900: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521ECAC: curl_easy_cleanup (in /usr/lib64/libcurl.so.4.5.0) by 0x28307B: cb_data_free (nm-connectivity.c:209) by 0x283C71: nm_connectivity_check_cancel (nm-connectivity.c:574) by 0x214AE8: concheck_handle_complete (nm-device.c:2601) by 0x222157: concheck_cb (nm-device.c:2725) by 0x28361F: easy_header_cb (nm-connectivity.c:424) by 0x520A1BB: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x520864F: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521CB87: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x5226D7A: ??? (in /usr/lib64/libcurl.so.4.5.0) Block was alloc'd at at 0x40327A5: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x5210C49: ??? (in /usr/lib64/libcurl.so.4.5.0) by 0x521E9D3: curl_easy_init (in /usr/lib64/libcurl.so.4.5.0) by 0x283A74: nm_connectivity_check_start (nm-connectivity.c:528) by 0x2190EC: concheck_start (nm-device.c:2772) by 0x219222: concheck_periodic_schedule_set (nm-device.c:2439) by 0x21937D: concheck_periodic_timeout_cb (nm-device.c:2265) by 0x4C73932: g_timeout_dispatch (gmain.c:4649) by 0x4C72EA7: g_main_dispatch (gmain.c:3176) by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) by 0x4C73591: g_main_loop_run (gmain.c:4098) by 0x136071: main (main.c:438) Invalid write of size 8 at 0x52267BE: curl_multi_remove_handle (in /usr/lib64/libcurl.so.4.5.0) by 0x283072: cb_data_free (nm-connectivity.c:208) by 0x283284: _con_curl_check_connectivity (nm-connectivity.c:275) by 0x28339C: _con_curl_socketevent_cb (nm-connectivity.c:359) by 0x4C72EA7: g_main_dispatch (gmain.c:3176) by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) by 0x4C73591: g_main_loop_run (gmain.c:4098) by 0x136071: main (main.c:438) Address 0xd1ab620 is 21,600 bytes inside a block of size 32,768 free'd at 0x40319AB: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x18A932: _nm_auto_free_impl (nm-macros-internal.h:76) by 0x18A932: event_handler_recvmsgs (nm-linux-platform.c:6579) by 0x18E07C: event_handler_read_netlink (nm-linux-platform.c:6790) by 0x18F09C: delayed_action_handle_WAIT_FOR_NL_RESPONSE (nm-linux-platform.c:3594) by 0x18F09C: delayed_action_handle_one (nm-linux-platform.c:3669) by 0x18F09C: delayed_action_handle_all (nm-linux-platform.c:3687) by 0x191D78: do_add_addrroute (nm-linux-platform.c:4577) by 0x192169: ip6_address_add (nm-linux-platform.c:6273) by 0x1A5097: nm_platform_ip6_address_add (nm-platform.c:3181) by 0x1A91D1: nm_platform_ip6_address_sync (nm-platform.c:3798) by 0x18183B: nm_ip6_config_commit (nm-ip6-config.c:549) by 0x21FC77: nm_device_set_ip_config (nm-device.c:11397) by 0x221318: ip_config_merge_and_apply (nm-device.c:6785) by 0x2345D4: activate_stage5_ip6_config_commit (nm-device.c:9964) Block was alloc'd at at 0x403077F: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x4C788D0: g_malloc (gmem.c:99) by 0x198012: nl_recv (nm-netlink.c:1403) by 0x18A632: event_handler_recvmsgs (nm-linux-platform.c:6584) by 0x18E07C: event_handler_read_netlink (nm-linux-platform.c:6790) by 0x18F09C: delayed_action_handle_WAIT_FOR_NL_RESPONSE (nm-linux-platform.c:3594) by 0x18F09C: delayed_action_handle_one (nm-linux-platform.c:3669) by 0x18F09C: delayed_action_handle_all (nm-linux-platform.c:3687) by 0x191D78: do_add_addrroute (nm-linux-platform.c:4577) by 0x192169: ip6_address_add (nm-linux-platform.c:6273) by 0x1A5097: nm_platform_ip6_address_add (nm-platform.c:3181) by 0x1A91D1: nm_platform_ip6_address_sync (nm-platform.c:3798) by 0x18183B: nm_ip6_config_commit (nm-ip6-config.c:549) by 0x21FC77: nm_device_set_ip_config (nm-device.c:11397) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c14 Yifan Jiang <yfjiang@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bjorn.lie@gmail.com, | |dimstar@opensuse.org --- Comment #14 from Yifan Jiang <yfjiang@suse.com> --- I saw the issue and the crash stack on both of my workstations, it feels disturbing since the computer's network is intermittently down during the crash. And the crashes can be continuously happening in a short intervals. I am using a reverted NM to trying to make the workstation usable, the network appeared better in the past 24 hours, while I may need to observe for more days to conclude anything useful: https://build.opensuse.org/package/show/home:yfjiang:branches:GNOME:Factory/... Regarding the impact, can we consider to revert the NM components before this issue is settled, especially the issue has not been reported (assuming not reproduced) in upstream? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c15 Martin Wilck <martin.wilck@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- URL| |https://bugzilla.gnome.org/ | |show_bug.cgi?id=797136 --- Comment #15 from Martin Wilck <martin.wilck@suse.com> --- I've filed a bug upstream, as we have only very few very harmless patches for NM in factory. https://bugzilla.gnome.org/show_bug.cgi?id=797136 On comment 13: All but the last trace shows libcurl4-internal callbacks accessing data freed in nm_connectivity_check_cancel(). The last one also occurs in a libcurl callback related to connectivity checking, but accesses a block freed elsewhere. That kind-of confirms my suspicion from comment 8 - the new asynchronous connectivity checking code seems to be susceptible to race conditions, in particular if nm_connectivity_check_cancel() is called. Unfortunately I've no time to dig into that deeper now. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c16 --- Comment #16 from Martin Wilck <martin.wilck@suse.com> --- Upstream is asking for the libcurl version. Please report. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c17 Martin Wilck <martin.wilck@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |adam@mizerski.pl Flags| |needinfo?(adam@mizerski.pl) --- Comment #17 from Martin Wilck <martin.wilck@suse.com> --- Please check if this also occurs with libcurl4 from devel:libraries:c_c++:
https://download.opensuse.org/repositories/devel:/libraries:/c_c++/openSUSE_...
-- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c18 --- Comment #18 from Martin Wilck <martin.wilck@suse.com> --- Arggh, sorry, you probably already have libcurl4-7.61.1-1.1.x86_64. It's already in factory. Please confirm. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c19 --- Comment #19 from Vojtech Zeisek <Vojtech.Zeisek@opensuse.org> --- Yes, I have libcurl4-7.61.1-1.1.x86_64. I did observe the problem mostly during weekend and on the beginning of the week. Now it is better. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c20 --- Comment #20 from Adam Mizerski <adam@mizerski.pl> --- I had 7.61.0. I'm updating to latest TW right now, so we'll see how it goes. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c21 --- Comment #21 from Martin Wilck <martin.wilck@suse.com> --- (In reply to Adam Mizerski from comment #20)
I had 7.61.0. I'm updating to latest TW right now, so we'll see how it goes.
7.61.0 should be fine. At least it contains the commit that upstream talked about. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c22 --- Comment #22 from Adam Mizerski <adam@mizerski.pl> --- valgrind: m_mallocfree.c:280 (mk_plain_bszB): Assertion 'bszB != 0' failed. valgrind: This is probably caused by your program erroneously writing past the end of a heap block and corrupting heap metadata. If you fix any invalid writes reported by Memcheck, this assertion failure will probably go away. Please try that before reporting this as a bug. host stacktrace: ==1533== at 0x5803D754: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux) ==1533== by 0x5803D864: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux) ==1533== by 0x5803D9E9: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux) ==1533== by 0x5804C00C: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux) ==1533== by 0x58004B83: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux) ==1533== by 0x58005097: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux) ==1533== by 0x58005228: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux) ==1533== by 0x5809B0CD: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux) ==1533== by 0x580AACB0: ??? (in /usr/lib64/valgrind/memcheck-amd64-linux) sched status: running_tid=1 Thread 1: status = VgTs_Runnable (lwpid 1533) ==1533== at 0x403077F: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==1533== by 0x4C788D0: g_malloc (gmem.c:99) ==1533== by 0x4C91C5B: g_memdup (gstrfuncs.c:391) ==1533== by 0x4C4C6C6: g_bytes_new (gbytes.c:97) ==1533== by 0x4CA7F43: g_variant_new_from_trusted (gvariant.c:320) ==1533== by 0x4CA801E: g_variant_new_byte (gvariant.c:410) ==1533== by 0x4ADFD4F: g_dbus_gvalue_to_gvariant (in /usr/lib64/libgio-2.0.so.0.5600.2) ==1533== by 0x174BD3: nm_dbus_utils_get_property (nm-dbus-utils.c:104) ==1533== by 0x141026: _obj_get_property (nm-dbus-manager.c:897) ==1533== by 0x142A0F: _nm_dbus_manager_obj_notify (nm-dbus-manager.c:1211) ==1533== by 0x1744C7: dispatch_properties_changed (nm-dbus-object.c:231) ==1533== by 0x4BE67F0: g_object_notify_by_pspec (in /usr/lib64/libgobject-2.0.so.0.5600.2) ==1533== by 0xB0D6F92: _notify (nm-wifi-ap.c:44) ==1533== by 0xB0D6F92: nm_wifi_ap_set_strength (nm-wifi-ap.c:289) ==1533== by 0xB0CED22: periodic_update (nm-device-wifi.c:438) ==1533== by 0xB0CED68: periodic_update_cb (nm-device-wifi.c:457) ==1533== by 0x4C73932: g_timeout_dispatch (gmain.c:4649) ==1533== by 0x4C72EA7: g_main_dispatch (gmain.c:3176) ==1533== by 0x4C72EA7: g_main_context_dispatch (gmain.c:3829) ==1533== by 0x4C73297: g_main_context_iterate.isra.26 (gmain.c:3902) ==1533== by 0x4C73591: g_main_loop_run (gmain.c:4098) ==1533== by 0x136071: main (main.c:438) Thread 2: status = VgTs_WaitSys (lwpid 2081) ==1533== at 0x57B12C9: poll (poll.c:29) ==1533== by 0x4C73205: g_main_context_poll (gmain.c:4203) ==1533== by 0x4C73205: g_main_context_iterate.isra.26 (gmain.c:3897) ==1533== by 0x4C7332B: g_main_context_iteration (gmain.c:3963) ==1533== by 0x4C73370: glib_worker_main (gmain.c:5772) ==1533== by 0x4C9AB74: g_thread_proxy (gthread.c:784) ==1533== by 0x54A9553: start_thread (pthread_create.c:463) ==1533== by 0x57BBCCE: clone (clone.S:95) Thread 4: status = VgTs_WaitSys (lwpid 2198) ==1533== at 0x57B12C9: poll (poll.c:29) ==1533== by 0x4C73205: g_main_context_poll (gmain.c:4203) ==1533== by 0x4C73205: g_main_context_iterate.isra.26 (gmain.c:3897) ==1533== by 0x4C73591: g_main_loop_run (gmain.c:4098) ==1533== by 0x4AFDA45: ??? (in /usr/lib64/libgio-2.0.so.0.5600.2) ==1533== by 0x4C9AB74: g_thread_proxy (gthread.c:784) ==1533== by 0x54A9553: start_thread (pthread_create.c:463) ==1533== by 0x57BBCCE: clone (clone.S:95) Note: see also the FAQ in the source distribution. It contains workarounds to several common problems. In particular, if Valgrind aborted or crashed after identifying problems in your program, there's a good chance that fixing those problems will prevent Valgrind aborting or crashing, especially if it happened in m_mallocfree.c. If that doesn't help, please report this bug to: www.valgrind.org In the bug report, send all the above text, the valgrind version, and what OS and version you are using. Thanks. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c23 --- Comment #23 from Adam Mizerski <adam@mizerski.pl> --- This thing in comment #22 is not related to curl, so probably should go to a separate bug report -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c24 Adam Mizerski <adam@mizerski.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED Flags|needinfo?(adam@mizerski.pl) | --- Comment #24 from Adam Mizerski <adam@mizerski.pl> --- With libcurl4-7.61.1 it seems to be stable. I'm marking this as resolved, but I'll keep running NM under valgrind for a few days. If anything happens, I'll reopen this ticket. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c25 Adam Mizerski <adam@mizerski.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED |--- Assignee|bnc-team-screening@forge.pr |adam@mizerski.pl |ovo.novell.com | --- Comment #25 from Adam Mizerski <adam@mizerski.pl> --- Created attachment 783482 --> http://bugzilla.opensuse.org/attachment.cgi?id=783482&action=edit NM log with curl 7.61.1 I had NM crashed again, with the same backtrace as in comment #0. Earlier there were reports from valgrind, the same as in comment #13 and comment #22. I have NetworkManager-1.12.2-1.2 and libcurl4-7.61.1-1.1 (TW snapshot 20180911). I'm attaching full log output, and here's the backtrace, just in case: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007f177cbed4e9 in __GI_abort () at abort.c:79 #2 0x00007f177cc469a7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f177cd504c2 "%s\n") at ../sysdeps/posix/libc_fatal.c:181 #3 0x00007f177cc4ce9c in malloc_printerr (str=str@entry=0x7f177cd4e62b "corrupted double-linked list") at malloc.c:5350 #4 0x00007f177cc4d0e4 in malloc_consolidate (av=av@entry=0x7f177cf86c40 <main_arena>) at malloc.c:4456 #5 0x00007f177cc4e93c in _int_free (av=0x7f177cf86c40 <main_arena>, p=0x55c773ec26c0, have_lock=<optimized out>) at malloc.c:4362 #6 0x00007f177d3d0a0b in ?? () from /usr/lib64/libcurl.so.4 #7 0x00007f177d3df02d in curl_easy_cleanup () from /usr/lib64/libcurl.so.4 #8 0x000055c772f6207c in cb_data_free (cb_data=0x55c773e23c60, state=NM_CONNECTIVITY_UNKNOWN, error=0x0, log_message=0x0) at src/nm-connectivity.c:209 #9 0x000055c772f622eb in _con_curl_check_connectivity (mhandle=0x55c773dca350, sockfd=<optimized out>, ev_bitmask=<optimized out>) at src/nm-connectivity.c:270 #10 0x000055c772f6239d in _con_curl_socketevent_cb (ch=0x55c773e8ed80, condition=<optimized out>, user_data=0x55c773df9ee0) at src/nm-connectivity.c:359 #11 0x00007f177d938ea8 in g_main_dispatch (context=0x55c773d77380) at gmain.c:3176 #12 g_main_context_dispatch (context=context@entry=0x55c773d77380) at gmain.c:3829 #13 0x00007f177d939298 in g_main_context_iterate (context=0x55c773d77380, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3902 #14 0x00007f177d939592 in g_main_loop_run (loop=0x55c773d774c0) at gmain.c:4098 #15 0x000055c772e15072 in main (argc=<optimized out>, argv=<optimized out>) at src/main.c:438 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c31 --- Comment #31 from Karl Cheng <qantas94heavy@gmail.com> --- The update to 1.14.0 in GNOME:Factory appears to have regressed this issue, with crashes occurring again. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c32 --- Comment #32 from Michiel Janssens <michiel@nexigon.net> --- Still running NetworkManager-1.12.2-1.2 on snapshot 20181009. After disabling conn check as described in https://bugzilla.gnome.org/show_bug.cgi?id=797136#c1 the core-dumps of networkmanager don't happen anymore or me. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c33 --- Comment #33 from Michiel Janssens <michiel@nexigon.net> --- The core dumps are still happening with nm 1.14.0 on snapshot 20181018 with conn check enabled. Disabling conn check keeps the core dumps from happening. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 Michal Hlavac <miso@hlavki.eu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |miso@hlavki.eu -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c34 --- Comment #34 from Jonathan Kang <sckang@suse.com> --- The fix somehow is not included in version 1.14.0. Version 1.14.2 which include the fix is available in GNOME:Factory right now, but not yet[1] in openSUSE:Factory(available for users to update). In the meantime, you can use the method mentioned in https://bugzilla.gnome.org/show_bug.cgi?id=797136#c1 as a "workaround". Or you can use NM 1.14.2 in GNOME:Factory. Anyway the new version should be available soon. *[1] https://build.opensuse.org/request/show/644486 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c35 Jonathan Kang <sckang@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution|--- |FIXED --- Comment #35 from Jonathan Kang <sckang@suse.com> --- NetworkManager 1.14.4 is available in openSUSE:Factory. If you're still using old NM version and suffering this issue, please make sure to update to 1.14.4 which includes the fix. Closing this as RESOLVED FIXED. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1107197 http://bugzilla.opensuse.org/show_bug.cgi?id=1107197#c36 Dominique Leuenberger <dimstar@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |develop7@develop7.info --- Comment #36 from Dominique Leuenberger <dimstar@opensuse.org> --- *** Bug 1108619 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are on the CC list for the bug.
participants (2)
-
bugzilla_noreply＠novell.com -
bugzilla_noreply＠suse.com