[opensuse-kernel] depmod pointer error in Kernel:HEAD aarch64
Hello, Kernel:HEAD aarch64 v4.9-rc1 kernel builds all crashed in depmod. A quick Google search didn't reveal any known bug. Now there may be a genuine upstream dependency bug triggering this, but surely depmod shouldn't behave like this. armv7hl and armv6hl also failed, but with module CRC warnings. Also very weird... aarch64: [15224s] DEPMOD 4.9.0-rc1-2.g11efe27-default [15318s] depmod: ERROR: Found 8 modules in dependency cycles! [15318s] depmod: ERROR: Cycle detected: remoteproc -> virtio [15318s] *** Error in `/sbin/depmod': free(): invalid pointer: 0x00000000143f9380 *** [15318s] ======= Backtrace: ========= [15318s] /lib64/libc.so.6(+0x6a884)[0xffff8cf19884] [15318s] /lib64/libc.so.6(+0x70d9c)[0xffff8cf1fd9c] [15318s] /lib64/libc.so.6(+0x73b40)[0xffff8cf22b40] [15318s] /sbin/depmod[0x4096ec] [15318s] /lib64/libc.so.6(__libc_start_main+0xdc)[0xffff8cece364] [15318s] /sbin/depmod[0x402cd8] [15318s] ======= Memory map: ======== [15318s] 00400000-00422000 r-xp 00000000 fd:10 4065780 /usr/bin/kmod [15318s] 0043f000-00440000 r--p 0002f000 fd:10 4065780 /usr/bin/kmod [15318s] 00440000-00441000 rw-p 00030000 fd:10 4065780 /usr/bin/kmod [15318s] 13f31000-157ae000 rw-p 00000000 00:00 0 [heap] [15318s] ffff8ce49000-ffff8ce5b000 r-xp 00000000 fd:10 131208 /lib64/libgcc_s.so.1 [15318s] ffff8ce5b000-ffff8ce78000 ---p 00012000 fd:10 131208 /lib64/libgcc_s.so.1 [15318s] ffff8ce78000-ffff8ce79000 r--p 0001f000 fd:10 131208 /lib64/libgcc_s.so.1 [15318s] ffff8ce79000-ffff8ce7a000 rw-p 00020000 fd:10 131208 /lib64/libgcc_s.so.1 [15318s] ffff8ce7a000-ffff8ce93000 r-xp 00000000 fd:10 131108 /lib64/libpthread-2.24.so [15318s] ffff8ce93000-ffff8cea9000 ---p 00019000 fd:10 131108 /lib64/libpthread-2.24.so [15318s] ffff8cea9000-ffff8ceaa000 r--p 0001f000 fd:10 131108 /lib64/libpthread-2.24.so [15318s] ffff8ceaa000-ffff8ceab000 rw-p 00020000 fd:10 131108 /lib64/libpthread-2.24.so [15318s] ffff8ceab000-ffff8ceaf000 rw-p 00000000 00:00 0 [15318s] ffff8ceaf000-ffff8cff9000 r-xp 00000000 fd:10 131080 /lib64/libc-2.24.so [15318s] ffff8cff9000-ffff8d00b000 ---p 0014a000 fd:10 131080 /lib64/libc-2.24.so [15318s] ffff8d00b000-ffff8d00f000 r--p 0014c000 fd:10 131080 /lib64/libc-2.24.so [15318s] ffff8d00f000-ffff8d011000 rw-p 00150000 fd:10 131080 /lib64/libc-2.24.so [15318s] ffff8d011000-ffff8d015000 rw-p 00000000 00:00 0 [15318s] ffff8d015000-ffff8d029000 r-xp 00000000 fd:10 131122 /lib64/libz.so.1.2.8 [15318s] ffff8d029000-ffff8d044000 ---p 00014000 fd:10 131122 /lib64/libz.so.1.2.8 [15318s] ffff8d044000-ffff8d045000 r--p 0001f000 fd:10 131122 /lib64/libz.so.1.2.8 [15318s] ffff8d045000-ffff8d046000 rw-p 00020000 fd:10 131122 /lib64/libz.so.1.2.8 [15318s] ffff8d046000-ffff8d069000 r-xp 00000000 fd:10 4063788 /usr/lib64/liblzma.so.5.2.2 [15318s] ffff8d069000-ffff8d085000 ---p 00023000 fd:10 4063788 /usr/lib64/liblzma.so.5.2.2 [15318s] ffff8d085000-ffff8d086000 r--p 0002f000 fd:10 4063788 /usr/lib64/liblzma.so.5.2.2 [15318s] ffff8d086000-ffff8d087000 rw-p 00030000 fd:10 4063788 /usr/lib64/liblzma.so.5.2.2 [15318s] ffff8d087000-ffff8d0a5000 r-xp 00000000 fd:10 131195 /lib64/ld-2.24.so [15318s] ffff8d0ac000-ffff8d0ae000 rw-p 00000000 00:00 0 [15318s] ffff8d0b1000-ffff8d0b4000 rw-p 00000000 00:00 0 [15318s] ffff8d0b4000-ffff8d0b5000 r--p 00000000 00:00 0 [vvar] [15318s] ffff8d0b5000-ffff8d0b6000 r-xp 00000000 00:00 0 [vdso] [15318s] ffff8d0b6000-ffff8d0b7000 r--p 0001f000 fd:10 131195 /lib64/ld-2.24.so [15318s] ffff8d0b7000-ffff8d0b8000 rw-p 00020000 fd:10 131195 /lib64/ld-2.24.so [15318s] ffff8d0b8000-ffff8d0b9000 rw-p 00000000 00:00 0 [15318s] ffffdcb0f000-ffffdcb31000 rw-p 00000000 00:00 0 [stack] [15318s] ../scripts/depmod.sh: line 57: 7427 Aborted (core dumped) "$DEPMOD" "$@" "$KERNELRELEASE" $SYMBOL_PREFIX [15318s] make[2]: *** [/home/abuild/rpmbuild/BUILD/kernel-default-4.9.rc1/linux-4.9-rc1/Makefile:1249: _modinst_post] Error 134 [15318s] make[1]: *** [Makefile:150: sub-make] Error 2 [15318s] make: *** [Makefile:24: __sub-make] Error 2 [15318s] error: Bad exit status from /var/tmp/rpm-tmp.WEXxgf (%install) armv7hl: [18637s] MODPOST 3846 modules [18991s] WARNING: "__gnu_mcount_nc" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "__memzero" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "memset" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "memcpy" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "_set_bit" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "_test_and_set_bit" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "_clear_bit" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "__aeabi_uidivmod" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "_test_and_clear_bit" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__gnu_mcount_nc" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__aeabi_uidiv" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__memzero" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "memset" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__put_user_1" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__aeabi_idiv" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__put_user_4" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "memcpy" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "_set_bit" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "_test_and_set_bit" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "_clear_bit" [sound/usb/snd-usb-audio.ko] has no CRC! [...] [19011s] make[3]: *** [../scripts/Makefile.modpost:94: __modpost] Error 1 [19011s] make[2]: *** [/home/abuild/rpmbuild/BUILD/kernel-default-4.9.rc1/linux-4.9-rc1/Makefile:1211: modules] Error 2 [19011s] make[1]: *** [Makefile:150: sub-make] Error 2 [19011s] make: *** [Makefile:24: __sub-make] Error 2 Anyone any ideas? Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On Okt 24 2016, Andreas Färber
armv7hl:
[18637s] MODPOST 3846 modules [18991s] WARNING: "__gnu_mcount_nc" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "__memzero" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "memset" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "memcpy" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "_set_bit" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "_test_and_set_bit" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "_clear_bit" [sound/usb/snd-usbmidi-lib.ko] has no CRC! [18991s] WARNING: "__aeabi_uidivmod" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "_test_and_clear_bit" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__gnu_mcount_nc" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__aeabi_uidiv" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__memzero" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "memset" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__put_user_1" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__aeabi_idiv" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "__put_user_4" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "memcpy" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "_set_bit" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "_test_and_set_bit" [sound/usb/snd-usb-audio.ko] has no CRC! [18991s] WARNING: "_clear_bit" [sound/usb/snd-usb-audio.ko] has no CRC!
This is expected, see commit 84d69848c9. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Am 24.10.2016 um 12:48 schrieb Andreas Färber:
Hello,
Kernel:HEAD aarch64 v4.9-rc1 kernel builds all crashed in depmod. A quick Google search didn't reveal any known bug.
Now there may be a genuine upstream dependency bug triggering this, but surely depmod shouldn't behave like this. [...]
With 4.9-rc3 kernel.git I locally get the following on aarch64: depmod: ERROR: Found 8 modules in dependency cycles! depmod: ERROR: Cycle detected: remoteproc -> virtio depmod: ERROR: Cycle detected: remoteproc -> virtio_ring depmod: ERROR: Cycle detected: qcom_mdt_loader -> remoteproc *** Error in `/sbin/depmod': free(): invalid next size (fast): 0x000000000186d770 *** [...] (gdb) bt #0 0x0000007fb7e28c80 in raise () from /lib64/libc.so.6 #1 0x0000007fb7e2a04c in abort () from /lib64/libc.so.6 #2 0x0000007fb7e61888 in __libc_message () from /lib64/libc.so.6 #3 0x0000007fb7e67d9c in malloc_printerr () from /lib64/libc.so.6 #4 0x0000007fb7e68638 in _int_free () from /lib64/libc.so.6 #5 0x00000000004096ec in depmod_report_cycles (depmod=0x7fffffa970, edges=0xede, stack=0xf00, users=0x188b4f0, n_roots=43920, n_mods=127) at tools/depmod.c:1519 #6 depmod_calculate_dependencies (depmod=0x7fffffa970) at tools/depmod.c:1596 #7 depmod_load (depmod=0x7fffffa970) at tools/depmod.c:1623 #8 do_depmod (argc=<optimized out>, argv=<optimized out>) at tools/depmod.c:2598 #9 0x0000007fb7e16364 in __libc_start_main () from /lib64/libc.so.6 #10 0x0000000000402cd8 in _start () at ../sysdeps/aarch64/start.S:81 Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) If I comment out the offending free() in vanilla kmod-23's depmod_report_cycles() then I see the first "real" cycle: depmod: ERROR: Found 8 modules in dependency cycles! depmod: ERROR: Cycle detected: remoteproc -> virtio depmod: ERROR: Cycle detected: remoteproc -> virtio_ring depmod: ERROR: Cycle detected: qcom_mdt_loader -> remoteproc depmod: ERROR: Cycle detected: qcom_wcnss_iris -> qcom_wcnss -> qcom_wcnss_iris *** Error in `/home/andreas/kmod-23/tools/depmod': realloc(): invalid next size: 0x000000002640ab30 *** Changing the DBG() to an ERR() hints that the cycle detection is flawed: depmod: ERROR: Found 8 modules in dependency cycles! depmod: ERROR: Cycle report: Trying smem visited=0 users=1 depmod: ERROR: Cycle report: Trying hwspinlock_core visited=0 users=0 depmod: ERROR: Cycle report: Trying virtio visited=0 users=1 depmod: ERROR: Cycle report: Trying virtio_ring visited=0 users=1 depmod: ERROR: Cycle report: Trying remoteproc visited=0 users=2 depmod: ERROR: Cycle report: Trying virtio visited=1 users=0 depmod: ERROR: Cycle detected: remoteproc -> virtio depmod: ERROR: Cycle report: Trying virtio_ring visited=1 users=0 depmod: ERROR: Cycle detected: remoteproc -> virtio_ring depmod: ERROR: Cycle report: Trying qcom_mdt_loader visited=0 users=1 depmod: ERROR: Cycle report: Trying remoteproc visited=1 users=1 depmod: ERROR: Cycle detected: qcom_mdt_loader -> remoteproc *** Error in `/home/andreas/kmod-23/tools/depmod': free(): invalid next size (fast): 0x000000002b61c370 *** I.e., whenever a module has been visited as part of a (non-circular) dependency chain it gets marked as ->visited = true and the next time it is encountered it is falsely reported as a cycle. Whether that misdetection causes any corruption is not yet clear to me, but it explains the first three "Cycle detected" lines. If I locally disable CONFIG_QCOM_WCNSS_PIL then no cycles get detected and it doesn't crash, finishing fine. But again, that's only a solution for getting some kernel built, not for getting sensible information on the next cycle... armv6hl has in the meantime succeeded building; armv7hl still faces this depmod issue. Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hi Andreas, On 11/01/2016 08:55 PM, Andreas Färber wrote:
Am 24.10.2016 um 12:48 schrieb Andreas Färber:
Hello,
Kernel:HEAD aarch64 v4.9-rc1 kernel builds all crashed in depmod. A quick Google search didn't reveal any known bug.
Now there may be a genuine upstream dependency bug triggering this, but surely depmod shouldn't behave like this. [...]
With 4.9-rc3 kernel.git I locally get the following on aarch64:
depmod: ERROR: Found 8 modules in dependency cycles! depmod: ERROR: Cycle detected: remoteproc -> virtio depmod: ERROR: Cycle detected: remoteproc -> virtio_ring depmod: ERROR: Cycle detected: qcom_mdt_loader -> remoteproc *** Error in `/sbin/depmod': free(): invalid next size (fast): 0x000000000186d770 *** [...] (gdb) bt #0 0x0000007fb7e28c80 in raise () from /lib64/libc.so.6 #1 0x0000007fb7e2a04c in abort () from /lib64/libc.so.6 #2 0x0000007fb7e61888 in __libc_message () from /lib64/libc.so.6 #3 0x0000007fb7e67d9c in malloc_printerr () from /lib64/libc.so.6 #4 0x0000007fb7e68638 in _int_free () from /lib64/libc.so.6 #5 0x00000000004096ec in depmod_report_cycles (depmod=0x7fffffa970, edges=0xede, stack=0xf00, users=0x188b4f0, n_roots=43920, n_mods=127) at tools/depmod.c:1519 #6 depmod_calculate_dependencies (depmod=0x7fffffa970) at tools/depmod.c:1596 #7 depmod_load (depmod=0x7fffffa970) at tools/depmod.c:1623 #8 do_depmod (argc=<optimized out>, argv=<optimized out>) at tools/depmod.c:2598 #9 0x0000007fb7e16364 in __libc_start_main () from /lib64/libc.so.6 #10 0x0000000000402cd8 in _start () at ../sysdeps/aarch64/start.S:81 Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)
If I comment out the offending free() in vanilla kmod-23's depmod_report_cycles() then I see the first "real" cycle:
depmod: ERROR: Found 8 modules in dependency cycles! depmod: ERROR: Cycle detected: remoteproc -> virtio depmod: ERROR: Cycle detected: remoteproc -> virtio_ring depmod: ERROR: Cycle detected: qcom_mdt_loader -> remoteproc depmod: ERROR: Cycle detected: qcom_wcnss_iris -> qcom_wcnss -> qcom_wcnss_iris *** Error in `/home/andreas/kmod-23/tools/depmod': realloc(): invalid next size: 0x000000002640ab30 ***
Changing the DBG() to an ERR() hints that the cycle detection is flawed:
depmod: ERROR: Found 8 modules in dependency cycles! depmod: ERROR: Cycle report: Trying smem visited=0 users=1 depmod: ERROR: Cycle report: Trying hwspinlock_core visited=0 users=0 depmod: ERROR: Cycle report: Trying virtio visited=0 users=1 depmod: ERROR: Cycle report: Trying virtio_ring visited=0 users=1 depmod: ERROR: Cycle report: Trying remoteproc visited=0 users=2 depmod: ERROR: Cycle report: Trying virtio visited=1 users=0 depmod: ERROR: Cycle detected: remoteproc -> virtio depmod: ERROR: Cycle report: Trying virtio_ring visited=1 users=0 depmod: ERROR: Cycle detected: remoteproc -> virtio_ring depmod: ERROR: Cycle report: Trying qcom_mdt_loader visited=0 users=1 depmod: ERROR: Cycle report: Trying remoteproc visited=1 users=1 depmod: ERROR: Cycle detected: qcom_mdt_loader -> remoteproc *** Error in `/home/andreas/kmod-23/tools/depmod': free(): invalid next size (fast): 0x000000002b61c370 ***
I.e., whenever a module has been visited as part of a (non-circular) dependency chain it gets marked as ->visited = true and the next time it is encountered it is falsely reported as a cycle. Whether that misdetection causes any corruption is not yet clear to me, but it explains the first three "Cycle detected" lines.
If I locally disable CONFIG_QCOM_WCNSS_PIL then no cycles get detected and it doesn't crash, finishing fine. But again, that's only a solution for getting some kernel built, not for getting sensible information on the next cycle...
Would you mind to try the patch attached? I suppose this fixes the building problem. I'm traveling and my internet connection isn't really reliable. Thanks, Matthias
armv6hl has in the meantime succeeded building; armv7hl still faces this depmod issue.
Regards, Andreas
Hi Matthias, Am 02.11.2016 um 18:13 schrieb Matthias Brugger:
On 11/01/2016 08:55 PM, Andreas Färber wrote:
Am 24.10.2016 um 12:48 schrieb Andreas Färber: If I locally disable CONFIG_QCOM_WCNSS_PIL then no cycles get detected and it doesn't crash, finishing fine. But again, that's only a solution for getting some kernel built, not for getting sensible information on the next cycle...
Would you mind to try the patch attached? I suppose this fixes the building problem. [...]
Thanks for looking into this. On master branch I already reverted the config option to quickly un-block us. I don't think it blocks any enablement. Bjorn is looking into the cycle and said this on ##linux-msm: <bamse> oh no... vinod merged peter's patch to change remoteproc to be user selectable hmm, no that's not it <bamse> ohh, never mind, it is broken...by my design sorry about that i have exported functions in both directions...how silly <bamse> afaerber: thanks for the report, i'll clean it up So the cycle is being taken care of, now that I've identified it. Instead I'd appreciate if anyone can help fix the depmod crash (kmod), so that next time there's a cycle we can see which module is causing the error and avoid having broken kernels for weeks and needing to locally hack up kmod to find out by chance what's wrong. Cheers, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Am 02.11.2016 um 19:04 schrieb Andreas Färber:
So the cycle is being taken care of, now that I've identified it.
Instead I'd appreciate if anyone can help fix the depmod crash (kmod), so that next time there's a cycle we can see which module is causing the error and avoid having broken kernels for weeks and needing to locally hack up kmod to find out by chance what's wrong.
https://bugzilla.opensuse.org/show_bug.cgi?id=1008186
Cheers, Andreas
-- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Am 02.11.2016 um 19:04 schrieb Andreas Färber:
Hi Matthias,
Am 02.11.2016 um 18:13 schrieb Matthias Brugger:
On 11/01/2016 08:55 PM, Andreas Färber wrote:
Am 24.10.2016 um 12:48 schrieb Andreas Färber: If I locally disable CONFIG_QCOM_WCNSS_PIL then no cycles get detected and it doesn't crash, finishing fine. But again, that's only a solution for getting some kernel built, not for getting sensible information on the next cycle...
Would you mind to try the patch attached? I suppose this fixes the building problem. [...]
Thanks for looking into this. On master branch I already reverted the config option to quickly un-block us. I don't think it blocks any enablement.
Bjorn is looking into the cycle and said this on ##linux-msm:
<bamse> oh no... vinod merged peter's patch to change remoteproc to be user selectable hmm, no that's not it
<bamse> ohh, never mind, it is broken...by my design sorry about that i have exported functions in both directions...how silly <bamse> afaerber: thanks for the report, i'll clean it up
So the cycle is being taken care of, now that I've identified it.
https://patchwork.kernel.org/patch/9411771/ Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
participants (3)
-
Andreas Färber -
Andreas Schwab -
Matthias Brugger