http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c2 --- Comment #2 from Stefan Seyfried --- Does kdump even work on raspberry pi 4? Last I tried (some years ago) it was basically x86_64 only ;-) But yes, I'll try. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c4 --- Comment #4 from Stefan Seyfried --- kdump works (at least after a clean boot ;-) on raspi4, I'm setting it up on the second machine now. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c7 --- Comment #7 from Stefan Seyfried --- first it looked as if enabling kdump worked around the issue, but today the machine crashed 4 times. I've uploaded the crashdumps to google drive: https://drive.google.com/drive/folders/1CLbnST5CeNA1kGLt8SEm7yZ8KM1WuIbc?usp... I'm also trying to install kernel-default-debuginfo and the crash tool. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c9 --- Comment #9 from Stefan Seyfried --- Unfortunately, 5.17.5 does not dump but just hang. I'll have to investigate this first... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c11 --- Comment #11 from Stefan Seyfried --- I now have a crash but no way to load it: raspi4c:/var/crash/2022-05-16-10:14 # crash Image-5.17.5-1-default vmcore crash 7.3.0 Copyright (C) 2002-2021 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2021 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. crash: Image-5.17.5-1-default: not a supported file format Usage: crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) crash [OPTION]... [NAMELIST] (live system form) Enter "crash -h" for details. Please advise :-( I only used crash on x86_64 before. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c14 --- Comment #14 from Stefan Seyfried --- the only vmlinuz is in /usr/lib/modules/<version>/vmlinuz.xz and it seems to work somehow, but not good enough. crash starts with "--minimal", but I would at least have wanted to do a "bt full" or such (my crash usage is rusty, has been years since I did kernel debugging stuff ;-)) and that's not available with --minimal :-( crash> bt full crash: bt: command not available in minimal mode NOTE: minimal mode commands: log, dis, rd, sym, eval, set, extend and exit For my knowledge level, the --minimal commands are pretty useless, the log (dmesg) is the same as posted earlier. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c18 --- Comment #18 from Stefan Seyfried --- I managed to build a seemingly working 8.0.1 version in home:seife:testing. I can get a backtrace now: PID: 2032 TASK: ffff481f84063f80 CPU: 3 COMMAND: "qemu-system-aar" #0 [ffff80000b46a7c0] machine_kexec at ffffcbfbb6bb99b0 #1 [ffff80000b46a7f0] __crash_kexec at ffffcbfbb6d1885c #2 [ffff80000b46a980] crash_kexec at ffffcbfbb6d18998 #3 [ffff80000b46a9a0] die at ffffcbfbb6ba1b08 #4 [ffff80000b46aa50] die_kernel_fault at ffffcbfbb790d5f8 #5 [ffff80000b46aaa0] do_page_fault at ffffcbfbb7932a90 #6 [ffff80000b46ab00] do_translation_fault at ffffcbfbb7932d68 #7 [ffff80000b46ab20] do_mem_abort at ffffcbfbb6bbcac8 #8 [ffff80000b46ab50] el1_abort at ffffcbfbb791f494 #9 [ffff80000b46ab80] el1h_64_sync_handler at ffffcbfbb7920d80 #10 [ffff80000b46acc0] el1h_64_sync at ffffcbfbb6b91b0c #11 [ffff80000b46ace0] blk_mq_submit_bio at ffffcbfbb712f914 #12 [ffff80000b46ad80] __submit_bio at ffffcbfbb711e884 #13 [ffff80000b46adc0] submit_bio_noacct at ffffcbfbb711ead8 #14 [ffff80000b46ae10] submit_bio at ffffcbfbb711eb90 #15 [ffff80000b46ae40] __swap_writepage at ffffcbfbb6ecc330 #16 [ffff80000b46aef0] swap_writepage at ffffcbfbb6ecc66c #17 [ffff80000b46af20] pageout at ffffcbfbb6e62ed0 #18 [ffff80000b46afc0] shrink_page_list at ffffcbfbb6e637bc #19 [ffff80000b46b0b0] shrink_lruvec at ffffcbfbb6e67750 #20 [ffff80000b46b220] shrink_node at ffffcbfbb6e680f4 #21 [ffff80000b46b2b0] do_try_to_free_pages at ffffcbfbb6e68818 #22 [ffff80000b46b330] try_to_free_pages at ffffcbfbb6e69f44 #23 [ffff80000b46b3e0] __alloc_pages at ffffcbfbb6ec0564 #24 [ffff80000b46b530] alloc_pages at ffffcbfbb6ee8e68 #25 [ffff80000b46b560] folio_alloc at ffffcbfbb6ee8f44 #26 [ffff80000b46b580] filemap_alloc_folio at ffffcbfbb6e44594 #27 [ffff80000b46b5b0] page_cache_ra_unbounded at ffffcbfbb6e58ff8 #28 [ffff80000b46b650] ondemand_readahead at ffffcbfbb6e59528 #29 [ffff80000b46b690] page_cache_async_ra at ffffcbfbb6e59770 #30 [ffff80000b46b6c0] filemap_get_pages at ffffcbfbb6e49204 #31 [ffff80000b46b770] filemap_read at ffffcbfbb6e49494 #32 [ffff80000b46b8a0] generic_file_read_iter at ffffcbfbb6e497f0 #33 [ffff80000b46b8f0] xfs_file_buffered_read at ffffcbfb669ece90 [xfs] #34 [ffff80000b46b920] xfs_file_read_iter at ffffcbfb669ed1a4 [xfs] #35 [ffff80000b46b950] io_read at ffffcbfbb6fc8ae4 #36 [ffff80000b46ba70] io_issue_sqe at ffffcbfbb6fcd1b4 #37 [ffff80000b46bb30] io_submit_sqes at ffffcbfbb6fcf188 #38 [ffff80000b46bc20] __arm64_sys_io_uring_enter at ffffcbfbb6fd09e0 #39 [ffff80000b46be10] invoke_syscall at ffffcbfbb6baa824 #40 [ffff80000b46be40] el0_svc_common.constprop.0 at ffffcbfbb6baaa2c #41 [ffff80000b46be70] do_el0_svc at ffffcbfbb6baaa64 #42 [ffff80000b46be80] el0_svc at ffffcbfbb791f8cc #43 [ffff80000b46bea0] el0t_64_sync_handler at ffffcbfbb7920f10 #44 [ffff80000b46bfe0] el0t_64_sync at ffffcbfbb6b91e34 PC: 0000ffff9f2d9c28 LR: 0000ffff9f8420c4 SP: 0000ffff9df6dbf0 X29: 0000ffff9df6dc10 X28: 0000000000001022 X27: 0000aaaafaad564c X26: 000000000080e1c0 X25: 0000aaaadd4dbcf2 X24: 0000aaaadd4def38 X23: 0000fffda8005770 X22: 0000fffda8005688 X21: 0000fffda8005680 X20: 0000000000000000 X19: 0000fffd9d7658f0 X18: 0000000000000001 X17: 0000ffff9f2d9c00 X16: 0000ffff9f85ffc8 X15: 0000000000000000 X14: 0000000000000000 X13: 0000ffff90013420 X12: 0000ffff900342a0 X11: 00ffffffffffffff X10: 0038af1bcc5c671f X9: 0000000000000004 X8: 00000000000001aa X7: 0000000000000000 X6: 0000000000000000 X5: 0000000000000008 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000000 X1: 0000000000000004 X0: 000000000000001d ORIG_X0: 000000000000001d SYSCALLNO: 1aa PSTATE: 40001000 However, that's about what's left of my crashdump-examination knowledge ;-) Could you advise me how to gather useful information to find the cause of the crash? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c21 --- Comment #21 from Stefan Seyfried --- [ 9341.426047] pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 9341.426054] pc : blk_mq_submit_bio+0x17c/0x6b0 [ 9341.426075] lr : blk_mq_submit_bio+0x160/0x6b0 [ 9341.426081] sp : ffff800008ee2d90 [ 9341.426086] x29: ffff800008ee2d90 x28: ffff470cc3861980 x27: fffffd1c36145648 [ 9341.426096] x26: fffffd1c36145640 x25: ffff800008ee3128 x24: 0000000000000001 [ 9341.426106] x23: 0000000000000000 x22: ffff470cc25c48e0 x21: ffff470cc4f8b138 [ 9341.426116] x20: ffff470cc2b43800 x19: ffff470cc4f90f00 x18: 0000000095f27000 [ 9341.426125] x17: 0000000000000001 x16: ffffab5ab3275db0 x15: 0000000000000000 [ 9341.426134] x14: 0000000000000000 x13: 0000000000000038 x12: 0000000000000000 [ 9341.426143] x11: 0000000000000000 x10: 0000000000001b80 x9 : ffffab5ab34bf1e0 [ 9341.426152] x8 : ffff470cc6fb7b20 x7 : 0000000000ac7ada x6 : ffffab5ab3ce3dc0 [ 9341.426160] x5 : 00000000410fd080 x4 : 0000000000000000 x3 : ffff470cc38ebc50 [ 9341.426169] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000004000001 [ 9341.426178] Call trace: [ 9341.426183] blk_mq_submit_bio+0x17c/0x6b0 [ 9341.426190] __submit_bio+0x118/0x17c [ 9341.426198] submit_bio_noacct+0x1ec/0x240 [ 9341.426204] submit_bio+0xa8/0x140 [ 9341.426210] __swap_writepage+0x1ac/0x494 [ 9341.426218] swap_writepage+0x50/0x10c [ 9341.426224] pageout+0x104/0x340 [ 9341.426232] shrink_page_list+0x66c/0xe9c [ 9341.426240] shrink_lruvec+0x548/0xb64 [ 9341.426247] shrink_node+0x3ec/0x74c [ 9341.426253] do_try_to_free_pages+0xf0/0x580 [ 9341.426260] try_to_free_pages+0x118/0x20c [ 9341.426266] __alloc_pages+0x48c/0xd60 [ 9341.426273] alloc_pages+0xb8/0x16c [ 9341.426280] folio_alloc+0x28/0x64 [ 9341.426286] filemap_alloc_folio+0xd8/0xf0 [ 9341.426292] page_cache_ra_unbounded+0xac/0x224 [ 9341.426314] ondemand_readahead+0x28c/0x2c0 [ 9341.426324] page_cache_sync_ra+0xe0/0xec [ 9341.426333] filemap_get_pages+0xb8/0x6b4 [ 9341.426342] filemap_read+0xc4/0x31c [ 9341.426351] generic_file_read_iter+0x114/0x1b0 [ 9341.426361] xfs_file_buffered_read+0xb4/0xdc [xfs 096b9df6336e2333cf64584a5f3515ae55e87984] [ 9341.426732] xfs_file_read_iter+0xa8/0x124 [xfs 096b9df6336e2333cf64584a5f3515ae55e87984] [ 9341.426915] io_read+0x104/0x3dc [ 9341.426932] io_issue_sqe+0x1f8/0x1c74 [ 9341.426944] io_submit_sqes+0x2a4/0x1614 [ 9341.426954] __arm64_sys_io_uring_enter+0x4c8/0x79c [ 9341.426965] invoke_syscall+0x78/0x100 [ 9341.426984] el0_svc_common.constprop.0+0x18c/0x190 [ 9341.426996] do_el0_svc+0x30/0x94 [ 9341.427006] el0_svc+0x28/0xe0 [ 9341.427020] el0t_64_sync_handler+0x10c/0x140 [ 9341.427031] el0t_64_sync+0x1a4/0x1a8 [ 9341.427045] Code: 37c00060 72001c1f 1a9f17e1 f9400a62 (79426842) [ 9341.427068] SMP: stopping secondary CPUs [ 9341.427111] Starting crashdump kernel... crash> dis -s blk_mq_submit_bio+0x17c FILE: ../block/blk-mq.c LINE: 2764 2759 return NULL; 2760 } 2761 2762 rq_qos_throttle(q, *bio); 2763 * 2764 if (blk_mq_get_hctx_type((*bio)->bi_opf) != rq->mq_hctx->type) 2765 return NULL; 2766 if (op_is_flush(rq->cmd_flags) != op_is_flush((*bio)->bi_opf)) 2767 return NULL; 2768 2769 rq->cmd_flags = (*bio)->bi_opf; 2770 plug->cached_rq = rq_list_next(rq); 2771 INIT_LIST_HEAD(&rq->queuelist); 2772 return rq; 2773 } Unfortuantely I'm not able to find out the address of "bio" to check if it is sane :-( 0x17c == 380dec 0xffffab5ab34ab104 : cbz x0, 0xffffab5ab34ab124 /usr/src/debug/kernel-default-5.17.7-1.1.aarch64/linux-5.17/linux-obj/../include/linux/bio.h: 238 0xffffab5ab34ab108 : ldrh w0, [x20, #20] /usr/src/debug/kernel-default-5.17.7-1.1.aarch64/linux-5.17/linux-obj/../block/blk-rq-qos.h: 194 0xffffab5ab34ab10c : mov x1, x20 /usr/src/debug/kernel-default-5.17.7-1.1.aarch64/linux-5.17/linux-obj/../include/linux/bio.h: 238 0xffffab5ab34ab110 : orr w0, w0, #0x400 0xffffab5ab34ab114 : strh w0, [x20, #20] /usr/src/debug/kernel-default-5.17.7-1.1.aarch64/linux-5.17/linux-obj/../block/blk-rq-qos.h: 194 0xffffab5ab34ab118 : ldr x0, [x21, #40] 0xffffab5ab34ab11c : bl 0xffffab5ab34bf1a0 <__rq_qos_throttle> /usr/src/debug/kernel-default-5.17.7-1.1.aarch64/linux-5.17/linux-obj/../block/blk-mq.c: 2764 0xffffab5ab34ab120 : ldr x20, [sp, #72] 0xffffab5ab34ab124 : mov w1, #0x2 // #2 0xffffab5ab34ab128 : ldr w0, [x20, #16] /usr/src/debug/kernel-default-5.17.7-1.1.aarch64/linux-5.17/linux-obj/../block/blk-mq.h: 96 0xffffab5ab34ab12c : tbnz w0, #24, 0xffffab5ab34ab138 /usr/src/debug/kernel-default-5.17.7-1.1.aarch64/linux-5.17/linux-obj/../block/blk-mq.h: 98 0xffffab5ab34ab130 : tst w0, #0xff 0xffffab5ab34ab134 : cset w1, eq // eq = none /usr/src/debug/kernel-default-5.17.7-1.1.aarch64/linux-5.17/linux-obj/../block/blk-mq.h: 100 0xffffab5ab34ab138 : ldr x2, [x19, #16] /usr/src/debug/kernel-default-5.17.7-1.1.aarch64/linux-5.17/linux-obj/../block/blk-mq.c: 2764 0xffffab5ab34ab13c : ldrh w2, [x2, #308] 0xffffab5ab34ab140 : cmp w2, w1 0xffffab5ab34ab144 : b.ne 0xffffab5ab34ab310 // b.any Unfortunately, I speak absolutely no assembler newer than Z80 :-( -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c30 --- Comment #30 from Stefan Seyfried --- I'll build a kernel locally. What should I test? The patch mentioned above against 5.18.0, the debug patch from Hannes or both? Regarding stack overflow or such: This is an 8GB raspberry pi4 and it is only crashing when running a huge VM (7.5GB) as OBS build job, building a package (kodi) which actually uses all this memory. It's always this one build job crashing the machine, relatively reliably. I have seen my other 8GB raspi4 which is running 2 "smaller" (3.7G) build jobs at once crash at least once with a similar log message, but after enabling kdump it has not happened again. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c34 --- Comment #34 from Stefan Seyfried --- This is an example commandline of the qemu invocation: /usr/bin/qemu-system-aarch64 -nodefaults -no-reboot -nographic -vga none -cpu host -enable-kvm -M virt,accel=kvm,usb=off,gic-version=host -sandbox on -object rng-random,filename=/dev/hwrng,id=rng0 -device virtio-rng-device,rng=rng0 -object iothread,id=io0 -runas qemu -net none -kernel /var/cache/obs/worker/root_2/.mount/boot/kernel -initrd /var/cache/obs/worker/root_2/.mount/boot/initrd -append root=/dev/disk/by-id/virtio-0 rootfstype=ext4 rootflags=noatime kvmclock mitigations=off elevator=noop nmi_watchdog=0 rw rd.driver.pre=binfmt_misc oops=panic panic=1 quiet console=hvc0 init=/.build/build -m 1750 -drive file=/var/cache/obs/worker/root_2/root,format=raw,if=none,id=disk,cache=unsafe -device virtio-blk-device,iothread=io0,drive=disk,serial=0 -drive file=/var/cache/obs/worker/root_2/swap,format=raw,if=none,id=swap,cache=unsafe -device virtio-blk-device,iothread=io0,drive=swap,serial=1 -device virtio-serial,max_ports=2 -device virtconsole,chardev=virtiocon0 -chardev stdio,id=virtiocon0 -chardev socket,id=monitor,server=on,wait=off,path=/var/cache/obs/worker/root_2/root.qemu/monitor -mon chardev=monitor,mode=readline -smp 2 If there is just another command line option for reverting this last qemu change, I could try replacing it with a wrapper script. If it involves more than just adding an option, it's probably not easily doable. * Tue Apr 19 2022 Dirk M�ller - enable aio=io_uring on all kvm architectures (bsc#1197699) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c39 --- Comment #39 from Stefan Seyfried --- Ah, ok. I was looking at the buildlog of my single Leap 15.3 worker, which of course does not enable this feature. The tumbleweed workers do indeed :-) Unfortunately I can't easily change the build script as it is not my OBS instance, so yes, let's just fix the kernel instead ;-) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c41 --- Comment #41 from Stefan Seyfried --- Just for fun I tried with only the patch from https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-next&id=5d05426e2d5fd7df8afc866b78c36b37b00188b7 on top of 5.18.0-suse This made the box crash much earlier (about 240 seconds into a build, and I'm not even sure it was the "known bad" build job, usually it crashes after about 9000 seconds close to the end of the build. The crash was also completely different: [ 126.798937] EXT4-fs (loop0): mounted filesystem with writeback data mode. Quota mode: none. [ 393.252701] Unable to handle kernel paging request at virtual address ffff066bfc222e00 [ 393.252735] Mem abort info: [ 393.252739] ESR = 0x96000004 [ 393.252744] EC = 0x25: DABT (current EL), IL = 32 bits [ 393.252749] SET = 0, FnV = 0 [ 393.252754] EA = 0, S1PTW = 0 [ 393.252757] FSC = 0x04: level 0 translation fault [ 393.252762] Data abort info: [ 393.252766] ISV = 0, ISS = 0x00000004 [ 393.252770] CM = 0, WnR = 0 [ 393.252774] swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000001cd22000 [ 393.252780] [ffff066bfc222e00] pgd=0000000000000000, p4d=0000000000000000 [ 393.252791] Internal error: Oops: 96000004 [#1] SMP [ 393.252799] Modules linked in: loop tun af_packet iscsi_ibft iscsi_boot_sysfs nls_iso8859_1 nls_cp437 xfs vfat libcrc32c fat btsdio cpufreq_dt hci_uart btqca brcmfmac btrtl btbcm brcmutil btintel bluetooth cfg80211 ecdh_generic rfkill raspberrypi_cpufreq bcm2711_thermal broadcom bcm_phy_lib iproc_rng200 genet mdio_bcm_unimac leds_gpio simpledrm drm_shmem_helper nvmem_rmem drm_kms_helper efi_pstore syscopyarea sysfillrect sysimgblt uio_pdrv_genirq fb_sys_fops uio fuse drm ip_tables x_tables ext4 mbcache jbd2 uas usb_storage xhci_pci xhci_pci_renesas xhci_hcd usbcore usb_common bcm2835_dma crct10dif_ce clk_raspberrypi gpio_raspberrypi_exp bcm2835_wdt raspberrypi_hwmon virt_dma sdhci_iproc gpio_regulator phy_generic sdhci_pltfm pcie_brcmstb sdhci mmc_core fixed sg efivarfs [ 393.252941] CPU: 1 PID: 20 Comm: ksoftirqd/1 Kdump: loaded Not tainted 5.18.0-seife #4 openSUSE Tumbleweed (unreleased) 67de12037530e9d1ee4775af6e6ed4ae03465d51 [ 393.252959] Hardware name: Unknown Unknown Product/Unknown Product, BIOS 2022.04 04/01/2022 [ 393.252966] pstate: 404000c5 (nZcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 393.252973] pc : percpu_ref_get_many+0x40/0xac [ 393.252996] lr : percpu_ref_get_many+0x1c/0xac [ 393.253002] sp : ffff8000080dbb00 [ 393.253008] x29: ffff8000080dbb00 x28: ffff6f74002bc000 x27: ffff6f7401893180 [ 393.253018] x26: ffff6f7400189600 x25: fffffc0000000000 x24: 0001000000000000 [ 393.253027] x23: ffff6f74052f9200 x22: 0000000000000000 x21: ffff6f7401893180 [ 393.253035] x20: ffff6f7401893180 x19: 0000000000000001 x18: 0000000000000000 [ 393.253045] x17: 0000000000000001 x16: ffffd87d06c26ab0 x15: 0000000000000000 [ 393.253053] x14: 0000000000000000 x13: 0000000000000030 x12: 0000000000000068 [ 393.253062] x11: 0000000000000100 x10: 0000000000001ba0 x9 : ffffd87d0695c860 [ 393.253070] x8 : ffff6f74002bdc00 x7 : ffff96f7f7285000 x6 : fffffffffffffcb0 [ 393.253078] x5 : ffff6f74001a4000 x4 : fffffffffff51e78 x3 : 0000000000000000 [ 393.253086] x2 : ffff96f7f7285000 x1 : ffff6f74002bc000 x0 : ffff066bfc222e00 [ 393.253097] Call trace: [ 393.253101] percpu_ref_get_many+0x40/0xac [ 393.253108] refill_obj_stock+0x6c/0x194 [ 393.253118] obj_cgroup_uncharge+0x20/0x2c [ 393.253127] memcg_slab_free_hook+0xa4/0x190 [ 393.253136] kmem_cache_free+0x2d4/0x310 [ 393.253144] __d_free+0x28/0x34 [ 393.253153] rcu_do_batch+0x174/0x6f0 [ 393.253162] rcu_core+0x264/0x3fc [ 393.253171] rcu_core_si+0x1c/0x30 [ 393.253176] __do_softirq+0x128/0x3b8 [ 393.253183] run_ksoftirqd+0x6c/0x94 [ 393.253192] smpboot_thread_fn+0x230/0x254 [ 393.253205] kthread+0x114/0x120 [ 393.253211] ret_from_fork+0x10/0x20 [ 393.253236] Code: 11000442 b9001022 d538d082 8b020000 (c85f7c04) [ 393.253253] SMP: stopping secondary CPUs [ 393.253304] Starting crashdump kernel... [ 393.253311] ------------[ cut here ]------------ [ 393.253318] Some CPUs may be stale, kdump will be unreliable. [ 393.253332] WARNING: CPU: 1 PID: 20 at arch/arm64/kernel/machine_kexec.c:187 machine_kexec+0x48/0x220 [ 393.253351] Modules linked in: loop tun af_packet iscsi_ibft iscsi_boot_sysfs nls_iso8859_1 nls_cp437 xfs vfat libcrc32c fat btsdio cpufreq_dt hci_uart btqca brcmfmac btrtl btbcm brcmutil btintel bluetooth cfg80211 ecdh_generic rfkill raspberrypi_cpufreq bcm2711_thermal broadcom bcm_phy_lib iproc_rng200 genet mdio_bcm_unimac leds_gpio simpledrm drm_shmem_helper nvmem_rmem drm_kms_helper efi_pstore syscopyarea sysfillrect sysimgblt uio_pdrv_genirq fb_sys_fops uio fuse drm ip_tables x_tables ext4 mbcache jbd2 uas usb_storage xhci_pci xhci_pci_renesas xhci_hcd usbcore usb_common bcm2835_dma crct10dif_ce clk_raspberrypi gpio_raspberrypi_exp bcm2835_wdt raspberrypi_hwmon virt_dma sdhci_iproc gpio_regulator phy_generic sdhci_pltfm pcie_brcmstb sdhci mmc_core fixed sg efivarfs [ 393.253473] CPU: 1 PID: 20 Comm: ksoftirqd/1 Kdump: loaded Not tainted 5.18.0-seife #4 openSUSE Tumbleweed (unreleased) 67de12037530e9d1ee4775af6e6ed4ae03465d51 [ 393.253485] Hardware name: Unknown Unknown Product/Unknown Product, BIOS 2022.04 04/01/2022 [ 393.253492] pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 393.253501] pc : machine_kexec+0x48/0x220 [ 393.253510] lr : machine_kexec+0x48/0x220 [ 393.253519] sp : ffff8000080db600 [ 393.253525] x29: ffff8000080db600 x28: ffff8000080db863 x27: ffffd87d078c5178 [ 393.253537] x26: ffffd87d078c5170 x25: ffffd87d0695c888 x24: 0000000000000001 [ 393.253548] x23: 0000000000000001 x22: ffff8000080db668 x21: ffffd87d0875a000 [ 393.253558] x20: ffff6f7401f52000 x19: ffff6f7401f52000 x18: ffffffffffffffff [ 393.253569] x17: 3065323232636662 x16: 3636306666666620 x15: 0720072007200720 [ 393.253580] x14: 0720072007200720 x13: 0720072007200720 x12: 0720072007200720 [ 393.253591] x11: 00000000ffffdfff x10: ffffd87d08489808 x9 : ffffd87d0670290c [ 393.253602] x8 : 000000000002ffe8 x7 : c0000000ffffdfff x6 : 00000000000affa8 [ 393.253613] x5 : ffff6f74fef5db48 x4 : 0000000000000000 x3 : 0000000000000027 [ 393.253623] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff6f74002bc000 [ 393.253634] Call trace: [ 393.253640] machine_kexec+0x48/0x220 [ 393.253650] __crash_kexec+0x80/0x130 [ 393.253660] crash_kexec+0x88/0xa0 [ 393.253669] die+0x16c/0x234 [ 393.253678] die_kernel_fault+0x38c/0x39c [ 393.253687] __do_kernel_fault+0xf0/0x224 [ 393.253695] do_translation_fault+0xa0/0xd0 [ 393.253704] do_mem_abort+0x4c/0xa0 [ 393.253712] el1_abort+0x74/0xdc [ 393.253721] el1h_64_sync_handler+0xa4/0xd0 [ 393.253731] el1h_64_sync+0x78/0x7c [ 393.253739] percpu_ref_get_many+0x40/0xac [ 393.253747] refill_obj_stock+0x6c/0x194 [ 393.253757] obj_cgroup_uncharge+0x20/0x2c [ 393.253766] memcg_slab_free_hook+0xa4/0x190 [ 393.253776] kmem_cache_free+0x2d4/0x310 [ 393.253784] __d_free+0x28/0x34 [ 393.253794] rcu_do_batch+0x174/0x6f0 [ 393.253802] rcu_core+0x264/0x3fc [ 393.253810] rcu_core_si+0x1c/0x30 [ 393.253818] __do_softirq+0x128/0x3b8 [ 393.253826] run_ksoftirqd+0x6c/0x94 [ 393.253835] smpboot_thread_fn+0x230/0x254 [ 393.253845] kthread+0x114/0x120 [ 393.253853] ret_from_fork+0x10/0x20 [ 393.253861] ---[ end trace 0000000000000000 ]--- [ 393.253869] Bye! the loop0 message is from initializing the build VM, the next message is already the crash. So at least the one patch in linux-next alone does definitely not fix the issue ;-) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c43 --- Comment #43 from Stefan Seyfried --- 5.19-rc1 crashes differently: [14956.861060] loop: module loaded [14956.873083] loop0: detected capacity change from 0 to 37748736 [14956.934270] EXT4-fs (loop0): mounted filesystem without journal. Quota mode: none. [14971.342291] EXT4-fs (loop0): unmounting filesystem. [17413.978424] Unable to handle kernel paging request at virtual address fffe75b44f2d4a00 [17413.978465] Mem abort info: [17413.978470] ESR = 0x0000000096000004 [17413.978476] EC = 0x25: DABT (current EL), IL = 32 bits [17413.978482] SET = 0, FnV = 0 [17413.978486] EA = 0, S1PTW = 0 [17413.978490] FSC = 0x04: level 0 translation fault [17413.978495] Data abort info: [17413.978499] ISV = 0, ISS = 0x00000004 [17413.978503] CM = 0, WnR = 0 [17413.978508] [fffe75b44f2d4a00] address between user and kernel address ranges [17413.978516] Internal error: Oops: 96000004 [#1] SMP [17413.978525] Modules linked in: loop tun af_packet iscsi_ibft iscsi_boot_sysfs nls_iso8859_1 xfs nls_cp437 libcrc32c vfat fat btsdio cpufreq_dt hci_uart btqca btrtl btbcm btintel brcmfmac brcmutil bluetooth broadcom cfg80211 bcm_phy_lib ecdh_generic rfkill raspberrypi_cpufreq iproc_rng200 bcm2711_thermal genet mdio_bcm_unimac simpledrm leds_gpio efi_pstore uio_pdrv_genirq drm_shmem_helper nvmem_rmem uio drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm fuse ip_tables x_tables ext4 mbcache jbd2 hid_generic usbhid uas usb_storage xhci_pci xhci_pci_renesas xhci_hcd usbcore usb_common crct10dif_ce clk_raspberrypi gpio_raspberrypi_exp bcm2835_dma virt_dma raspberrypi_hwmon bcm2835_wdt sdhci_iproc sdhci_pltfm gpio_regulator sdhci mmc_core pcie_brcmstb phy_generic fixed sg efivarfs [17413.978689] CPU: 0 PID: 13 Comm: ksoftirqd/0 Kdump: loaded Not tainted 5.19.0-rc1-seife #2 8f2caf59d69bf3bff9185834c09c651834367aaa [17413.978703] Hardware name: Unknown Unknown Product/Unknown Product, BIOS 2022.04 04/01/2022 [17413.978710] pstate: 404000c5 (nZcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [17413.978719] pc : percpu_ref_get_many+0x44/0xbc [17413.978740] lr : percpu_ref_get_many+0x40/0xbc [17413.978747] sp : ffff80000808bb50 [17413.978751] x29: ffff80000808bb50 x28: ffff1a1fc026c000 x27: ffffbe8c31d3da00 [17413.978761] x26: ffffbe8c3154c008 x25: 0000000000000000 x24: ffff1a1fc64fc000 [17413.978771] x23: ffff1a1fc50f2e00 x22: 0000000000000000 x21: ffff1a1fc026c000 [17413.978780] x20: 0000000000000001 x19: ffff1a1fc18e5a00 x18: 0000000000000000 [17413.978790] x17: 0000000000000000 x16: ffffbe8c2fef4210 x15: 0000000000000000 [17413.978799] x14: 0000000000000000 x13: 0000000000000030 x12: 0101010101010101 [17413.978809] x11: ffff1a20bef53690 x10: fffffc687f08f6c8 x9 : ffffbe8c30181294 [17413.978818] x8 : fffffc687f191c80 x7 : ffff1a1fc6598500 x6 : 0000000055555556 [17413.978828] x5 : 0000000000000001 x4 : 0000000000000007 x3 : ffff5b948d9ef000 [17413.978837] x2 : 0000000000000000 x1 : ffff1a1fc026c000 x0 : fffe75b44f2d4a00 [17413.978846] Call trace: [17413.978852] percpu_ref_get_many+0x44/0xbc [17413.978861] refill_obj_stock+0x6c/0x180 [17413.978868] obj_cgroup_uncharge+0x20/0x2c [17413.978875] kmem_cache_free+0x304/0x3c4 [17413.978883] __d_free+0x28/0x34 [17413.978891] rcu_do_batch+0x174/0x6f0 [17413.978900] rcu_core+0x264/0x3fc [17413.978906] rcu_core_si+0x1c/0x30 [17413.978912] __do_softirq+0x128/0x3b8 [17413.978920] run_ksoftirqd+0x6c/0x94 [17413.978928] smpboot_thread_fn+0x230/0x254 [17413.978936] kthread+0x114/0x120 [17413.978942] ret_from_fork+0x10/0x20 [17413.978952] Code: 11000400 b90012a0 97fff700 8b000260 (c85f7c02) [17413.978968] SMP: stopping secondary CPUs [17413.979013] Starting crashdump kernel... I have also seen similar crashes during early system startup that I could not investigate. Maybe I really have a config option missing or -rc1 is still not stable enough. I'll wait until 5.19-rc appears in Kernel:HEAD and continue testing then. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1199355 http://bugzilla.opensuse.org/show_bug.cgi?id=1199355#c45 --- Comment #45 from Stefan Seyfried --- It looks like 5.18.9 is already much better, no new crashes yet (and the "bad" package has been built multiple times now). Maybe I was not the only one experiencing this and it got actually fixed upstream ;-) I did not spot relevant changes in qemu change log, so the fix probably has to have come from the kernel... -- You are receiving this mail because: You are on the CC list for the bug.