[opensuse-kernel] Re: dracut (050+suse.58.g0b998bee ) exec segfaults for kernel:stable + GCC10 , but creates initrd anyway ?
hi, On 3/24/20 1:56 PM, Daniel Molkentin wrote:
Why/how's the dracut/mkinitrd exec getting past a segfault, to successfully create an initrd?
What is crashing here (probably due to a GCC 10 behavior/bug, it works fine with the current GCC on TW), is the dracut-install helper binary that gets invoked several times during the run of dracut (which is a shell script) and is allowed to fail. Hence, in the end, an (incomplete) initrd is being assembled that lacks the kernel modules in the command line mentioned in above output.
Assuming I am reading your description correctly, you are using dracut-050 is straight from Base:System and has hence not been built with GCC 10, right? So dracut was built with GCC 9, and the rest with GCC 10?
nope, sorry ... perhaps my miscomm atm, "my" dracut's from https://build.opensuse.org/package/show/home:pgnd:Base:System/dracut?expand=... Links , a simple link to Base:System / dracut but built with "my" GCC10 https://build.opensuse.org/projects/home:pgnd:Base:System/meta i.e., this^ dracut _is_ built with GCC10, as well as kernel, built with latest GCC10/head. very recent fix to GCC fixed the kernel-build incompat, https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94238 and has already been pulled into Opensuse GCC10 pkgs, https://build.opensuse.org/package/rdiff/devel:gcc/gcc10?linkrev=base&rev=134 fwiw, grub, atm, is not building with GCC10; different issue: https://lists.gnu.org/archive/html/grub-devel/2020-03/msg00216.html -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 3/24/20 2:10 PM, PGNet Dev wrote:
So dracut was built with GCC 9, and the rest with GCC 10?
sanity-checking THAT case, replacing gcc10-build dracut rpm -qa | grep dracut dracut-050+suse.58.g0b998bee-lp151.6.1.x86_64 with a GCC9-build from https://build.opensuse.org/package/binaries/home:pgnd:test:dracut+gcc9/dracu... installed rpm -Uvh --force ./dracut-050+suse.58.g0b998bee-lp151.3.1.x86_64.rpm rpm -qa | grep dracut dracut-050+suse.58.g0b998bee-lp151.3.1.x86_64 exec of, mkinitrd -i /boot/initrd-5.5.11-24.g794b2e9-default -k /boot/vmlinuz-5.5.11-24.g794b2e9-default fails as before, with the GCC10-built dracut ... dracut: *** Including module: suse-initrd *** dracut: *** Including modules done *** /usr/lib/dracut/dracut-init.sh: line 941: 24358 Segmentation fault (core dumped) $DRACUT_INSTALL ${initdir:+-D "$initdir"} ${dracutsysrootdir:+-r "$dracutsysrootdir"} ${loginstall:+-L "$loginstall"} ${hostonly:+-H} ${check_supported:+--check-supported} ${omit_drivers:+-N "$omit_drivers"} ${srcmods:+--kerneldir "$srcmods"} ${_optional:+-o} ${_silent:+--silent} -m "$@" dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.0HHHgc/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m raid0 raid1 raid10 raid456 dm_mod dm-snapshot sata_sil24 thermal fan edd cpuid usbhid usb_storage ehci_hcd ohci_pci uas ohci_hcd ehci_pci dm-crypt cryptd crypto_simd br_netfilter sha1-ssse3 sha256-ssse3 sha512-ssse3 sha3_generic tcp_htcp tcp_bbr xennet xenblk dracut: *** Installing kernel module dependencies *** dracut: *** Installing kernel module dependencies done *** ... -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
exploring the segfault'd stanza E: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.0HHHgc/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m raid0 raid1 raid10 raid456 dm_mod dm-snapshot sata_sil24 thermal fan edd cpuid usbhid usb_storage ehci_hcd ohci_pci uas ohci_hcd ehci_pci dm-crypt cryptd crypto_simd br_netfilter sha1-ssse3 sha256-ssse3 sha512-ssse3 sha3_generic tcp_htcp tcp_bbr xennet xenblk for export DRACUT_FIRMWARE_PATH="/lib/firmware/updates:/lib/firmware:/lib/firmware/5.5.11-24.g794b2e9-default" mkdir -p /var/tmp/dracut.TEST/initramfs i noted (1) _with_ module usb_storage exec of $ /usr/lib/dracut/dracut-install -d \ -D /var/tmp/dracut.TEST/initramfs \ -N i2o_scsi \ --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ \ -m raid0 raid1 usb_storage ... faults Segmentation fault (core dumped) logs Mar 24 15:46:24 test kernel: dracut-install[54910]: segfault at 7ffd36368ff8 ip 00007fa2a4f57d8b sp 00007ffd36369000 error 6 in libc-2.26.so[7fa2a4eda000+1b1000] Mar 24 15:46:24 test kernel: Code: 53 08 0f b6 00 48 83 c4 08 5b 5d c3 0f 1f 00 e8 7b bf ff ff eb d5 66 0f 1f 84 00 00 00 00 00 48 85 d2 0f 84 cf 00 00 00 41 57 <41> 56 49 89 ff 41 55 41 54 4c 8d 25 45 2b 33 00 55 48 8d 2d a5 38 Mar 24 15:46:24 test kernel: potentially unexpected fatal signal 11. Mar 24 15:46:24 test kernel: CPU: 4 PID: 54910 Comm: dracut-install Tainted: P W OE 5.5.11-24.g794b2e9-default #1 openSUSE Tumbleweed (unreleased) Mar 24 15:46:24 test kernel: Hardware name: System manufacturer System Product Name/PRIME X570-PRO, BIOS 1405 11/19/2019 Mar 24 15:46:24 test kernel: RIP: 0033:0x7fa2a4f57d8b Mar 24 15:46:24 test kernel: Code: 53 08 0f b6 00 48 83 c4 08 5b 5d c3 0f 1f 00 e8 7b bf ff ff eb d5 66 0f 1f 84 00 00 00 00 00 48 85 d2 0f 84 cf 00 00 00 41 57 <41> 56 49 89 ff 41 55 41 54 4c 8d 25 45 2b 33 00 55 48 8d 2d a5 38 Mar 24 15:46:24 test kernel: RSP: 002b:00007ffd36369000 EFLAGS: 00010206 Mar 24 15:46:24 test kernel: RAX: 00007fa2a528b120 RBX: 00007fa2a4ccb041 RCX: 0000000000000000 Mar 24 15:46:24 test kernel: RDX: 0000000000000028 RSI: 000055ac7b2d15a0 RDI: 00007ffd36369590 Mar 24 15:46:24 test kernel: RBP: 00007ffd36369580 R08: 0000000000000028 R09: 00000000ffffffff Mar 24 15:46:24 test kernel: R10: 000055ac7b2d15a0 R11: 0000000000000000 R12: 00007ffd36369710 Mar 24 15:46:24 test kernel: R13: 0000000000000000 R14: 00007ffd36369590 R15: 0000000000000028 Mar 24 15:46:24 test kernel: FS: 00007fa2a5446100 GS: 0000000000000000 (2) withOUT usb_storage exec of $ /usr/lib/dracut/dracut-install -d \ -D /var/tmp/dracut.TEST/initramfs \ -N i2o_scsi \ --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ \ -m raid0 raid1 ... $ no segfault ... :-/ i checked execs for each of the included modules, one module at a time, and get failures for ONLY -m usb_storage -m uas checking the complete stanza, sans both of these, the 'usb_storage' & 'uas' mods, it execs /usr/lib/dracut/dracut-install -d \ -D /var/tmp/dracut.TEST/initramfs \ -N i2o_scsi \ --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ \ -m raid0 raid1 raid10 raid456 dm_mod dm-snapshot sata_sil24 thermal fan edd cpuid usbhid ehci_hcd ohci_pci ohci_hcd ehci_pci dm-crypt cryptd crypto_simd br_netfilter sha1-ssse3 sha256-ssse3 sha512-ssse3 sha3_generic tcp_htcp tcp_bbr xennet xenblk and no segfault. so something common/unique to those two mods is involved ... here, find /lib/modules/5.5.11-24.g794b2e9-default/kernel | egrep "/usb[-_]storage|/uas" /lib/modules/5.5.11-24.g794b2e9-default/kernel/drivers/usb/storage/usb-storage.ko /lib/modules/5.5.11-24.g794b2e9-default/kernel/drivers/usb/storage/uas.ko -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
without debug symbols in place, yet gdb /usr/lib/dracut/dracut-install ... (gdb) r -d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m usb_storage Starting program: /usr/lib/dracut/dracut-install -d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m usb_storage Missing separate debuginfos, use: zypper install dracut-debuginfo-050+suse.58.g0b998bee-lp151.6.1.x86_64 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7a9ae27 in _IO_default_xsputn () from /lib64/libc.so.6 (gdb) bt #0 0x00007ffff7a9ae27 in _IO_default_xsputn () from /lib64/libc.so.6 #1 0x00007ffff7a6d105 in vfprintf () from /lib64/libc.so.6 #2 0x00007ffff7b245a9 in __vsnprintf_chk () from /lib64/libc.so.6 #3 0x00007ffff7b244d5 in __snprintf_chk () from /lib64/libc.so.6 #4 0x00007ffff77eff02 in ?? () from /usr/lib64/libkmod.so.2 #5 0x00007ffff77f07fb in ?? () from /usr/lib64/libkmod.so.2 #6 0x00007ffff77f57e6 in ?? () from /usr/lib64/libkmod.so.2 #7 0x00007ffff77f5979 in ?? () from /usr/lib64/libkmod.so.2 #8 0x00007ffff77f6530 in kmod_module_get_softdeps () from /usr/lib64/libkmod.so.2 #9 0x000055555555a8a8 in install_dependent_modules () #10 0x000055555555a754 in install_dependent_modules () #11 0x000055555555a8bd in install_dependent_modules () ... (LOTS of these) ... #80410 0x000055555555a754 in install_dependent_modules () #80411 0x000055555555a8bd in install_dependent_modules () #80412 0x000055555555a754 in install_dependent_modules () #80413 0x000055555555aaed in install_module () #80414 0x000055555555b15e in install_modules () #80415 0x000055555555737e in main () (gdb) show args Argument list to give program being debugged when it is started is "-d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m usb_storage". -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 25. 03. 20, 0:19, PGNet Dev wrote:
without debug symbols in place, yet
This is the right direction you go. The final step is really to install debug symbols and retry... It still might be miscompilation of modules, .modinfo section layout in particular. Without debuginfo, hard to tell... regards, -- js suse labs -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 3/25/20 1:09 AM, Jiri Slaby wrote:
On 25. 03. 20, 0:19, PGNet Dev wrote:
without debug symbols in place, yet
This is the right direction you go. The final step is really to install debug symbols and retry...
It still might be miscompilation of modules, .modinfo section layout in particular. Without debuginfo, hard to tell...
i 'downgraded' to same kernel & dracut versions, but both built with GCC*9*. running mkinitrd -i /boot/initrd-5.5.11-24.g794b2e9-default -k /boot/vmlinuz-5.5.11-24.g794b2e9-default on the just-installed kernel, without reboot, FAILs, with segfault, as above. i'll install debug symbols, and get a trace ... -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 3/25/20 8:23 AM, PGNet Dev wrote:
i'll install debug symbols, and get a trace ...
as 1st, next step, this is for GCC*9*-built dracut & kernel; if more/different's needed, feel free to specify gdb /usr/lib/dracut/dracut-install ... Reading symbols from /usr/lib/dracut/dracut-install... Reading symbols from /usr/lib/debug/usr/lib/dracut/dracut-install-050+suse.58.g0b998bee-lp151.8.1.x86_64.debug... (gdb) ... (gdb) r -d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m usb_storage Starting program: /usr/lib/dracut/dracut-install -d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m usb_storage [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7a9d2d7 in __GI__IO_default_xsputn (f=0x7fffff7ff5d0, data=<optimized out>, n=40) at genops.c:429 429 genops.c: No such file or directory. (gdb) bt #0 0x00007ffff7a9d2d7 in __GI__IO_default_xsputn (f=0x7fffff7ff5d0, data=<optimized out>, n=40) at genops.c:429 #1 0x00007ffff7a6f4c5 in _IO_vfprintf_internal (s=s@entry=0x7fffff7ff5d0, format=format@entry=0x7ffff780d041 "%s/%s.bin", ap=ap@entry=0x7fffff7ff750) at vfprintf.c:1643 #2 0x00007ffff7b26ab9 in ___vsnprintf_chk (s=0x7fffff7ff840 "", maxlen=<optimized out>, flags=1, slen=<optimized out>, format=0x7ffff780d041 "%s/%s.bin", args=args@entry=0x7fffff7ff750) at vsnprintf_chk.c:63 #3 0x00007ffff7b269e5 in ___snprintf_chk (s=s@entry=0x7fffff7ff840 "", maxlen=maxlen@entry=4096, flags=flags@entry=1, slen=slen@entry=4096, format=format@entry=0x7ffff780d041 "%s/%s.bin") at snprintf_chk.c:34 #4 0x00007ffff77eef02 in snprintf (__fmt=0x7ffff780d041 "%s/%s.bin", __n=4096, __s=0x7fffff7ff840 "") at /usr/include/bits/stdio2.h:64 #5 kmod_search_moddep (ctx=<optimized out>, name=0x7fffff8008b0 "uas") at libkmod/libkmod.c:579 #6 0x00007ffff77ef7fb in kmod_lookup_alias_from_moddep_file (ctx=ctx@entry=0x555555769520, name=name@entry=0x7fffff8008b0 "uas", list=list@entry=0x7fffff8018f0) at libkmod/libkmod.c:609 #7 0x00007ffff77f47e6 in kmod_module_new_from_lookup (ctx=0x555555769520, given_alias=<optimized out>, list=0x7fffff8018f0) at libkmod/libkmod-module.c:563 #8 0x00007ffff77f4979 in lookup_softdep (ctx=0x555555769520, array=<optimized out>, count=<optimized out>) at libkmod/libkmod-module.c:1528 #9 0x00007ffff77f5530 in kmod_module_get_softdeps (mod=0x55555576a6c0, pre=0x7fffff801998, post=0x7fffff8019a0) at libkmod/libkmod-module.c:1579 #10 0x000055555555ac10 in install_dependent_modules (modlist=0x555555c5a6f0) at install/dracut-install.c:1525 #11 0x000055555555aa65 in install_dependent_modules (modlist=0x555555c5a6d0) at install/dracut-install.c:1523 #12 0x000055555555ac25 in install_dependent_modules (modlist=0x555555cbe500) at install/dracut-install.c:1527 #13 0x000055555555aa65 in install_dependent_modules (modlist=0x555555cbe4e0) at install/dracut-install.c:1523 #14 0x000055555555ac25 in install_dependent_modules (modlist=0x555555cbe090) at install/dracut-install.c:1527 #15 0x000055555555aa65 in install_dependent_modules (modlist=0x555555cbe540) at install/dracut-install.c:1523 #16 0x000055555555ac25 in install_dependent_modules (modlist=0x555555cbd160) at install/dracut-install.c:1527 ... loops endlessly ... #77307 0x000055555555aa65 in install_dependent_modules (modlist=0x5555557d7fb0) at install/dracut-install.c:1523 #77308 0x000055555555ac25 in install_dependent_modules (modlist=0x5555557d3f00) at install/dracut-install.c:1527 ... etc ... -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
checking re: 0x00007ffff7a9d2d7 in __GI__IO_default_xsputn (f=0x7fffff7ff5d0, data=<optimized out>, n=40) at genops.c:429 429 genops.c: No such file or directory. in upstream sources cd glibc git branch -a | grep \* * (HEAD detached at origin/release/2.26/master) git describe glibc-2.26-237-g2b802d55b7 cat libio/genops.c ... _IO_size_t _IO_default_xsputn (_IO_FILE *f, const void *data, _IO_size_t n) { const char *s = (char *) data; _IO_size_t more = n; if (more <= 0) return 0; for (;;) { 429 /* Space available. */ if (f->_IO_write_ptr < f->_IO_write_end) { _IO_size_t count = f->_IO_write_end - f->_IO_write_ptr; if (count > more) count = more; if (count > 20) { #ifdef _LIBC ... -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Am 25.03.20 um 21:09 schrieb PGNet Dev:
checking re:
0x00007ffff7a9d2d7 in __GI__IO_default_xsputn (f=0x7fffff7ff5d0, data=<optimized out>, n=40) at genops.c:429 429 genops.c: No such file or directory> in upstream sources
This is not the place where the problem is. look in the https://github.com/opensuse/dracut.git sources, branch SUSE/050 The function install_dependent_modules calls itself, and apparently does an endless recursion. The crash happens probably after memory is exhausted or such. Since this function did not change since version 049, I'd think it is likely that something changed in libkmod or something changed in the module metadata of the kernel that now triggers this. So just discuss this with the suse dracut maintainers in your bugzilla entry, it is quite likely that the kernel hackers on opensuse-kernel are the wrong audience for this problem ;-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 3/25/20 1:43 PM, Stefan Seyfried wrote:
in upstream sources
This is not the place where the problem is.
look in the https://github.com/opensuse/dracut.git sources, branch SUSE/050
The function install_dependent_modules calls itself, and apparently does
an endless recursion. The crash happens probably after memory is
exhausted or such.
Since this function did not change since version 049, I'd think it is
likely that something changed in libkmod or something changed in the
module metadata of the kernel that now triggers this.
thx. noted.
So just discuss this with the suse dracut maintainers in your bugzilla
entry, it is quite likely that the kernel hackers on opensuse-kernel are
the wrong audience for this problem ;-)
there's no bugzilla entry -- as this is not about a released product, but pre-release find-n-fix. happy to take it elsewhere if that's needed, or leave it here. @danimo, where's preferred 4 you? -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 3/25/20 5:32 PM, PGNet Dev wrote:
Since this function did not change since version 049, I'd think it is
likely that something changed in libkmod or something changed in the
module metadata of the kernel that now triggers this.
in the immediate interim ... dropping from dracut/050, @ https://build.opensuse.org/package/show/home:pgnd:Base:System/dracut to dracut/049, @ https://build.opensuse.org/package/show/home:pgnd:Base:System:dracut-049/dra... which builds/installs with no error, results in exec of 'downgraded' dracut/mkinitrd with _NO_ segfault. that's with same kernel, *kmod, etc. and, the generated initrd, with this^ kernel/*kmod, is complete -- or complete enough -- to boot a couple of test boxes. i.e., back to 'normal ops', with dracut/049 -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 25. 03. 20, 20:45, PGNet Dev wrote:
On 3/25/20 8:23 AM, PGNet Dev wrote:
i'll install debug symbols, and get a trace ...
as 1st, next step, this is for GCC*9*-built dracut & kernel; if more/different's needed, feel free to specify
gdb /usr/lib/dracut/dracut-install ... Reading symbols from /usr/lib/dracut/dracut-install... Reading symbols from /usr/lib/debug/usr/lib/dracut/dracut-install-050+suse.58.g0b998bee-lp151.8.1.x86_64.debug... (gdb)
... (gdb) r -d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m usb_storage Starting program: /usr/lib/dracut/dracut-install -d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m usb_storage [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7a9d2d7 in __GI__IO_default_xsputn (f=0x7fffff7ff5d0, data=<optimized out>, n=40) at genops.c:429 429 genops.c: No such file or directory. (gdb) bt
#0 0x00007ffff7a9d2d7 in __GI__IO_default_xsputn (f=0x7fffff7ff5d0, data=<optimized out>, n=40) at genops.c:429 #1 0x00007ffff7a6f4c5 in _IO_vfprintf_internal (s=s@entry=0x7fffff7ff5d0, format=format@entry=0x7ffff780d041 "%s/%s.bin", ap=ap@entry=0x7fffff7ff750) at vfprintf.c:1643 #2 0x00007ffff7b26ab9 in ___vsnprintf_chk (s=0x7fffff7ff840 "", maxlen=<optimized out>, flags=1, slen=<optimized out>, format=0x7ffff780d041 "%s/%s.bin", args=args@entry=0x7fffff7ff750) at vsnprintf_chk.c:63 #3 0x00007ffff7b269e5 in ___snprintf_chk (s=s@entry=0x7fffff7ff840 "", maxlen=maxlen@entry=4096, flags=flags@entry=1, slen=slen@entry=4096, format=format@entry=0x7ffff780d041 "%s/%s.bin") at snprintf_chk.c:34 #4 0x00007ffff77eef02 in snprintf (__fmt=0x7ffff780d041 "%s/%s.bin", __n=4096, __s=0x7fffff7ff840 "") at /usr/include/bits/stdio2.h:64 #5 kmod_search_moddep (ctx=<optimized out>, name=0x7fffff8008b0 "uas") at libkmod/libkmod.c:579
Could you dump ctx here? I.e. do "up" until you get to this frame and then "p *ctx" here. It looks like ctx->dirname is empty. So get_kernel_release is defunct... If so, I would debug kmod_new and get_kernel_release by adding a break point to kmod_new and stepping. Do not forget to install kmod debugsource. regards, -- js suse labs -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 3/25/20 11:53 PM, Jiri Slaby wrote:
bumped kernel - 5.5.11-24.g794b2e9-default + 5.5.12-24.g19257da-default rpm -qa dracut* dracut-050+suse.58.g0b998bee-lp151.8.1.x86_64 dracut-debuginfo-050+suse.58.g0b998bee-lp151.8.1.x86_64
Do not forget to install kmod debugsource.
rpm -qa *kmod* *glibc* | grep debugsource glibc-debugsource-2.26-lp151.19.3.1.x86_64 glibc-utils-src-debugsource-2.26-lp151.19.3.1.x86_64 kmod-debugsource-25-lp151.6.1.x86_64 gdb /usr/lib/dracut/dracut-install (gdb) r -d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.12-24.g19257da-default/ -m usb_storage ... Program received signal SIGSEGV, Segmentation fault. _IO_vfprintf_internal (s=s@entry=0x7fffff7ff550, format=format@entry=0x7ffff780d041 "%s/%s.bin", ap=ap@entry=0x7fffff7ff6d0) at vfprintf.c:1320 1320 f = lead_str_end = __find_specmb ((const UCHAR_T *) format); (gdb) bt #0 _IO_vfprintf_internal (s=s@entry=0x7fffff7ff550, format=format@entry=0x7ffff780d041 "%s/%s.bin", ap=ap@entry=0x7fffff7ff6d0) at vfprintf.c:1320 #1 0x00007ffff7b26ab9 in ___vsnprintf_chk (s=0x7fffff7ff7c0 "", maxlen=<optimized out>, flags=1, slen=<optimized out>, format=0x7ffff780d041 "%s/%s.bin", args=args@entry=0x7fffff7ff6d0) at vsnprintf_chk.c:63 #2 0x00007ffff7b269e5 in ___snprintf_chk (s=s@entry=0x7fffff7ff7c0 "", maxlen=maxlen@entry=4096, flags=flags@entry=1, slen=slen@entry=4096, format=format@entry=0x7ffff780d041 "%s/%s.bin") at snprintf_chk.c:34 #3 0x00007ffff77eef02 in snprintf (__fmt=0x7ffff780d041 "%s/%s.bin", __n=4096, __s=0x7fffff7ff7c0 "") at /usr/include/bits/stdio2.h:64 #4 kmod_search_moddep (ctx=<optimized out>, name=0x7fffff800830 "uas") at libkmod/libkmod.c:579 #5 0x00007ffff77ef7fb in kmod_lookup_alias_from_moddep_file (ctx=ctx@entry=0x555555769520, name=name@entry=0x7fffff800830 "uas", list=list@entry=0x7fffff801870) at libkmod/libkmod.c:609 #6 0x00007ffff77f47e6 in kmod_module_new_from_lookup (ctx=0x555555769520, given_alias=<optimized out>, list=0x7fffff801870) at libkmod/libkmod-module.c:563 #7 0x00007ffff77f4979 in lookup_softdep (ctx=0x555555769520, array=<optimized out>, count=<optimized out>) at libkmod/libkmod-module.c:1528 #8 0x00007ffff77f5530 in kmod_module_get_softdeps (mod=0x55555576a6c0, pre=0x7fffff801918, post=0x7fffff801920) at libkmod/libkmod-module.c:1579 #9 0x000055555555ac10 in install_dependent_modules (modlist=0x555555cbe500) at install/dracut-install.c:1525 #10 0x000055555555aa65 in install_dependent_modules (modlist=0x555555cbe4e0) at install/dracut-install.c:1523 ...
Could you dump ctx here?
... (gdb) up #4 kmod_search_moddep (ctx=<optimized out>, name=0x7fffff800830 "uas") at libkmod/libkmod.c:579 579 snprintf(fn, sizeof(fn), "%s/%s.bin", ctx->dirname, (gdb) p *ctx value has been optimized out (gdb)
by adding a break point to kmod_new and stepping
iiuc, (gdb) b kmod_new Breakpoint 1 at 0x7ffff77ef2f0: file libkmod/libkmod.c, line 251. (gdb) r ... Breakpoint 1, kmod_new (dirname=0x555555767180 "/lib/modules/5.5.12-24.g19257da-default/", config_paths=0x0) at libkmod/libkmod.c:251 251 { (gdb) step 256 ctx = calloc(1, sizeof(struct kmod_ctx)); (gdb) step 251 { (gdb) step 256 ctx = calloc(1, sizeof(struct kmod_ctx)); (gdb) step __libc_calloc (n=1, elem_size=120) at malloc.c:3368 3368 { (gdb) step ... (gdb) step 3509 } (gdb) step kmod_new (dirname=0x555555767180 "/lib/modules/5.5.12-24.g19257da-default/", config_paths=0x0) at libkmod/libkmod.c:257 257 if (!ctx) (gdb) step 256 ctx = calloc(1, sizeof(struct kmod_ctx)); (gdb) step 257 if (!ctx) (gdb) step 261 ctx->log_fn = log_filep; (gdb) step 265 ctx->dirname = get_kernel_release(dirname); (gdb) step 261 ctx->log_fn = log_filep; (gdb) step 262 ctx->log_data = stderr; (gdb) step 260 ctx->refcount = 1; (gdb) step 265 ctx->dirname = get_kernel_release(dirname); (gdb) step get_kernel_release (dirname=dirname@entry=0x555555767180 "/lib/modules/5.5.12-24.g19257da-default/") at libkmod/libkmod.c:212 212 { (gdb) step 216 if (dirname != NULL) (gdb) step 217 return path_make_absolute_cwd(dirname); (gdb) step path_make_absolute_cwd (p=p@entry=0x555555767180 "/lib/modules/5.5.12-24.g19257da-default/") at shared/util.c:362 362 { (gdb) step 367 if (path_is_absolute(p)) (gdb) step 368 return strdup(p); (gdb) step __GI___strdup (s=0x555555767180 "/lib/modules/5.5.12-24.g19257da-default/") at strdup.c:40 40 { (gdb) step 41 size_t len = strlen (s) + 1; (gdb) step __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:79 79 pxor %xmm0, %xmm0 (gdb) step 80 pxor %xmm1, %xmm1 (gdb) step 81 pxor %xmm2, %xmm2 (gdb) step 82 pxor %xmm3, %xmm3 (gdb) step 83 movq %rdi, %rax (gdb) step 84 movq %rdi, %rcx (gdb) step 85 andq $4095, %rcx (gdb) step 87 cmpq $4047, %rcx (gdb) step 89 ja L(cross_page) (gdb) step 120 movdqu (%rax), %xmm4 (gdb) step 121 PCMPEQ %xmm0, %xmm4 (gdb) step 122 pmovmskb %xmm4, %edx (gdb) step 123 test %edx, %edx (gdb) step 124 je L(next48_bytes) (gdb) step 131 andq $-16, %rax (gdb) step 132 PCMPEQ 16(%rax), %xmm1 (gdb) step 133 PCMPEQ 32(%rax), %xmm2 (gdb) step 134 PCMPEQ 48(%rax), %xmm3 (gdb) step 135 pmovmskb %xmm1, %edx (gdb) step 136 pmovmskb %xmm2, %r8d (gdb) step 137 pmovmskb %xmm3, %ecx (gdb) step 138 salq $16, %rdx (gdb) step 139 salq $16, %rcx (gdb) step 140 orq %r8, %rcx (gdb) step 141 salq $32, %rcx (gdb) step 142 orq %rcx, %rdx (gdb) step 147 PROLOG(loop) (gdb) step __GI___strdup (s=0x555555767180 "/lib/modules/5.5.12-24.g19257da-default/") at strdup.c:42 42 void *new = malloc (len); (gdb) step Program received signal SIGSEGV, Segmentation fault. _IO_vfprintf_internal (s=s@entry=0x7fffff7ff550, format=format@entry=0x7ffff780d041 "%s/%s.bin", ap=ap@entry=0x7fffff7ff6d0) at vfprintf.c:1320 1320 f = lead_str_end = __find_specmb ((const UCHAR_T *) format); (gdb) -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hello, On Thu, 26 Mar 2020, Jiri Slaby wrote:
gdb /usr/lib/dracut/dracut-install ... Reading symbols from /usr/lib/dracut/dracut-install... Reading symbols from /usr/lib/debug/usr/lib/dracut/dracut-install-050+suse.58.g0b998bee-lp151.8.1.x86_64.debug... (gdb)
... (gdb) r -d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m usb_storage Starting program: /usr/lib/dracut/dracut-install -d -D /var/tmp/dracut.TEST/initramfs -N i2o_scsi --kerneldir /lib/modules/5.5.11-24.g794b2e9-default/ -m usb_storage [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7a9d2d7 in __GI__IO_default_xsputn (f=0x7fffff7ff5d0, data=<optimized out>, n=40) at genops.c:429 429 genops.c: No such file or directory. (gdb) bt
#0 0x00007ffff7a9d2d7 in __GI__IO_default_xsputn (f=0x7fffff7ff5d0, data=<optimized out>, n=40) at genops.c:429 #1 0x00007ffff7a6f4c5 in _IO_vfprintf_internal (s=s@entry=0x7fffff7ff5d0, format=format@entry=0x7ffff780d041 "%s/%s.bin", ap=ap@entry=0x7fffff7ff750) at vfprintf.c:1643 #2 0x00007ffff7b26ab9 in ___vsnprintf_chk (s=0x7fffff7ff840 "", maxlen=<optimized out>, flags=1, slen=<optimized out>, format=0x7ffff780d041 "%s/%s.bin", args=args@entry=0x7fffff7ff750) at vsnprintf_chk.c:63 #3 0x00007ffff7b269e5 in ___snprintf_chk (s=s@entry=0x7fffff7ff840 "", maxlen=maxlen@entry=4096, flags=flags@entry=1, slen=slen@entry=4096, format=format@entry=0x7ffff780d041 "%s/%s.bin") at snprintf_chk.c:34 #4 0x00007ffff77eef02 in snprintf (__fmt=0x7ffff780d041 "%s/%s.bin", __n=4096, __s=0x7fffff7ff840 "") at /usr/include/bits/stdio2.h:64 #5 kmod_search_moddep (ctx=<optimized out>, name=0x7fffff8008b0 "uas") at libkmod/libkmod.c:579
Could you dump ctx here? I.e. do "up" until you get to this frame and then "p *ctx" here. It looks like ctx->dirname is empty. So get_kernel_release is defunct... If so, I would debug kmod_new and get_kernel_release by adding a break point to kmod_new and stepping. Do not forget to install kmod debugsource.
No need for any of this really. The fact that there were ca. 80000 frames upstack all containing the same function calling itself does show that the segfault simply is stack exhaustion and Seife found the culprit for that already. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 3/26/20 10:09 AM, Michael Matz wrote:
No need for any of this really. The fact that there were ca. 80000 frames upstack all containing the same function calling itself does show that the segfault simply is stack exhaustion and Seife found the culprit for that already.
is that^ fix split-off/pending elsewhere? got a link perhaps? or TBD here ? -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hello, On Sun, 29 Mar 2020, PGNet Dev wrote:
On 3/26/20 10:09 AM, Michael Matz wrote:
No need for any of this really. The fact that there were ca. 80000 frames upstack all containing the same function calling itself does show that the segfault simply is stack exhaustion and Seife found the culprit for that already.
is that^ fix split-off/pending elsewhere? got a link perhaps? or TBD here ?
I have no insights into dracut development, I was going from the info in this thread. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
participants (4)
-
Jiri Slaby -
Michael Matz -
PGNet Dev -
Stefan Seyfried