https://bugzilla.suse.com/show_bug.cgi?id=1188589
Bug ID: 1188589 Summary: libbpf: failed to find valid kernel BTF Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-bugs@opensuse.org Reporter: mrostecki@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: ---
Since the update to the TW snapshot with the kernel 5.13.2-1, I see the following errors when trying to load BPF CO-RE skeletons with libbpf:
libbpf: failed to find valid kernel BTF libbpf: Error loading vmlinux BTF: -3
The full example - I'm trying to load BPF programs which I have in this project: https://github.com/rancher-sandbox/lockc
and the full output is:
��� sudo ./out/lockcd libbpf: elf: skipping unrecognized data section(9) .rodata.str1.1 libbpf: failed to find valid kernel BTF libbpf: Error loading vmlinux BTF: -3 libbpf: failed to load object 'lockc_bpf' libbpf: failed to load BPF skeleton 'lockc_bpf': -3 Error: libbpf error Caused by: System error, errno: 3
Most likely, the solution for them would be setting the following kernel parameters to "y":
CONFIG_DEBUG_INFO_REDUCED CONFIG_DEBUG_INFO_SPLIT
But I'm not entirely sure yet. trying to test that...
https://bugzilla.suse.com/show_bug.cgi?id=1188589 https://bugzilla.suse.com/show_bug.cgi?id=1188589#c1
--- Comment #1 from Micha�� Rostecki mrostecki@suse.com --- This is the strace output:
access("/boot/vmlinux-5.13.2-1-default", R_OK) = -1 ENOENT (No such file or directory) access("/lib/modules/5.13.2-1-default/vmlinux-5.13.2-1-default", R_OK) = -1 ENOENT (No such file or directory) access("/lib/modules/5.13.2-1-default/build/vmlinux", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/modules/5.13.2-1-default/kernel/vmlinux", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/debug/boot/vmlinux-5.13.2-1-default", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/debug/boot/vmlinux-5.13.2-1-default.debug", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/debug/lib/modules/5.13.2-1-default/vmlinux", R_OK) = -1 ENOENT (No such file or directory) write(2, "libbpf: failed to find valid ker"..., 40libbpf: failed to find valid kernel BTF
Seems like we are not exposing BTF info in any directory that libbpf expects it to be in.
https://bugzilla.suse.com/show_bug.cgi?id=1188589 https://bugzilla.suse.com/show_bug.cgi?id=1188589#c2
--- Comment #2 from Micha�� Rostecki mrostecki@suse.com --- We have the
/sys/kernel/btf/vmlinux
file though. Wondering why libbpf doesn't honor it...
https://bugzilla.suse.com/show_bug.cgi?id=1188589 https://bugzilla.suse.com/show_bug.cgi?id=1188589#c3
--- Comment #3 from Micha�� Rostecki mrostecki@suse.com --- full strace output:
https://paste.opensuse.org/3215684
https://bugzilla.suse.com/show_bug.cgi?id=1188589
Micha�� Rostecki mrostecki@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |glin@suse.com
https://bugzilla.suse.com/show_bug.cgi?id=1188589
Gary Ching-Pang Lin glin@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |shung-hsi.yu@suse.com
https://bugzilla.suse.com/show_bug.cgi?id=1188589 https://bugzilla.suse.com/show_bug.cgi?id=1188589#c4
--- Comment #4 from Micha�� Rostecki mrostecki@suse.com --- My bad, I didn't notice before that the /sys/kernel/btf/vmlinux file was actually used, but then libbpf started checking the other files anyway:
access("/sys/kernel/btf/vmlinux", R_OK) = 0 openat(AT_FDCWD, "/sys/kernel/btf/vmlinux", O_RDONLY) = 3 newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=4511209, ...}, AT_EMPTY_PATH) = 0 read(3, "\237\353\1\0\30\0\0\0\0\0\0\0\350\223(\0\350\223(\0\351A\34\0\1\0\0\0\0\0\0\1"..., 4096) = 4096 newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=4511209, ...}, AT_EMPTY_PATH) = 0 lseek(3, 4509696, SEEK_SET) = 4509696 read(3, "t\0pirq_vlsi_get\0pirq_sis_set\0pir"..., 4096) = 1513 lseek(3, 0, SEEK_SET) = 0 mmap(NULL, 4513792, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3945622000 read(3, "\237\353\1\0\30\0\0\0\0\0\0\0\350\223(\0\350\223(\0\351A\34\0\1\0\0\0\0\0\0\1"..., 4509696) = 4096 read(3, "\224\n\0\0Y\2\0\0\0/\0\0\236\n\0\0I\0\0\0\0001\0\0\253\n\0\0I\0\0\0"..., 4505600) = 4096 read(3, "_pm_parse_addr\0mptcp_pm_nl_data_"..., 4096) = 4096 read(3, "t\0pirq_vlsi_get\0pirq_sis_set\0pir"..., 4096) = 1513 mmap(NULL, 4513792, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f39451d4000 close(0) = 0 munmap(0x7f39451d4000, 4513792) = 0 munmap(0x7f3945622000, 4513792) = 0 close(3) = 0 access("/boot/vmlinux-5.13.2-1-default", R_OK) = -1 ENOENT (No such file or directory) access("/lib/modules/5.13.2-1-default/vmlinux-5.13.2-1-default", R_OK) = -1 ENOENT (No such file or directory) access("/lib/modules/5.13.2-1-default/build/vmlinux", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/modules/5.13.2-1-default/kernel/vmlinux", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/debug/boot/vmlinux-5.13.2-1-default", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/debug/boot/vmlinux-5.13.2-1-default.debug", R_OK) = -1 ENOENT (No such file or directory) access("/usr/lib/debug/lib/modules/5.13.2-1-default/vmlinux", R_OK) = -1 ENOENT (No such file or directory) write(2, "libbpf: failed to find valid ker"..., 40libbpf: failed to find valid kernel BTF ) = 40 write(2, "libbpf: Error loading vmlinux BT"..., 38libbpf: Error loading vmlinux BTF: -3 ) = 38 write(2, "libbpf: failed to load object 'l"..., 42libbpf: failed to load object 'lockc_bpf' ) = 42 write(2, "libbpf: failed to load BPF skele"..., 52libbpf: failed to load BPF skeleton 'lockc_bpf': -3
My theory so far is that processing the first vmlinux file fails somewhere here:
https://github.com/torvalds/linux/blob/master/tools/lib/bpf/btf.c#L4431-L443...
But the `pr_debug` message doesn't show up. And what's more, the debug mode can't be enabled in BPF skeletons... it can be set only when executing `bpftool` commands.
I will try to dig in more and see what's happening in that file.
https://bugzilla.suse.com/show_bug.cgi?id=1188589 https://bugzilla.suse.com/show_bug.cgi?id=1188589#c5
--- Comment #5 from Micha�� Rostecki mrostecki@suse.com --- I noticed that dumping BTF with bpftool also doesn't work on >=5.13 openSUSE kernels:
��� sudo bpftool btf dump file /sys/kernel/btf/vmlinux Error: failed to load BTF from /sys/kernel/btf/vmlinux: Invalid argumen
https://bugzilla.suse.com/show_bug.cgi?id=1188589 https://bugzilla.suse.com/show_bug.cgi?id=1188589#c6
Shung-Hsi Yu shung-hsi.yu@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |mrostecki@suse.com Flags| |needinfo?(mrostecki@suse.co | |m)
--- Comment #6 from Shung-Hsi Yu shung-hsi.yu@suse.com --- Hi Micha��, is this still failing?
FWIW I submitted a service request on build.opensuse.org for updating libbpf v0.5, perhaps that will fix it.
https://bugzilla.suse.com/show_bug.cgi?id=1188589 https://bugzilla.suse.com/show_bug.cgi?id=1188589#c7
Jiri Slaby jslaby@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED CC| |jslaby@suse.com Resolution|--- |NORESPONSE
--- Comment #7 from Jiri Slaby jslaby@suse.com --- Feel free to reopen when providing the requested info.
-
bugzilla_noreply＠suse.com