[Bug 1229450] New: libbpf: failed to find '.BTF' ELF section in vmlinux
https://bugzilla.suse.com/show_bug.cgi?id=1229450 Bug ID: 1229450 Summary: libbpf: failed to find '.BTF' ELF section in vmlinux Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-bugs@opensuse.org Reporter: jslaby@suse.com QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: --- Blocker: --- Created attachment 876820 --> https://bugzilla.suse.com/attachment.cgi?id=876820&action=edit Devel:Kernel:master/kernel-source:kernel-vanilla/LEGACYX86/i586 build log The kernel builds result in random build failures while creating BTF. For example https://build.opensuse.org/package/live_build_log/Kernel:vanilla/kernel-sour...:
libbpf: failed to find '.BTF' ELF section in vmlinux FAILED: load BTF from vmlinux: No data available make[2]: *** [../scripts/Makefile.vmlinux:34: vmlinux] Error 255 make[2]: *** Deleting file 'vmlinux' make[1]: *** [/home/abuild/rpmbuild/BUILD/kernel-vanilla-6.11~rc3.338.gc3f2d783a459/linux-6.11-rc3-338-gc3f2d783a459/Makefile:1158: vmlinux] Error 2 make: *** [../Makefile:224: __sub-make] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.olf5Nu (%build)
arm seems to fail reliably. But for example in ibs://Devel:Kernel:master/kernel-source:kernel-vanilla/LEGACYX86/i586, it looks like it is sort of random. kernel-debug and kernel-vanilla fails for i586, but -default and -pae are built currently. See the job history for kernel-vanilla in there (the failed boot log is attached):
time package reason code build time worker 2024-07-15 07:41:48 kernel-source:kernel-vanilla source change succeeded 47m 53s h01-ch5a:3 2024-07-15 23:56:48 kernel-source:kernel-vanilla source change succeeded 47m 57s h01-ch5b:5 2024-07-18 11:12:40 kernel-source:kernel-vanilla source change succeeded 48m 46s h01-ch5b:2 2024-07-26 20:01:29 kernel-source:kernel-vanilla source change succeeded 52m 40s h01-ch4b:2 2024-07-29 09:07:35 kernel-source:kernel-vanilla source change succeeded 52m 53s h01-ch4a:6 2024-08-03 11:38:55 kernel-source:kernel-vanilla source change failed 25m 16s h01-ch4b:8 2024-08-04 23:48:15 kernel-source:kernel-vanilla source change failed 25m 14s h01-ch4b:7 2024-08-06 16:37:35 kernel-source:kernel-vanilla source change succeeded 53m 25s h01-ch4b:4 2024-08-09 19:44:44 kernel-source:kernel-vanilla source change failed 26m 4s h01-ch5a:2 2024-08-11 22:48:56 kernel-source:kernel-vanilla source change failed 0m 38s h01-ch4b:6 2024-08-12 16:44:52 kernel-source:kernel-vanilla source change succeeded 54m 0s h01-ch5b:5 2024-08-15 21:49:11 kernel-source:kernel-vanilla source change failed 25m 25s h01-ch4b:1 2024-08-19 01:53:01 kernel-source:kernel-vanilla source change succeeded 50m 9s h01-ch4a:5 2024-08-19 11:33:50 kernel-source:kernel-vanilla source change succeeded 55m 9s h01-ch5b:1 2024-08-19 17:23:59 kernel-source:kernel-vanilla source change failed 25m 16s h01-ch5a:3 -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 Jiri Slaby <jslaby@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|kernel-bugs@opensuse.org |shung-hsi.yu@suse.com CC| |mkoutny@suse.com, | |msuchanek@suse.com, | |tonyj@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c1 --- Comment #1 from Jiri Slaby <jslaby@suse.com> --- The most important part from the log:
btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory. btf_encoder__encode: failed to tag kfuncs!
It's clear why this is random... -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c2 --- Comment #2 from Jiri Slaby <jslaby@suse.com> --- (In reply to Jiri Slaby from comment #1)
The most important part from the log:
btf_encoder__tag_kfuncs: Failed to get ELF section(62) data: out of memory.
BTW the machine is run with -m 40960. But given it happens either on full 32bit (armv7l builds) or 64bit kernel + 32bit userspace (i586), the encoder likely needs slightly more than 3G of space... It might be reproducible locally with ulimit -m $((3<<20)). Or with a cgroup (mkoutny might provide the right sequence of echos to cgroup fs). -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 Michal Kubeček <mkubecek@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mkubecek@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c3 --- Comment #3 from Jiri Slaby <jslaby@suse.com> --- Created attachment 876821 --> https://bugzilla.suse.com/attachment.cgi?id=876821&action=edit mappings of the crashed process The VM appears to be completely full on 32bit. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c4 --- Comment #4 from Jiri Slaby <jslaby@suse.com> --- Created attachment 876822 --> https://bugzilla.suse.com/attachment.cgi?id=876822&action=edit strace of mmaps pahole simply runs out of virtual memory. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c5 --- Comment #5 from Michal Suchanek <msuchanek@suse.com> --- Then disable BTF on 32bit and be done with it. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c6 --- Comment #6 from Jiri Slaby <jslaby@suse.com> --- (In reply to Michal Suchanek from comment #5)
Then disable BTF on 32bit and be done with it.
Perhaps. Leaving up to BTF guys. FWIW, steps to reproduce locally: docker pull jirislaby/pahole_crash docker run -it jirislaby/pahole_crash It will run pahole in gdb. You can quit it, then there is vmlinux called "my" and my.cmd script to invoke pahole (under gdb). -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c7 --- Comment #7 from Jiri Slaby <jslaby@suse.com> --- (In reply to Jiri Slaby from comment #6)
(In reply to Michal Suchanek from comment #5)
Then disable BTF on 32bit and be done with it.
Perhaps. Leaving up to BTF guys.
FWIW, steps to reproduce locally: docker pull jirislaby/pahole_crash docker run -it jirislaby/pahole_crash
Ah, it's not reproducible on my machine with 16 CPU cores. On a machine with 160 cores, it is. So it likely depends on the number of cores. pahole creates a lot of threads, I see. So maybe limit the number of threads/cores to use by pahole? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c8 --- Comment #8 from Michal Suchanek <msuchanek@suse.com> --- -j, --jobs=N Run N jobs in parallel. Defaults to number of online processors + 10% (like the 'ninja' build system) if no argument is specified. So with 160 CPUs you would get 176 threads. There is scripts/pahole-flags.sh in the kernel tree which determines the flags to pass to pahole, I think it's reasonable to patch that. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c9 --- Comment #9 from Jiri Slaby <jslaby@suse.com> --- FWIW: -j, --jobs[=NR_JOBS] run N jobs in parallel [default to number of online processors + 10%] But how many? The build logs show -smp 12 for i586 and 4 for arm and they still fail. 12 gives 13, 4 gives 4, 160 gives 176. If I use -j32 on my machine, it indeed fails. So I updated the docker image accordingly. Maybe we should just use -j1 (or no -j at all) on 32bit as with pahole < 122. See scripts/Makefile.btf: pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j (Or disable btf on 32bit completely.) -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c10 --- Comment #10 from Jiri Slaby <jslaby@suse.com> --- (In reply to Michal Suchanek from comment #8)
There is scripts/pahole-flags.sh in the kernel tree which determines the flags to pass to pahole, I think it's reasonable to patch that.
Maybe, only it is now in scripts/Makefile.btf since: commit 72d091846de935e0942a8a0f1fe24ff739d85d76 Author: Masahiro Yamada <masahiroy@kernel.org> Date: Thu Oct 19 00:19:48 2023 +0900 kbuild: avoid too many execution of scripts/pahole-flags.sh (6.7) -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c11 --- Comment #11 from Michal Suchanek <msuchanek@suse.com> --- I am not sure how to detect running on 32bit, especially in the case that it's 32bit userspace on 64bit kernel. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c12 --- Comment #12 from Jiri Slaby <jslaby@suse.com> --- --- a/scripts/Makefile.btf +++ b/scripts/Makefile.btf @@ -12,7 +12,9 @@ endif pahole-flags-$(call test-ge, $(pahole-ver), 121) += --btf_gen_floats +ifeq ($(CONFIG_64BIT),y) pahole-flags-$(call test-ge, $(pahole-ver), 122) += -j +endif pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized will likely work. But we should likely detect architecture of pahole, not the target kernel. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c13 --- Comment #13 from Jiri Slaby <jslaby@suse.com> --- (In reply to Michal Suchanek from comment #11)
I am not sure how to detect running on 32bit, especially in the case that it's 32bit userspace on 64bit kernel.
Some kind of checking of ELF64 eq: readelf -h /usr/bin/pahole | sed -n 's/.*Class: *// p' -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c14 --- Comment #14 from Jiri Slaby <jslaby@suse.com> --- Created attachment 876826 --> https://bugzilla.suse.com/attachment.cgi?id=876826&action=edit potential fix (In reply to Jiri Slaby from comment #13)
Some kind of checking of ELF64 eq: readelf -h /usr/bin/pahole | sed -n 's/.*Class: *// p'
So something like this? Let me bring this upstream. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c15 --- Comment #15 from Shung-Hsi Yu <shung-hsi.yu@suse.com> --- Hmm I didn't know pahole use so much memory per thread (not that I'm familiar with it, yet). Please CC me in the upstream mail as well if possible. Thanks! -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c16 --- Comment #16 from Michal Suchanek <msuchanek@suse.com> --- (In reply to Jiri Slaby from comment #14)
Created attachment 876826 [details] potential fix
(In reply to Jiri Slaby from comment #13)
Some kind of checking of ELF64 eq: readelf -h /usr/bin/pahole | sed -n 's/.*Class: *// p'
So something like this? Let me bring this upstream.
readelf is provided by GNU binutils. What about LLVM builds? Also pahole is specified by PAHOLE variable, and it may include arguments because shell and stuff. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c17 --- Comment #17 from Shung-Hsi Yu <shung-hsi.yu@suse.com> --- (In reply to Michal Suchanek from comment #16) [...]
readelf is provided by GNU binutils. What about LLVM builds?
Also pahole is specified by PAHOLE variable, and it may include arguments because shell and stuff.
Would it be better to have pahole to cap the thread count depending on virtual memory available irregardless to architecture? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c18 --- Comment #18 from Jiri Slaby <jslaby@suse.com> --- (In reply to Michal Suchanek from comment #16)
readelf is provided by GNU binutils. What about LLVM builds?
Also pahole is specified by PAHOLE variable, and it may include arguments because shell and stuff.
Yes, it's broken in many ways. Sent to upstream for discussion as an RFC: https://lore.kernel.org/all/20240820085950.200358-1-jirislaby@kernel.org/ (In reply to Shung-Hsi Yu from comment #17)
Would it be better to have pahole to cap the thread count depending on virtual memory available irregardless to architecture?
Maybe set it to 1 when sizeof(long) == 4? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 Sangeetha Thackarajan <sangeetha.thackarajan@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sangeetha.thackarajan@suse. | |com -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c19 --- Comment #19 from Jiri Slaby <jslaby@suse.com> --- (In reply to Jiri Slaby from comment #14)
Created attachment 876826 [details] potential fix
FWIW that seems to work (in our workflow only) -- just to confirm -j1 does the right thing: https://build.suse.de/project/monitor/home:jirislaby:pahole -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c20 --- Comment #20 from Jiri Slaby <jslaby@suse.com> --- Created attachment 876861 --> https://bugzilla.suse.com/attachment.cgi?id=876861&action=edit valgrind --tool=massif output Run on 64bit: pahole -j32 -> 4.102 GB pahole -j16 -> 3.895 GB pahole -j1 -> 3.706 GB On 32bit: pahole -j32 -> 2.870 GB (crash) pahole -j16 -> 2.810 GB pahole -j1 -> 2.444 GB The first one reached the limit. The other two are not really much better... The top eater:
99.74% (3,966,953,649B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. ->87.57% (3,482,844,356B) 0x49B7EB4: _obstack_newchunk (obstack.c:261) | ->83.08% (3,304,324,608B) 0x488BF53: UnknownInlinedFun (dwarves.c:40) | | ->83.08% (3,304,324,608B) 0x488BF53: cu__zalloc (dwarves.c:50) | | ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun (dwarf_loader.c:462) | | | ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun (dwarf_loader.c:477) | | | ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun (dwarf_loader.c:959) | | | ->24.01% (954,816,480B) 0x489B4AB: die__process_class (dwarf_loader.c:1815) | | | ->22.53% (896,148,576B) 0x489A129: UnknownInlinedFun (dwarf_loader.c:1492) | | | | ->22.53% (896,148,576B) 0x489A129: __die__process_tag (dwarf_loader.c:2203) | | | | ->22.53% (896,079,488B) 0x489C250: die__process_unit (dwarf_loader.c:2243) | | | | | ->22.53% (896,079,488B) 0x489C364: die__process (dwarf_loader.c:2858) | | | | | ->22.53% (896,079,488B) 0x489CE5C: UnknownInlinedFun (dwarf_loader.c:2873) | | | | | ->22.53% (896,079,488B) 0x489CE5C: UnknownInlinedFun (dwarf_loader.c:3243) | | | | | ->22.53% (896,079,488B) 0x489CE5C: UnknownInlinedFun (dwarf_loader.c:3259) | | | | | ->22.53% (896,079,488B) 0x489CE5C: UnknownInlinedFun (dwarf_loader.c:3388) | | | | | ->22.53% (896,079,488B) 0x489CE5C: UnknownInlinedFun (dwarf_loader.c:3402) | | | | | ->22.53% (896,079,488B) 0x489CE5C: UnknownInlinedFun (dwarf_loader.c:3537) | | | | | ->22.53% (896,079,488B) 0x489CE5C: cus__process_dwflmod (dwarf_loader.c:3581) | | | | | ->22.53% (896,079,488B) 0x4B66130: dwfl_getmodules (in /usr/lib64/libdw-0.191.so) | | | | | ->22.53% (896,079,488B) 0x489D73F: UnknownInlinedFun (dwarf_loader.c:3647) | | | | | ->22.53% (896,079,488B) 0x489D73F: dwarf__load_file.lto_priv.0 (dwarf_loader.c:3684) | | | | | ->22.53% (896,079,488B) 0x488928E: cus__load_file (dwarves.c:2134) | | | | | ->22.53% (896,079,488B) 0x488DD67: cus__load_files (dwarves.c:2637) | | | | | ->22.53% (896,079,488B) 0x10CB07: main (pahole.c:3805) -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c21 --- Comment #21 from Jiri Slaby <jslaby@suse.com> --- (In reply to Jiri Slaby from comment #20)
| | | ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun (dwarf_loader.c:959)
So given this struct class_member is the largest consumer, running pahole on pahole. The below results in 4.102 GB -> 3.585 GB savings. --- a/dwarves.h +++ b/dwarves.h @@ -487,14 +487,14 @@ int cu__for_all_tags(struct cu *cu, */ struct tag { struct list_head node; + const char *attribute; + void *priv; type_id_t type; uint16_t tag; + uint16_t recursivity_level; bool visited; bool top_level; bool has_btf_type_tag; - uint16_t recursivity_level; - const char *attribute; - void *priv; }; // To use with things like type->type_enum == perf_event_type+perf_user_event_type @@ -1086,17 +1086,17 @@ static inline int function__inlined(const struct function *func) struct class_member { struct tag tag; const char *name; + uint64_t const_value; uint32_t bit_offset; uint32_t bit_size; uint32_t byte_offset; int hole; size_t byte_size; + uint32_t alignment; int8_t bitfield_offset; uint8_t bitfield_size; uint8_t bit_hole; uint8_t bitfield_end:1; - uint64_t const_value; - uint32_t alignment; uint8_t visited:1; uint8_t is_static:1; uint8_t has_bit_offset:1; -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c22 --- Comment #22 from Jiri Slaby <jslaby@suse.com> --- (In reply to Michal Suchanek from comment #5)
Then disable BTF on 32bit and be done with it.
This bug doesn't look optimistic wrt to a proper fix in the near future. So let's really disable BTF on 32bit archs (armv6hl, armv7hl, i386) for now: commit c02f88f4934e94c23c37449407ef90166dbdcdec (HEAD -> master, origin/users/jslaby/master/for-next) Author: Jiri Slaby <jslaby@suse.cz> Date: Fri Aug 30 08:51:34 2024 +0200 Update config files. Disable BTF on 32bit architectures (bsc#1229450) -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 https://bugzilla.suse.com/show_bug.cgi?id=1229450#c23 --- Comment #23 from Jiri Slaby <jslaby@suse.com> --- (In reply to Jiri Slaby from comment #21)
(In reply to Jiri Slaby from comment #20)
| | | ->24.01% (954,816,480B) 0x489B4AB: UnknownInlinedFun (dwarf_loader.c:959)
So given this struct class_member is the largest consumer, running pahole on pahole. The below results in 4.102 GB -> 3.585 GB savings.
FTR, the upstream is going to release a new pahole (dwarves 1.28) with many structures and allocations compaction. This however does not help in -j$CPUS case on 32bit. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1229450 Maintenance Automation <maint-coord+maintenance-robot@suse.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com