Firefox/i586 needs help: FF 116 fails to build
Dear TW hackers, Firefox 116 has just been sent for inclusion into Tumbleweed - with a note that it fails to build on i586. Looking into the devel prj, the picture looks even worse, as aarch64 and armv7l are also failing to build. https://build.opensuse.org/package/show/mozilla:Factory/MozillaFirefox Can you all please help out the maintainer to get the failing archs building too? (i586 is mandatory in Staging - and I still hope to get aarch64 and ppc64le on that level too one day) Cheers, Dominique
Am 04.08.23 um 10:28 schrieb Dominique Leuenberger:
Dear TW hackers,
Firefox 116 has just been sent for inclusion into Tumbleweed - with a note that it fails to build on i586.
Looking into the devel prj, the picture looks even worse, as aarch64 and armv7l are also failing to build.
https://build.opensuse.org/package/show/mozilla:Factory/MozillaFirefox
Can you all please help out the maintainer to get the failing archs building too? (i586 is mandatory in Staging - and I still hope to get aarch64 and ppc64le on that level too one day)
I haven't checked aarch64 before but it seems not too hard to fix. I'll look into it. And I honestly cannot remember why ppc64le was disabled once. Will enable and see what the state is. Wolfgang
Am 04.08.23 um 10:28 schrieb Dominique Leuenberger:
Firefox 116 has just been sent for inclusion into Tumbleweed - with a note that it fails to build on i586.
For those too lazy to click the link, linking libxul.so (the largest binary in the package and maybe the entire distribution) fails with: /usr/i586-suse-linux/bin/ld: can't relax section: memory exhausted collect2: error: ld returned 1 exit status The machine has lots of memory, but i586 only provides a 2GB user virtual address space for every process. We've already reduced debug info to -g1 earlier, and this change reduces it to -g0. But this only applies to C/C++ code, maybe the same should be done for Rust, which is still at -g1 as far as I can see. [1] We could also try -Os instead of -O2 ("-C opt-level=s" for Rust), but I'm afraid that probably won't shave off a lot. (At least not for GCC, which is quite conscious about size even at -O2, for Rust/LLVM it might help a bit.) If that doesn't do it, I would recommend reaching out to the binutils maintainers for help.
Looking into the devel prj, the picture looks even worse, as aarch64 and armv7l are also failing to build.
Can't see the log because they're both building at the moment, but armv7l is likely the same issue. Aaron [1] <https://build.opensuse.org/package/view_file/mozilla:Factory/MozillaFirefox/mozilla-reduce-rust-debuginfo.patch>
On Sunday 2023-08-06 23:35, Aaron Puchert wrote:
Am 04.08.23 um 10:28 schrieb Dominique Leuenberger:
Firefox 116 has just been sent for inclusion into Tumbleweed - with a note that it fails to build on i586.
For those too lazy to click the link, linking libxul.so (the largest binary in the package and maybe the entire distribution) fails with:
/usr/i586-suse-linux/bin/ld: can't relax section: memory exhausted
If that doesn't do it, I would recommend reaching out to the binutils maintainers for help.
Perhaps -Wl,--no-keep-memory The other approach is to just throw fewer .o and .a files at the linker, and generate more intermediate .so files with self-contained APIs.
Am 07.08.23 um 00:33 schrieb Jan Engelhardt:
Perhaps -Wl,--no-keep-memory
Already active as far as I can see, together with -Wl,--reduce-memory-overheads -Wl,--no-map-whole-files -Wl,--hash-size=31. The last flag was added in <https://build.opensuse.org/package/rdiff/mozilla:Factory/MozillaFirefox?linkrev=base&rev=1083>. Aaron
On 06. 08. 23, 23:35, Aaron Puchert wrote:
Am 04.08.23 um 10:28 schrieb Dominique Leuenberger:
Firefox 116 has just been sent for inclusion into Tumbleweed - with a note that it fails to build on i586.
For those too lazy to click the link, linking libxul.so (the largest binary in the package and maybe the entire distribution) fails with:
/usr/i586-suse-linux/bin/ld: can't relax section: memory exhausted collect2: error: ld returned 1 exit status
The machine has lots of memory, but i586 only provides a 2GB user virtual address space for every process.
Huh, no. AFAIK, we are using 64bit kernels for building even on 32bit. So you have full 4 gigs per process. Is this an LTO build? Maybe that would be tough -- but can't we cross-compile from x86_64? regards, -- js suse labs
On Mon, 7 Aug 2023, Jiri Slaby wrote:
On 06. 08. 23, 23:35, Aaron Puchert wrote:
Am 04.08.23 um 10:28 schrieb Dominique Leuenberger:
Firefox 116 has just been sent for inclusion into Tumbleweed - with a note that it fails to build on i586.
For those too lazy to click the link, linking libxul.so (the largest binary in the package and maybe the entire distribution) fails with:
/usr/i586-suse-linux/bin/ld: can't relax section: memory exhausted collect2: error: ld returned 1 exit status
The machine has lots of memory, but i586 only provides a 2GB user virtual address space for every process.
Huh, no. AFAIK, we are using 64bit kernels for building even on 32bit. So you have full 4 gigs per process.
Is this an LTO build?
Maybe that would be tough -- but can't we cross-compile from x86_64?
In the old days when ppc64 had a 32bit user-land we had gdb64 etc., we could do the same for ld. The gdb baselibs.conf still has gdb +/usr/bin/gdb -> /usr/bin/gdb<extension> # kill package for i586 32bit targetarch x86_64 block! prereq -glibc-x86 not sure how trivial that would be do to for ld. Note it might be possible to use -Wl,--no-relax as well (but not sure if that helps). Richard.
On Monday 2023-08-07 12:33, Richard Biener wrote:
In the old days when ppc64 had a 32bit user-land we had gdb64 etc., we could do the same for ld. The gdb baselibs.conf still has
gdb +/usr/bin/gdb -> /usr/bin/gdb<extension> # kill package for i586 32bit targetarch x86_64 block! prereq -glibc-x86
The gdb baselibs.conf is weird; strace falls into the same category but does not seem to need the targetarch block thing. strace baselibs.conf has more details..
not sure how trivial that would be do to for ld.
rpmrc lists architectures which are considered compatible, because those identifiers, in rpm, are used to denote ELF32-dominant-but-ELF64-capable environments: arch_compat: ppc64: ppc arch_compat: sparc64: sparcv9 However, this is not the case for i586, because compatibility is just one-way. The pairing (if any) would be x32<->x86_64 instead. Unless you want to repurpose the "i586" identifer inside rpm (you should rather use a different one IMO), installation of binutils.x86_64 inside i586 is not going to fly, for reason of that one-way compatibility. Verifying... ################################# [100%] Preparing... ################################# [100%] package acl-2.3.1-2.9.x86_64 is intended for a different architecture
On Mon, 7 Aug 2023, Jan Engelhardt wrote:
On Monday 2023-08-07 12:33, Richard Biener wrote:
In the old days when ppc64 had a 32bit user-land we had gdb64 etc., we could do the same for ld. The gdb baselibs.conf still has
gdb +/usr/bin/gdb -> /usr/bin/gdb<extension> # kill package for i586 32bit targetarch x86_64 block! prereq -glibc-x86
The gdb baselibs.conf is weird; strace falls into the same category but does not seem to need the targetarch block thing. strace baselibs.conf has more details..
not sure how trivial that would be do to for ld.
rpmrc lists architectures which are considered compatible, because those identifiers, in rpm, are used to denote ELF32-dominant-but-ELF64-capable environments:
arch_compat: ppc64: ppc arch_compat: sparc64: sparcv9
However, this is not the case for i586, because compatibility is just one-way. The pairing (if any) would be x32<->x86_64 instead.
Unless you want to repurpose the "i586" identifer inside rpm (you should rather use a different one IMO), installation of binutils.x86_64 inside i586 is not going to fly, for reason of that one-way compatibility.
We could also force that install, it would be really a special hack tailored to our build environment which we know is x86-64 capable ...
Am 07.08.23 um 07:59 schrieb Jiri Slaby:
On 06. 08. 23, 23:35, Aaron Puchert wrote:
The machine has lots of memory, but i586 only provides a 2GB user virtual address space for every process.
Huh, no. AFAIK, we are using 64bit kernels for building even on 32bit. So you have full 4 gigs per process.
Thanks for pointing that out, I was incorrectly assuming that only the lower 2GB would be available. The log says 2nd stage started in virtual machine machine type: x86_64 [...] switching personality to 8... 2nd stage started in virtual machine machine type: i686 Presumably that refers to /usr/include/sys/personality.h: PER_LINUX32 = 0x0008, and there is a separate ADDR_LIMIT_3GB = 0x8000000 /* ... */ PER_LINUX32_3GB = 0x0008 | ADDR_LIMIT_3GB, The man page says PER_LINUX32 (since Linux 2.2) [To be documented.] PER_LINUX32_3GB (since Linux 2.4) Implies ADDR_LIMIT_3GB. ADDR_LIMIT_3GB (since Linux 2.4.0) With this flag set, use 0xc0000000 as the offset at which to search a virtual memory chunk on mmap(2); otherwise use 0xffffe000. So despite our entry not being documented, it seems that 0xffffe000 is the limit, which gives us almost the full 4GB. In any event, since we're talking about virtual address space exhaustion, it could also be a matter of fragmentation. Do we have ASLR active in OBS and might it help to turn it off? I know that we have MALLOC_CHECK_, which is quite slow in multi-threaded processes, but I'm not sure about the memory overhead. Aaron
Hi Aaron! On Sun, 2023-08-06 at 23:35 +0200, Aaron Puchert wrote:
Am 04.08.23 um 10:28 schrieb Dominique Leuenberger:
Firefox 116 has just been sent for inclusion into Tumbleweed - with a note that it fails to build on i586.
For those too lazy to click the link, linking libxul.so (the largest binary in the package and maybe the entire distribution) fails with:
/usr/i586-suse-linux/bin/ld: can't relax section: memory exhausted collect2: error: ld returned 1 exit status
The machine has lots of memory, but i586 only provides a 2GB user virtual address space for every process.
We've already reduced debug info to -g1 earlier, and this change reduces it to -g0. But this only applies to C/C++ code, maybe the same should be done for Rust, which is still at -g1 as far as I can see. [1]
Debian has a patch that overrides the Rust LTO settings, see: https://salsa.debian.org/mozilla-team/firefox/-/blob/release/master/debian/p... FWIW, Firefox builds on Debian on armv7, arm64, i386, so I assume Mike Homney who is also Firefox upstream developer already found solutions for these issues: https://buildd.debian.org/status/package.php?p=firefox Adrian
Sorry if this is irrelevant, but I found a few possibly related links while trying to read up for my own learning, and wanted to share in case anything is helpful / triggers thoughts in someone more knowledgeable? https://bugzilla.mozilla.org/show_bug.cgi?id=709193 Work from years ago that the Mozilla team was doing on address space restrictions for compilation https://www.linuxquestions.org/questions/linux-from-scratch-13/firefox-error... Someone compiling on a personal device who seemed to find that disk space and swap space (in addition to memory) were limiting factors https://gcc.gnu.org/bugzilla/show_bug.cgi?id=45375#c1 GCC team - reference to limiting mmap of .o files? Thanks, John Kizer
On 8/4/23 10:28, Dominique Leuenberger wrote:
Looking into the devel prj, the picture looks even worse, as aarch64 and armv7l are also failing to build.
https://build.opensuse.org/package/show/mozilla:Factory/MozillaFirefox
With a little bit of playing around, the issue is that the linker runs out of memory. Removing -Wl,-z,pack-relative-relocs keeps the linker at about 1GB peak usage while with this option in-place, memory usage blows up past 4GB - Adam
On Wed, 9 Aug 2023, Adam Majer wrote:
On 8/4/23 10:28, Dominique Leuenberger wrote:
Looking into the devel prj, the picture looks even worse, as aarch64 and armv7l are also failing to build.
https://build.opensuse.org/package/show/mozilla:Factory/MozillaFirefox
With a little bit of playing around, the issue is that the linker runs out of memory. Removing
-Wl,-z,pack-relative-relocs
keeps the linker at about 1GB peak usage while with this option in-place, memory usage blows up past 4GB
Might be worth an upstream bugreport. Richard. -- Richard Biener <rguenther@suse.de> SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg, Germany; GF: Ivo Totev, Andrew McDonald, Werner Knoblich; (HRB 36809, AG Nuernberg)
participants (9)
-
Aaron Puchert -
Adam Majer -
Adrian Glaubitz -
Dominique Leuenberger -
Jan Engelhardt -
Jiri Slaby -
John Kizer -
Richard Biener -
Wolfgang Rosenauer