[opensuse-buildservice] Firefox build fails on i586 Tumbleweed
Just noticed that MozillaFirefox is not in the i586 openSUSE_Tumbleweed repo anymore: https://download.opensuse.org/tumbleweed/repo/oss/suse/i586/ Digging a little deeper it seems that the build is failing for quite some time due to out of memory: https://build.opensuse.org/build/openSUSE:Factory/standard/i586/MozillaFiref... Could somebody look into that? Tom -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Sonntag, 12. November 2017, 17:16:56 CET wrote Tom M.:
Just noticed that MozillaFirefox is not in the i586 openSUSE_Tumbleweed repo anymore: https://download.opensuse.org/tumbleweed/repo/oss/suse/i586/
Digging a little deeper it seems that the build is failing for quite some time due to out of memory: https://build.opensuse.org/build/openSUSE:Factory/standard/i586/MozillaFiref...
Could somebody look into that?
Out of memory error ... so it seems the package needs to increase it's need for memory in the _constraints file. However, you need to report this to the maintainer or to the factory people in first place. Or even create a submit request for that ;) You can find the right development project and maintainer in the owner search: https://build.opensuse.org/search/owner?utf8=%E2%9C%93&search_text=MozillaFirefox&owner=1&package=0&project=0&commit=Senden&limit=1 -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi, Am 13.11.2017 um 08:51 schrieb Adrian Schröter:
On Sonntag, 12. November 2017, 17:16:56 CET wrote Tom M.:
Just noticed that MozillaFirefox is not in the i586 openSUSE_Tumbleweed repo anymore: https://download.opensuse.org/tumbleweed/repo/oss/suse/i586/
Digging a little deeper it seems that the build is failing for quite some time due to out of memory: https://build.opensuse.org/build/openSUSE:Factory/standard/i586/MozillaFiref...
Could somebody look into that?
Out of memory error ... so it seems the package needs to increase it's need for memory in the _constraints file.
Firefox already has 6GB defined for memory. Typically in my experience changing that higher does not even fix the issue. We already have a patch in Firefox which tries to reduce linking executions to use less memory which still seems not to be small enough for 32bit systems. In addition the upcoming Firefox 57 will not build on x86-32 for other reasons. But I will certainly look into it. It'll just not be a quick thing. Wolfgang -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
It seems to build with 8 jobs in parallel. How about hardcoding MOZ_MAKE_FLAGS=-j2 or similar? Tom 2017-11-13 9:12 GMT+01:00 Wolfgang Rosenauer <wolfgang@rosenauer.org>:
Hi,
Am 13.11.2017 um 08:51 schrieb Adrian Schröter:
On Sonntag, 12. November 2017, 17:16:56 CET wrote Tom M.:
Just noticed that MozillaFirefox is not in the i586 openSUSE_Tumbleweed repo anymore: https://download.opensuse.org/tumbleweed/repo/oss/suse/i586/
Digging a little deeper it seems that the build is failing for quite some time due to out of memory: https://build.opensuse.org/build/openSUSE:Factory/standard/i586/MozillaFiref...
Could somebody look into that?
Out of memory error ... so it seems the package needs to increase it's need for memory in the _constraints file.
Firefox already has 6GB defined for memory. Typically in my experience changing that higher does not even fix the issue. We already have a patch in Firefox which tries to reduce linking executions to use less memory which still seems not to be small enough for 32bit systems.
In addition the upcoming Firefox 57 will not build on x86-32 for other reasons. But I will certainly look into it. It'll just not be a quick thing.
Wolfgang -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Montag, 13. November 2017 09:12:17 CET Wolfgang Rosenauer wrote:
Hi,
Am 13.11.2017 um 08:51 schrieb Adrian Schröter:
On Sonntag, 12. November 2017, 17:16:56 CET wrote Tom M.:
Just noticed that MozillaFirefox is not in the i586 openSUSE_Tumbleweed repo anymore: https://download.opensuse.org/tumbleweed/repo/oss/suse/i586/
Digging a little deeper it seems that the build is failing for quite some time due to out of memory: https://build.opensuse.org/build/openSUSE:Factory/standard/i586/MozillaFi refox/_log
Could somebody look into that?
Out of memory error ... so it seems the package needs to increase it's need for memory in the _constraints file.
Firefox already has 6GB defined for memory. Typically in my experience changing that higher does not even fix the issue. We already have a patch in Firefox which tries to reduce linking executions to use less memory which still seems not to be small enough for 32bit systems.
All 32bit workers are only able to use 3GB of "physical" memory, plus the typical 1-2GByte of swap. Currently, kernel-obs-build packages the "default" flavour, which does not enable pae/lpae (ix86/armv7). Non-PAE kernels are only able to use 3GByte, no matter how much memory you pass to the VM via "-m". To verify, dump the contents of /proc/meminfo (MemTotal) or grep for "Memory" in the kernel log/dmesg. Kind regards, Stefan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Monday 2017-11-13 16:39, Brüns, Stefan wrote:
All 32bit workers are only able to use 3GB of "physical" memory, plus the typical 1-2GByte of swap.
That is wrong - some 32-bit workers can use 4G, notably those with a 64-bit kernel and those which have the special 4:4 split feature enabled in their i386 kernel. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Montag, 13. November 2017 16:48:05 CET Jan Engelhardt wrote:
On Monday 2017-11-13 16:39, Brüns, Stefan wrote:
All 32bit workers are only able to use 3GB of "physical" memory, plus the typical 1-2GByte of swap.
That is wrong - some 32-bit workers can use 4G, notably those with a 64-bit kernel and those which have the special 4:4 split feature enabled in their i386 kernel.
See https://build.opensuse.org/package/live_build_log/ home:StefanBruens:CheckMemory/check-memory/openSUSE_Tumbleweed/i586 _constraints set to 6G, VM is started with -m 8192, available memory (Mem + Swap) in the build environment: 3G + 2G -> _constraints is *not* honored Regards, Stefan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Montag, 13. November 2017, 17:57:21 CET wrote Brüns, Stefan:
On Montag, 13. November 2017 16:48:05 CET Jan Engelhardt wrote:
On Monday 2017-11-13 16:39, Brüns, Stefan wrote:
All 32bit workers are only able to use 3GB of "physical" memory, plus the typical 1-2GByte of swap.
That is wrong - some 32-bit workers can use 4G, notably those with a 64-bit kernel and those which have the special 4:4 split feature enabled in their i386 kernel.
See https://build.opensuse.org/package/live_build_log/ home:StefanBruens:CheckMemory/check-memory/openSUSE_Tumbleweed/i586
_constraints set to 6G, VM is started with -m 8192, available memory (Mem + Swap) in the build environment: 3G + 2G
-> _constraints is *not* honored
Huh? It is, the VM is providing 8GB what is more than 6GB. It is not up to the constraints system what the content (here the kernel) is doing with that... -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Dienstag, 14. November 2017 07:33:22 CET you wrote:
On Montag, 13. November 2017, 17:57:21 CET wrote Brüns, Stefan:
On Montag, 13. November 2017 16:48:05 CET Jan Engelhardt wrote:
On Monday 2017-11-13 16:39, Brüns, Stefan wrote:
All 32bit workers are only able to use 3GB of "physical" memory, plus the typical 1-2GByte of swap.
That is wrong - some 32-bit workers can use 4G, notably those with a 64-bit kernel and those which have the special 4:4 split feature enabled in their i386 kernel.
See https://build.opensuse.org/package/live_build_log/ home:StefanBruens:CheckMemory/check-memory/openSUSE_Tumbleweed/i586
_constraints set to 6G, VM is started with -m 8192, available memory (Mem + Swap) in the build environment: 3G + 2G
-> _constraints is *not* honored
Huh? It is, the VM is providing 8GB what is more than 6GB.
It is not up to the constraints system what the content (here the kernel) is doing with that...
From the user perspective this is clearly unexpected behaviour - the user has requested a worker with 8 GB of memory, but only gets 4...5 GByte (RAM + Swap, as documented).
A worker which is started with more than 3 GByte (-m >= 3000) should be using a PAE/HIGHMEM64G kernel. Currently, the constraints have the opposite effect to the users expectations - a job is more likely to fail on a 8GB/-smp 8 system than on a 4GB/-smp 4 system. Kind regards, Stefan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tuesday 2017-11-14 12:04, Brüns, Stefan wrote:
_constraints set to 6G, VM is started with -m 8192, available memory (Mem + Swap) in the build environment: 3G + 2G
-> _constraints is *not* honored
Huh? It is, the VM is providing 8GB what is more than 6GB.
It is not up to the constraints system what the content (here the kernel) is doing with that...
From the user perspective this is clearly unexpected behaviour - the user has requested a worker with 8 GB of memory, but only gets 4...5 GByte (RAM + Swap, as documented).
A worker which is started with more than 3 GByte (-m >= 3000) should be using a PAE/HIGHMEM64G kernel.
I can confirm; kvm is started with 64-bit "hardware", but uses the non-PAE 32-bit kernel (kernel-default). It either has to use kernel-pae, or has to use the 64-bit kernel-default. init=/.build/build will switch to the ILP32 personality anyway. I have a feeling though, that such won't be enough to get those unspeakably fat browsers to link - it may become necessary to use a 64-bit toolchain with -m32 to complete it, and then the baselibs mechanism to export the build results. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Nov 14 2017, Jan Engelhardt <jengelh@inai.de> wrote:
I have a feeling though, that such won't be enough to get those unspeakably fat browsers to link - it may become necessary to use a 64-bit toolchain with -m32 to complete it, and then the baselibs mechanism to export the build results.
Getting all build dependencies ready for -m32 would be a challenge, though. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tuesday 2017-11-14 16:47, Andreas Schwab wrote:
On Nov 14 2017, Jan Engelhardt <jengelh@inai.de> wrote:
I have a feeling though, that such won't be enough to get those unspeakably fat browsers to link - it may become necessary to use a 64-bit toolchain with -m32 to complete it, and then the baselibs mechanism to export the build results.
Getting all build dependencies ready for -m32 would be a challenge, though.
It could yes. The alternative would be to run a 32-bit userspace with a 64-bit linker program, requiring only a few -64bit baselibs. If we ever do x32, modifying mkbaselibs will be a necessity anyway. Might as well do it now :-) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Dienstag, 14. November 2017 17:08:18 CET Jan Engelhardt wrote:
On Tuesday 2017-11-14 16:47, Andreas Schwab wrote:
I have a feeling though, that such won't be enough to get those unspeakably fat browsers to link - it may become necessary to use a 64-bit toolchain with -m32 to complete it, and then the baselibs mechanism to export the build results.> Getting all build dependencies ready for -m32 would be a challenge,
On Nov 14 2017, Jan Engelhardt <jengelh@inai.de> wrote: though.
It could yes. The alternative would be to run a 32-bit userspace with a 64-bit linker program, requiring only a few -64bit baselibs. If we ever do x32, modifying mkbaselibs will be a necessity anyway. Might as well do it now :-)
For the time being, can we just package kernel-(l)pae as kernel-obs-build on 32 bit archs, i.e. i586/armv7l? A PAE-enabled kernel has a slight overhead, but this is likely more than compensated by hitting the guests swap later. This does not change the 3GByte per process limit, which may cause failing linker runs. Using a 64bit linker is much more effort, so I would vote for a PAE kernel first and *if* we still see OOM errors, reconsider if we should go the 64bit userspace route (i586 is quite dead, but armv7l may stay for a bit longer). CC'ed opensuse-kernel, as I already sent a patch addressing kernel-obs-build a week ago. Kind regards, Stefan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Dienstag, 14. November 2017 16:36:12 CET you wrote:
On Tuesday 2017-11-14 12:04, Brüns, Stefan wrote:
_constraints set to 6G, VM is started with -m 8192, available memory (Mem + Swap) in the build environment: 3G + 2G
-> _constraints is *not* honored
Huh? It is, the VM is providing 8GB what is more than 6GB.
It is not up to the constraints system what the content (here the kernel) is doing with that...
From the user perspective this is clearly unexpected behaviour - the user has requested a worker with 8 GB of memory, but only gets 4...5 GByte (RAM + Swap, as documented).
A worker which is started with more than 3 GByte (-m >= 3000) should be using a PAE/HIGHMEM64G kernel.
I can confirm; kvm is started with 64-bit "hardware", but uses the non-PAE 32-bit kernel (kernel-default).
It either has to use kernel-pae, or has to use the 64-bit kernel-default. init=/.build/build will switch to the ILP32 personality anyway.
I have a feeling though, that such won't be enough to get those unspeakably fat browsers to link - it may become necessary to use a 64-bit toolchain with -m32 to complete it, and then the baselibs mechanism to export the build results.
You may be right and it will fail later anyway, but currently, the failing processes are at about 1.5 GByte each (from the log in the first message): [ 2881s] virtual memory exhausted: Cannot allocate memory [ 2881s] virtual memory exhausted: Cannot allocate memory [ 2881s] [ 2881s] cc1plus: out of memory allocating 335128 bytes after a total of 13721600 bytes [ 2881s] [ 2881s] /usr/lib/gcc/i586-suse-linux/7/../../../../i586-suse-linux/bin/as: out of memory allocating 4064 bytes after a total of 16891904 bytes [ 2881s] {standard input}: Assembler messages: [ 2881s] {standard input}:291371: Fatal error: can't close Unified_cpp_layout_generic3.o: Memory exhausted [ 2881s] [ 2881s] /usr/lib/gcc/i586-suse-linux/7/../../../../i586-suse-linux/bin/as: out of memory allocating 4064 bytes after a total of 13647872 bytes Kind regards, Stefan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (6)
-
Adrian Schröter -
Andreas Schwab -
Brüns, Stefan -
Jan Engelhardt -
Tom M. -
Wolfgang Rosenauer