[opensuse-factory] Firefox builds on OBS fail
Hi, since Firefox 44 I have severe issues building it in OBS. Currently I'm trying to prepare 44.0.2 and it basically fails everywhere for reasons not totally clear to me. There are different build failures not always reproducible. If it's not an out of memory as below it is a very strange other error you find at the bottom: Examples: https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/i586/Mozil... [ 1130s] [ 1115.084681] Out of memory: Kill process 23582 (cc1plus) score 146 or sacrifice child [ 1130s] [ 1115.086356] Killed process 23582 (cc1plus) total-vm:787324kB, anon-rss:205252kB, file-rss:0kB [ 1134s] {standard input}: Assembler messages: [ 1134s] {standard input}:493902: Warning: end of file not at end of a line; newline inserted [ 1134s] {standard input}:493962: Error: unbalanced parenthesis in operand 1. [ 1169s] {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive [ 1171s] c++: internal compiler error: Killed (program cc1plus) [ 1171s] Please submit a full bug report, [ 1171s] with preprocessed source if appropriate. https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/x86_64/Moz... [ 3179s] ERROR: link target doesn't exist (neither in build root nor in installed system): [ 3179s] /usr/lib/debug/.build-id/32/872edd419c96970ed2daa4c0a4f6a04f94c0dc.debug -> /usr/lib/debug/usr/lib64/firefox/libxul.so.debug [ 3179s] Add the package providing the target to BuildRequires and Requires [ 3179s] INFO: relinking /usr/bin/firefox -> ../lib64/firefox/firefox.sh (was ../../usr/lib64/firefox/firefox.sh) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox-gnome.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] error: Bad exit status from /var/tmp/rpm-tmp.1i9Jyt (%install) [ 3179s] Retriggering builds fixes them eventually. But it's like really working 1 time out of 6 or something like this which is quite annoying. Somehow it seems related to memory usage. I could try to disable debuginfo but not sure if that is appropriate. Any other ideas? Wolfgang -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 02/12/2016 06:32 PM, Wolfgang Rosenauer wrote:
Hi, [...]
Examples: https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/i586/Mozil...
[ 1130s] [ 1115.084681] Out of memory: Kill process 23582 (cc1plus)
It seems the build requires more memory that's currently available on the build host so OOM is triggered. -- markos -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 12/02/2016 20:32, Wolfgang Rosenauer wrote:
Hi,
since Firefox 44 I have severe issues building it in OBS. Currently I'm trying to prepare 44.0.2 and it basically fails everywhere for reasons not totally clear to me. There are different build failures not always reproducible. If it's not an out of memory as below it is a very strange other error you find at the bottom:
Examples: https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/i586/Mozil...
[ 1130s] [ 1115.084681] Out of memory: Kill process 23582 (cc1plus) score 146 or sacrifice child [ 1130s] [ 1115.086356] Killed process 23582 (cc1plus) total-vm:787324kB, anon-rss:205252kB, file-rss:0kB [ 1134s] {standard input}: Assembler messages: [ 1134s] {standard input}:493902: Warning: end of file not at end of a line; newline inserted [ 1134s] {standard input}:493962: Error: unbalanced parenthesis in operand 1. [ 1169s] {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive [ 1171s] c++: internal compiler error: Killed (program cc1plus) [ 1171s] Please submit a full bug report, [ 1171s] with preprocessed source if appropriate.
https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/x86_64/Moz...
[ 3179s] ERROR: link target doesn't exist (neither in build root nor in installed system): [ 3179s] /usr/lib/debug/.build-id/32/872edd419c96970ed2daa4c0a4f6a04f94c0dc.debug -> /usr/lib/debug/usr/lib64/firefox/libxul.so.debug [ 3179s] Add the package providing the target to BuildRequires and Requires [ 3179s] INFO: relinking /usr/bin/firefox -> ../lib64/firefox/firefox.sh (was ../../usr/lib64/firefox/firefox.sh) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox-gnome.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] error: Bad exit status from /var/tmp/rpm-tmp.1i9Jyt (%install) [ 3179s]
Retriggering builds fixes them eventually. But it's like really working 1 time out of 6 or something like this which is quite annoying.
Somehow it seems related to memory usage. I could try to disable debuginfo but not sure if that is appropriate. Any other ideas?
Wolfgang You need to add a _constraints file to get more memory for the build, I was having similar problems with blender a while ago. Regards Dave P -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 13.02.2016 um 16:43 schrieb Dave Plater:
On 12/02/2016 20:32, Wolfgang Rosenauer wrote:
Hi,
since Firefox 44 I have severe issues building it in OBS. Currently I'm trying to prepare 44.0.2 and it basically fails everywhere for reasons not totally clear to me. There are different build failures not always reproducible. If it's not an out of memory as below it is a very strange other error you find at the bottom:
Examples: https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/i586/Mozil...
[ 1130s] [ 1115.084681] Out of memory: Kill process 23582 (cc1plus) score 146 or sacrifice child [ 1130s] [ 1115.086356] Killed process 23582 (cc1plus) total-vm:787324kB, anon-rss:205252kB, file-rss:0kB [ 1134s] {standard input}: Assembler messages: [ 1134s] {standard input}:493902: Warning: end of file not at end of a line; newline inserted [ 1134s] {standard input}:493962: Error: unbalanced parenthesis in operand 1. [ 1169s] {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive [ 1171s] c++: internal compiler error: Killed (program cc1plus) [ 1171s] Please submit a full bug report, [ 1171s] with preprocessed source if appropriate.
https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/x86_64/Moz...
[ 3179s] ERROR: link target doesn't exist (neither in build root nor in installed system): [ 3179s] /usr/lib/debug/.build-id/32/872edd419c96970ed2daa4c0a4f6a04f94c0dc.debug -> /usr/lib/debug/usr/lib64/firefox/libxul.so.debug [ 3179s] Add the package providing the target to BuildRequires and Requires [ 3179s] INFO: relinking /usr/bin/firefox -> ../lib64/firefox/firefox.sh (was ../../usr/lib64/firefox/firefox.sh) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox-gnome.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] error: Bad exit status from /var/tmp/rpm-tmp.1i9Jyt (%install) [ 3179s]
Retriggering builds fixes them eventually. But it's like really working 1 time out of 6 or something like this which is quite annoying.
Somehow it seems related to memory usage. I could try to disable debuginfo but not sure if that is appropriate. Any other ideas?
Wolfgang You need to add a _constraints file to get more memory for the build, I was having similar problems with blender a while ago.
There is already one: I even increased the size from 5000 to 6000 when it started to fail: <memory> <size unit="M">6000</size> </memory> Somehow I think 6GB should be enough, shouldn't it? Wolfgang -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/02/2016 18:21, Wolfgang Rosenauer wrote:
Am 13.02.2016 um 16:43 schrieb Dave Plater:
On 12/02/2016 20:32, Wolfgang Rosenauer wrote:
Hi,
since Firefox 44 I have severe issues building it in OBS. Currently I'm trying to prepare 44.0.2 and it basically fails everywhere for reasons not totally clear to me. There are different build failures not always reproducible. If it's not an out of memory as below it is a very strange other error you find at the bottom:
Examples: https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/i586/Mozil...
[ 1130s] [ 1115.084681] Out of memory: Kill process 23582 (cc1plus) score 146 or sacrifice child [ 1130s] [ 1115.086356] Killed process 23582 (cc1plus) total-vm:787324kB, anon-rss:205252kB, file-rss:0kB [ 1134s] {standard input}: Assembler messages: [ 1134s] {standard input}:493902: Warning: end of file not at end of a line; newline inserted [ 1134s] {standard input}:493962: Error: unbalanced parenthesis in operand 1. [ 1169s] {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive [ 1171s] c++: internal compiler error: Killed (program cc1plus) [ 1171s] Please submit a full bug report, [ 1171s] with preprocessed source if appropriate.
https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/x86_64/Moz...
[ 3179s] ERROR: link target doesn't exist (neither in build root nor in installed system): [ 3179s] /usr/lib/debug/.build-id/32/872edd419c96970ed2daa4c0a4f6a04f94c0dc.debug -> /usr/lib/debug/usr/lib64/firefox/libxul.so.debug [ 3179s] Add the package providing the target to BuildRequires and Requires [ 3179s] INFO: relinking /usr/bin/firefox -> ../lib64/firefox/firefox.sh (was ../../usr/lib64/firefox/firefox.sh) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox-gnome.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] error: Bad exit status from /var/tmp/rpm-tmp.1i9Jyt (%install) [ 3179s]
Retriggering builds fixes them eventually. But it's like really working 1 time out of 6 or something like this which is quite annoying.
Somehow it seems related to memory usage. I could try to disable debuginfo but not sure if that is appropriate. Any other ideas?
Wolfgang You need to add a _constraints file to get more memory for the build, I was having similar problems with blender a while ago.
There is already one:
I even increased the size from 5000 to 6000 when it started to fail:
<memory> <size unit="M">6000</size> </memory>
Somehow I think 6GB should be enough, shouldn't it?
Wolfgang
I had the most trouble with packman factory, you've also got disk size and vitrual memory to play with: <hardware> <disk> <size unit="G">18</size> </disk> <physicalmemory> <size unit="M">1024</size> </physicalmemory> <memory> <size unit="M">1536</size> </memory> </hardware> Try increasing the disk size, that solved the problem for me. I also prebuilt one particularly resource hungry lib first with -j1 Here's another trick I used to limit the builds to 2 : %define jobs %(echo $jobs) %if 0%{?jobs} > 2 %define _smp_mflags -j2 %endif make %{?_smp_mflags} Dave P -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 13/02/2016 18:21, Wolfgang Rosenauer wrote:
Am 13.02.2016 um 16:43 schrieb Dave Plater:
On 12/02/2016 20:32, Wolfgang Rosenauer wrote:
Hi,
since Firefox 44 I have severe issues building it in OBS. Currently I'm trying to prepare 44.0.2 and it basically fails everywhere for reasons not totally clear to me. There are different build failures not always reproducible. If it's not an out of memory as below it is a very strange other error you find at the bottom:
Examples: https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/i586/Mozil...
[ 1130s] [ 1115.084681] Out of memory: Kill process 23582 (cc1plus) score 146 or sacrifice child [ 1130s] [ 1115.086356] Killed process 23582 (cc1plus) total-vm:787324kB, anon-rss:205252kB, file-rss:0kB [ 1134s] {standard input}: Assembler messages: [ 1134s] {standard input}:493902: Warning: end of file not at end of a line; newline inserted [ 1134s] {standard input}:493962: Error: unbalanced parenthesis in operand 1. [ 1169s] {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive [ 1171s] c++: internal compiler error: Killed (program cc1plus) [ 1171s] Please submit a full bug report, [ 1171s] with preprocessed source if appropriate.
https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/x86_64/Moz...
[ 3179s] ERROR: link target doesn't exist (neither in build root nor in installed system): [ 3179s] /usr/lib/debug/.build-id/32/872edd419c96970ed2daa4c0a4f6a04f94c0dc.debug -> /usr/lib/debug/usr/lib64/firefox/libxul.so.debug [ 3179s] Add the package providing the target to BuildRequires and Requires [ 3179s] INFO: relinking /usr/bin/firefox -> ../lib64/firefox/firefox.sh (was ../../usr/lib64/firefox/firefox.sh) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox-gnome.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] error: Bad exit status from /var/tmp/rpm-tmp.1i9Jyt (%install) [ 3179s]
Retriggering builds fixes them eventually. But it's like really working 1 time out of 6 or something like this which is quite annoying.
Somehow it seems related to memory usage. I could try to disable debuginfo but not sure if that is appropriate. Any other ideas?
Wolfgang You need to add a _constraints file to get more memory for the build, I was having similar problems with blender a while ago.
There is already one:
I even increased the size from 5000 to 6000 when it started to fail:
<memory> <size unit="M">6000</size> </memory>
Somehow I think 6GB should be enough, shouldn't it?
Wolfgang
sorry I missed a part: # A further attempt to stop out of memory failures. export jobs=$(getconf _NPROCESSORS_ONLN) %define jobs %(echo $jobs) %if 0%{?jobs} > 2 %define _smp_mflags -j2 %endif make %{?_smp_mflags} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Saturday 2016-02-13 18:27, Dave Plater wrote:
sorry I missed a part: # A further attempt to stop out of memory failures. export jobs=$(getconf _NPROCESSORS_ONLN)
%define jobs %(echo $jobs) %if 0%{?jobs} > 2 %define _smp_mflags -j2 %endif make %{?_smp_mflags}
Better use %if 0%{?jobs} > 2 %define jobs 2 %endif make %{?_smp_mflags} -j%jobs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Better use
%if 0%{?jobs} > 2 %define jobs 2 %endif make %{?_smp_mflags} -j%jobs
no isn't better ... is evaluated as "make -j(ncpus) -j2" terrible idea
from rpm 4.12.0 ( for example openSUSE:Tumbleweed) is possible use
"%_smp_ncpus_max" to limit max value of %_smp_mflags -->
%_smp_ncpus_max 3
%make %{?_smp_mflags}
is evaluated as "make -j3" or lower
On 13 February 2016 at 18:44, Jan Engelhardt
On Saturday 2016-02-13 18:27, Dave Plater wrote:
sorry I missed a part: # A further attempt to stop out of memory failures. export jobs=$(getconf _NPROCESSORS_ONLN)
%define jobs %(echo $jobs) %if 0%{?jobs} > 2 %define _smp_mflags -j2 %endif make %{?_smp_mflags}
Better use
%if 0%{?jobs} > 2 %define jobs 2 %endif make %{?_smp_mflags} -j%jobs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Somehow I think 6GB should be enough, shouldn't it?
It depends on what value of -j (number of simultaneous jobs) you are
passing to "make" in the %build section.
I suppose you are passing a macro that sets -j to the number of
processor cores available. OBS build workers are strange in that they
have many cores but little memory - typically just 4GB.
I would suggest that you examine the failing build logs to find out
the value of -j that is being used and compare it to the ones where
the build succeeds.
It is highly likely that if you try to build locally in a VM with the
same number of -j and the same amount of memory, it will fail in a
similar way.
HTH,
Nathan
On Sat, Feb 13, 2016 at 5:21 PM, Wolfgang Rosenauer
Am 13.02.2016 um 16:43 schrieb Dave Plater:
On 12/02/2016 20:32, Wolfgang Rosenauer wrote:
Hi,
since Firefox 44 I have severe issues building it in OBS. Currently I'm trying to prepare 44.0.2 and it basically fails everywhere for reasons not totally clear to me. There are different build failures not always reproducible. If it's not an out of memory as below it is a very strange other error you find at the bottom:
Examples: https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/i586/Mozil...
[ 1130s] [ 1115.084681] Out of memory: Kill process 23582 (cc1plus) score 146 or sacrifice child [ 1130s] [ 1115.086356] Killed process 23582 (cc1plus) total-vm:787324kB, anon-rss:205252kB, file-rss:0kB [ 1134s] {standard input}: Assembler messages: [ 1134s] {standard input}:493902: Warning: end of file not at end of a line; newline inserted [ 1134s] {standard input}:493962: Error: unbalanced parenthesis in operand 1. [ 1169s] {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive [ 1171s] c++: internal compiler error: Killed (program cc1plus) [ 1171s] Please submit a full bug report, [ 1171s] with preprocessed source if appropriate.
https://build.opensuse.org/build/mozilla:Factory/openSUSE_Factory/x86_64/Moz...
[ 3179s] ERROR: link target doesn't exist (neither in build root nor in installed system): [ 3179s] /usr/lib/debug/.build-id/32/872edd419c96970ed2daa4c0a4f6a04f94c0dc.debug -> /usr/lib/debug/usr/lib64/firefox/libxul.so.debug [ 3179s] Add the package providing the target to BuildRequires and Requires [ 3179s] INFO: relinking /usr/bin/firefox -> ../lib64/firefox/firefox.sh (was ../../usr/lib64/firefox/firefox.sh) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] INFO: relinking /usr/share/pixmaps/firefox-gnome.png -> ../../lib64/firefox/browser/icons/mozicon128.png (was /usr/lib64/firefox/browser/icons/mozicon128.png) [ 3179s] error: Bad exit status from /var/tmp/rpm-tmp.1i9Jyt (%install) [ 3179s]
Retriggering builds fixes them eventually. But it's like really working 1 time out of 6 or something like this which is quite annoying.
Somehow it seems related to memory usage. I could try to disable debuginfo but not sure if that is appropriate. Any other ideas?
Wolfgang You need to add a _constraints file to get more memory for the build, I was having similar problems with blender a while ago.
There is already one:
I even increased the size from 5000 to 6000 when it started to fail:
<memory> <size unit="M">6000</size> </memory>
Somehow I think 6GB should be enough, shouldn't it?
Wolfgang
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 14/02/2016 01:07, Nathan Cutler wrote:
I would suggest that you examine the failing build logs to find out the value of -j that is being used and compare it to the ones where the build succeeds.
It is highly likely that if you try to build locally in a VM with the same number of -j and the same amount of memory, it will fail in a similar way.
HTH, Nathan I found that the obs factory builds that were failing with blender were -j4, that's why I limited it to 2. The increase in disk size in _constraints fixed the Packman Factory build, which is limited to -j1 anyway. Dave P -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (6)
-
Dave Plater -
Jan Engelhardt -
Markos Chandras -
Nathan Cutler -
Ondřej Súkup -
Wolfgang Rosenauer