On Donnerstag, 7. September 2017, 13:23:29 CEST wrote Michal Srb:

Hi,

I am working with llvm4 package, which is usually building very long time: https://build.opensuse.org/package/show/devel:tools:compiler/llvm4 https://build.opensuse.org/package/show/ home:michalsrb:branches:devel:tools:compiler/llvm4

I am seeing random but frequent build failures where the build log simply ends (sometimes even in the middle of a line) and after long wait the build is terminated. For example:

...

... [14840s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/ llvm4-4.0.1-38.5.i386/usr/include/llvm/Transforms/Utils/CtorUtils.h [14840s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/ llvm4-4.0.1-38.5.i386/usr/include/llvm/Transforms/Utils/EscapeEnumerator.h [14840s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/ llvm4-4.0.1-38.5.i386/usr/include/llvm/Transf

Job seems to be stuck here, killed. (after 28800 seconds of inactivity)

It never happened in local build. In build service it happens randomly and in different parts of the build. For example in the middle of "make install" in %install section, or in the middle of running tests in %check section, or in the middle of debuginfo extraction. That makes me think that the problem is not in the package, but something is wrong in build service.

but not during %build? Would point to some IO or disk space problem maybe. Can you detect any pattern in the jobhistory? Eg. It only fails on lamb7x systems or alike? You may need require more disk space then ...

Could there be something broken in the package? If it is problem in buildservice, can I do something to reduce the chance of it happening?

Hard to say, you could try local build using "--vm-type=kvm" to build like on our workers. Can you reproduce it then? It could be also a kernel bug from the used distro. Does it happen only on distro X maybe? In worst case you need to ping me when you see the build is hanging on some worker and I need to trigger a kernel trace to get an idea why it is hanging .... -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On čtvrtek 7. září 2017 13:37:16 CEST Adrian Schröter wrote:

...

It never happened in local build. In build service it happens randomly and in different parts of the build. For example in the middle of "make install" in %install section, or in the middle of running tests in %check section, or in the middle of debuginfo extraction. That makes me think that the problem is not in the package, but something is wrong in build service.

but not during %build?

Sometimes it happens during %build too.

Would point to some IO or disk space problem maybe.

The package has _constraints file that asks for 30GB of disk space, which should be enough. But yes, maybe some IO problem.

Can you detect any pattern in the jobhistory? Eg. It only fails on lamb7x systems or alike?

I have only 7 samples right now, I don't know how to get to logs from older builds, if it is possible. In those 7 samples it failed on SLE_12_SP2, openSUSE_Factory, openSUSE_Leap_42.2 and openSUSE_Leap_42.3. Architectures were x86_64, i586, armv6l. Build hosts were lamb77, lamb76, lamb74, lamb78 and armbuild15.

Hard to say, you could try local build using "--vm-type=kvm" to build like on our workers. Can you reproduce it then?

I tried that and it built correctly. But I tried only once or twice, so I can't tell if I wasn't just lucky.

In worst case you need to ping me when you see the build is hanging on some worker and I need to trigger a kernel trace to get an idea why it is hanging ....

I'll ping you if I catch it happening. Thanks! Michal Srb -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org