[opensuse-arm] hanging armv7l builds?
HI! armv7l builds on OBS seem to be scheduled but never get executed. During the last days OBS was quite busy but currently the number of active build workers is below 200. For example: https://build.opensuse.org/package/show/devel:languages:python/python-ldap Ciao, Michael.
Michael Ströder <michael@stroeder.com> writes:
armv7l builds on OBS seem to be scheduled but never get executed. During the last days OBS was quite busy but currently the number of active build workers is below 200.
There have never been that many armv7 workers. Note that nowadays armv7 jobs are also executed by aarch64 workers. According to the monitor the armv7 and aarch64 workers have all been busy during the whole week.
For example: https://build.opensuse.org/package/show/devel:languages:python/python-ldap
Looks good right now. $ osc jobhist devel:languages:python/python-ldap openSUSE_Factory_ARM armv7l time package reason code build time worker 2016-07-16 04:16:31 python-ldap new build succeeded 1m 24s armbuild03:8 2016-07-24 16:48:56 python-ldap new build succeeded 2m 58s armbuild17:2 Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Andreas Schwab wrote:
Michael Ströder <michael@stroeder.com> writes:
armv7l builds on OBS seem to be scheduled but never get executed. During the last days OBS was quite busy but currently the number of active build workers is below 200.
There have never been that many armv7 workers. Note that nowadays armv7 jobs are also executed by aarch64 workers. According to the monitor the armv7 and aarch64 workers have all been busy during the whole week.
For example: https://build.opensuse.org/package/show/devel:languages:python/python-ldap
Looks good right now.
$ osc jobhist devel:languages:python/python-ldap openSUSE_Factory_ARM armv7l time package reason code build time worker 2016-07-16 04:16:31 python-ldap new build succeeded 1m 24s armbuild03:8 2016-07-24 16:48:56 python-ldap new build succeeded 2m 58s armbuild17:2
But it was never published. Currently it hangs in state "scheduled" for hours...................................... Ciao, Michael.
On Mo, Jul 25 2016, Michael Ströder <michael@stroeder.com> wrote:
But it was never published.
Repositories are only published once all package are built.
Currently it hangs in state "scheduled" for hours......................................
There are still more than 5000 Packages scheduled. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Andreas Schwab wrote:
On Mo, Jul 25 2016, Michael Ströder <michael@stroeder.com> wrote:
But it was never published.
Repositories are only published once all package are built.
Currently it hangs in state "scheduled" for hours......................................
There are still more than 5000 Packages scheduled.
On https://build.opensuse.org/ it seems half of the workers are not busy. Ciao, Michael.
On Dienstag, 26. Juli 2016, 09:22:30 CEST wrote Michael Ströder:
Andreas Schwab wrote:
On Mo, Jul 25 2016, Michael Ströder <michael@stroeder.com> wrote:
But it was never published.
Repositories are only published once all package are built.
Currently it hangs in state "scheduled" for hours......................................
There are still more than 5000 Packages scheduled.
On https://build.opensuse.org/ it seems half of the workers are not busy.
You speak about the /monitor page? I can not see that, all armv7 workers are used (check the graphs at the bottom). Same for aarch64. -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Hi Adrian,
I can not see that, all armv7 workers are used (check the graphs at the bottom).
All workers that are *online* are being used. we have a lot more workers that need repair right now though. Greetings, Dirk -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Hi Michael,
last days OBS was quite busy but currently the number of active build workers is below 200.
yeah, there are network issues again in the obs transfer network, most jobs are failing to setup, which slows down the build results. This unfortunately needs to be fixed by a sysadmin though. Greetings, Dirk -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
El 2016-07-24 20:59, Dirk Müller escribió:
Hi Michael,
last days OBS was quite busy but currently the number of active build workers is below 200.
yeah, there are network issues again in the obs transfer network, most jobs are failing to setup, which slows down the build results. This unfortunately needs to be fixed by a sysadmin though.
Greetings, Dirk
Hi, I'm having problems with some package too. In my case, kernel-default at home:ocurero:branches:Kernel:HEAD is crashing randomly but I don't see why. [ 6399s] make[4]: *** [../scripts/Makefile.build:455: drivers/media/usb] Error 2 [ 6399s] make[3]: *** [../scripts/Makefile.build:455: drivers/media] Error 2 [ 6399s] make[2]: *** [/home/abuild/rpmbuild/BUILD/kernel-default-4.7.0/linux-4.7/Makefile:1013: drivers] Error 2 [ 6399s] make[1]: *** [Makefile:150: sub-make] Error 2 [ 6399s] make: *** [Makefile:24: __sub-make] Error 2 [ 6399s] + test 2 -eq 0 [ 6399s] + test 00 -gt 0 [ 6399s] + exit 1 [ 6399s] error: Bad exit status from /var/tmp/rpm-tmp.phVj6r (%build) Is it related to the problems you say? Thanks! -- Cheers -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
On Di, Jul 26 2016, Oscar C <oscar-listas@naiandei.net> wrote:
I'm having problems with some package too. In my case, kernel-default at home:ocurero:branches:Kernel:HEAD is crashing randomly but I don't see why.
It just went OOM. [ 6276s] [ 6208.053222] Out of memory: Kill process 9439 (cc1) score 15 or sacrifice child [ 6276s] [ 6208.054441] Killed process 9439 (cc1) total-vm:61456kB, anon-rss:38356kB, file-rss:12272kB, shmem-rss:0kB Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
El 2016-07-26 12:42, Andreas Schwab escribió:
On Di, Jul 26 2016, Oscar C <oscar-listas@naiandei.net> wrote:
I'm having problems with some package too. In my case, kernel-default at home:ocurero:branches:Kernel:HEAD is crashing randomly but I don't see why.
It just went OOM.
[ 6276s] [ 6208.053222] Out of memory: Kill process 9439 (cc1) score 15 or sacrifice child [ 6276s] [ 6208.054441] Killed process 9439 (cc1) total-vm:61456kB, anon-rss:38356kB, file-rss:12272kB, shmem-rss:0kB
Thanks Andreas! Should I open a bug? The ouput below the OOM says so,but I don't know if is caused by another problem. -- Cheers -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Am 26.07.2016 um 19:55 schrieb Oscar C:
El 2016-07-26 12:42, Andreas Schwab escribió:
On Di, Jul 26 2016, Oscar C <oscar-listas@naiandei.net> wrote:
I'm having problems with some package too. In my case, kernel-default at home:ocurero:branches:Kernel:HEAD is crashing randomly but I don't see why.
It just went OOM.
[ 6276s] [ 6208.053222] Out of memory: Kill process 9439 (cc1) score 15 or sacrifice child [ 6276s] [ 6208.054441] Killed process 9439 (cc1) total-vm:61456kB, anon-rss:38356kB, file-rss:12272kB, shmem-rss:0kB
Thanks Andreas!
Should I open a bug? The ouput below the OOM says so,but I don't know if is caused by another problem.
Before you do, could you explain why you are branching Kernel:HEAD's kernel-default in the first place? It gets rebuilt frequently and occupies build power for almost a day, so do you have any config change or patch that we could simply get into the official Kernel:HEAD and avoid you building your own copy? :) Otherwise, Kernel:stable is now at 4.7, too. At some point Kernel:HEAD will get updated to 4.8-rc1 and all ARM configs will get disabled again. In general you can add or modify the _constraints file to get your build job scheduled on a machine with more RAM. Regards, Andreas F. -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
El 2016-07-26 20:50, Andreas Färber escribió:
Am 26.07.2016 um 19:55 schrieb Oscar C:
El 2016-07-26 12:42, Andreas Schwab escribió:
On Di, Jul 26 2016, Oscar C <oscar-listas@naiandei.net> wrote:
I'm having problems with some package too. In my case, kernel-default at home:ocurero:branches:Kernel:HEAD is crashing randomly but I don't see why.
It just went OOM.
[ 6276s] [ 6208.053222] Out of memory: Kill process 9439 (cc1) score 15 or sacrifice child [ 6276s] [ 6208.054441] Killed process 9439 (cc1) total-vm:61456kB, anon-rss:38356kB, file-rss:12272kB, shmem-rss:0kB
Thanks Andreas!
Should I open a bug? The ouput below the OOM says so,but I don't know if is caused by another problem.
Before you do, could you explain why you are branching Kernel:HEAD's kernel-default in the first place? It gets rebuilt frequently and occupies build power for almost a day, so do you have any config change or patch that we could simply get into the official Kernel:HEAD and avoid you building your own copy? :)
I'm trying to test a patch series [1] so I can boot my udoo board. Actually, I cannot use it because the usb subsystem hangs due to clocks or something like that.
Otherwise, Kernel:stable is now at 4.7, too. At some point Kernel:HEAD will get updated to 4.8-rc1 and all ARM configs will get disabled again.
In general you can add or modify the _constraints file to get your build job scheduled on a machine with more RAM.
Thanks, I will see of that helps.
Regards, Andreas F.
[1] https://lkml.org/lkml/2016/8/1/656 -- Cheers -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Michael Ströder wrote:
armv7l builds on OBS seem to be scheduled but never get executed. During the last days OBS was quite busy but currently the number of active build workers is below 200.
For example: https://build.opensuse.org/package/show/devel:languages:python/python-ldap
It's now more than two weeks ago since I posted this and not even a single armv7l package has been published to http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_... Looking at https://build.opensuse.org/monitor arch armv7l shows that there are big waves of blocked build jobs. The whole algorithm of recalculating the build state seems not to terminate. Ciao, Michael.
On Di, Aug 09 2016, Michael Ströder <michael@stroeder.com> wrote:
It's now more than two weeks ago since I posted this and not even a single armv7l package has been published to
http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_...
The project contains many packages, and the repository is configured for recusive rebuild strategy. That makes it hard to catch up and finish it for the poor armv7 workers.
Looking at https://build.opensuse.org/monitor arch armv7l shows that there are big waves of blocked build jobs. The whole algorithm of recalculating the build state seems not to terminate.
The build state is up-to-date right now. There are currently 1509 scheduled jobs and 179 blocked jobs. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
participants (6)
-
Adrian Schröter -
Andreas Färber -
Andreas Schwab -
Dirk Müller -
Michael Ströder -
Oscar C