[opensuse-buildservice] Builds not working?
Good evening there, I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too. My repository is: home:ItachiSan:archlinux Is this a known problem (server maintenance?) or should I report it? Regards -- Giovanni Santini My blog: http://giovannisantini.tk My code: https://git{hub,lab}.com/ItachiSan My GPG: 2FADEBF5 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mai 27 2018, Giovanni Santini <itachi.sama.amaterasu@gmail.com> wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago. $ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3 Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Il 27/05/2018 20:03, Andreas Schwab ha scritto:
On Mai 27 2018, Giovanni Santini <itachi.sama.amaterasu@gmail.com> wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago.
$ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3
Andreas.
You are correct, although the build for *telegram-desktop-dev* continues to be blocked... $ osc jobhist home:ItachiSan:archlinux standard x86_64 -l 10 time package reason code build time worker 2018-05-22 06:13:11 ffmpeg meta change succeeded 19m 44s lamb57:12 2018-05-22 06:19:47 gecode meta change succeeded 26m 20s lamb12:3 2018-05-22 07:39:13 telegram-desktop-dev meta change failed 1h 25m 7s lamb20:2 2018-05-25 07:11:44 telegram-desktop-dev meta change failed 1h 32m 31s lamb25:8 2018-05-25 08:37:09 telegram-desktop-dev meta change succeeded 1h 18m 2s build84:4 2018-05-26 16:59:09 ffmpeg meta change succeeded 37m 46s cloud113:4 2018-05-26 17:10:21 gecode meta change succeeded 48m 56s cloud125:3 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode Still, I updated *telegram-desktop-dev* yesterday, the 26th. -- Giovanni Santini My blog: http://giovannisantini.tk My code: https://git{hub,lab}.com/ItachiSan My GPG: 2FADEBF5 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Sonntag, 27. Mai 2018, 22:02:02 CEST wrote Giovanni Santini:
Il 27/05/2018 20:03, Andreas Schwab ha scritto:
On Mai 27 2018, Giovanni Santini <itachi.sama.amaterasu@gmail.com> wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago.
$ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3
Andreas.
You are correct, although the build for *telegram-desktop-dev* continues to be blocked...
I notice an extreme slow-down on our backend responsible for home projects atm. Will need to debug where this is comming from suddenly, but most likely not today :/ -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi Giovanni, On Montag, 28. Mai 2018 08:47:57 Adrian Schröter wrote:
On Sonntag, 27. Mai 2018, 22:02:02 CEST wrote Giovanni Santini:
Il 27/05/2018 20:03, Andreas Schwab ha scritto:
On Mai 27 2018, Giovanni Santini <itachi.sama.amaterasu@gmail.com> wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago.
$ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3
Andreas.
You are correct, although the build for *telegram-desktop-dev* continues to be blocked...
I notice an extreme slow-down on our backend responsible for home projects atm.
Will need to debug where this is comming from suddenly, but most likely not today :/
Have a look here: https://build.opensuse.org/monitor (bottom of page) As long as repo recalculation count keep above 500, OBS "feels" really slow. Symptom: package builds keep in finished state for a long time. Since RRC is way above 2000 since at least two days, OBS "feels" almost stalled. This is a rare event: during the last year RRC was bigger only once (in January), where it exceeded 10000. Let's hope, that things settle again in the next few days. Cheers, Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Montag, 28. Mai 2018, 12:30:51 CEST wrote Hans-Peter Jansen:
Hi Giovanni,
On Montag, 28. Mai 2018 08:47:57 Adrian Schröter wrote:
On Sonntag, 27. Mai 2018, 22:02:02 CEST wrote Giovanni Santini:
Il 27/05/2018 20:03, Andreas Schwab ha scritto:
On Mai 27 2018, Giovanni Santini <itachi.sama.amaterasu@gmail.com> wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago.
$ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3
Andreas.
You are correct, although the build for *telegram-desktop-dev* continues to be blocked...
I notice an extreme slow-down on our backend responsible for home projects atm.
Will need to debug where this is comming from suddenly, but most likely not today :/
Have a look here: https://build.opensuse.org/monitor (bottom of page)
As long as repo recalculation count keep above 500, OBS "feels" really slow. Symptom: package builds keep in finished state for a long time. Since RRC is way above 2000 since at least two days, OBS "feels" almost stalled.
This is a rare event: during the last year RRC was bigger only once (in January), where it exceeded 10000.
Let's hope, that things settle again in the next few days.
yes. JFYI, one issue got solved and the scheduler is a bit faster again. But has still a quite large backlog. btw, this affects "only" x86_64 in home:* projects. Please use local builds (osc build ..) if you feel slowed down, this is the fastest way to test or debug something as usual and just submit your working version. That one will build on the server eventually :) -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am 28.05.2018 um 12:40 schrieb Adrian Schröter:
yes. JFYI, one issue got solved and the scheduler is a bit faster again. But has still a quite large backlog.
btw, this affects "only" x86_64 in home:* projects.
Please use local builds (osc build ..) if you feel slowed down, this is the fastest way to test or debug something as usual and just submit your working version. That one will build on the server eventually:)
I think there is still something wrong. The builds are fine now but recalculation/publication does still not work. -- Johannes Weberhofer Weberhofer GmbH, Austria, Vienna -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Dienstag, 29. Mai 2018, 08:08:24 CEST wrote Johannes Weberhofer:
Am 28.05.2018 um 12:40 schrieb Adrian Schröter:
yes. JFYI, one issue got solved and the scheduler is a bit faster again. But has still a quite large backlog.
btw, this affects "only" x86_64 in home:* projects.
Please use local builds (osc build ..) if you feel slowed down, this is the fastest way to test or debug something as usual and just submit your working version. That one will build on the server eventually:)
I think there is still something wrong. The builds are fine now but recalculation/publication does still not work.
It does work, but it takes forever ..... However, we solved a number of IO bottlenecks (network and disk) this morning. It seems it has improved the situation, but I suspect it will take until tomorrow until we see the full effect. The caches on various workers and servers need to get filled again. But we have a trend change in any case, the steady increasing number of to repositories which needs to be recalculate is decreasing now again (currently already <4000 repos on x86_64 home:* projects). -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Dienstag, 29. Mai 2018, 10:42:33 CEST wrote Adrian Schröter:
On Dienstag, 29. Mai 2018, 08:08:24 CEST wrote Johannes Weberhofer:
Am 28.05.2018 um 12:40 schrieb Adrian Schröter:
yes. JFYI, one issue got solved and the scheduler is a bit faster again. But has still a quite large backlog.
btw, this affects "only" x86_64 in home:* projects.
Please use local builds (osc build ..) if you feel slowed down, this is the fastest way to test or debug something as usual and just submit your working version. That one will build on the server eventually:)
I think there is still something wrong. The builds are fine now but recalculation/publication does still not work.
It does work, but it takes forever .....
However, we solved a number of IO bottlenecks (network and disk) this morning.
It seems it has improved the situation, but I suspect it will take until tomorrow until we see the full effect. The caches on various workers and servers need to get filled again.
But we have a trend change in any case, the steady increasing number of to repositories which needs to be recalculate is decreasing now again (currently already <4000 repos on x86_64 home:* projects).
JFYI, we down to 200 repos ... a usual value. You can see the effect also on https://build.opensuse.org/monitor the drop of medium prio events for repo recalculation in the gfx on bottom left. Just a bit higher number of waiting jobs, but we have also all workers running. So we should be fine again. -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (5)
-
Adrian Schröter -
Andreas Schwab -
Giovanni Santini -
Hans-Peter Jansen -
Johannes Weberhofer