Good evening there,
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
Is this a known problem (server maintenance?) or should I report it?
Regards
On Mai 27 2018, Giovanni Santini itachi.sama.amaterasu@gmail.com wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago.
$ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3
Andreas.
Il 27/05/2018 20:03, Andreas Schwab ha scritto:
On Mai 27 2018, Giovanni Santini itachi.sama.amaterasu@gmail.com wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago.
$ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3
Andreas.
You are correct, although the build for *telegram-desktop-dev* continues to be blocked...
$ osc jobhist home:ItachiSan:archlinux standard x86_64 -l 10 time package reason code build time worker 2018-05-22 06:13:11 ffmpeg meta change succeeded 19m 44s lamb57:12 2018-05-22 06:19:47 gecode meta change succeeded 26m 20s lamb12:3 2018-05-22 07:39:13 telegram-desktop-dev meta change failed 1h 25m 7s lamb20:2 2018-05-25 07:11:44 telegram-desktop-dev meta change failed 1h 32m 31s lamb25:8 2018-05-25 08:37:09 telegram-desktop-dev meta change succeeded 1h 18m 2s build84:4 2018-05-26 16:59:09 ffmpeg meta change succeeded 37m 46s cloud113:4 2018-05-26 17:10:21 gecode meta change succeeded 48m 56s cloud125:3 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode
Still, I updated *telegram-desktop-dev* yesterday, the 26th.
On Sonntag, 27. Mai 2018, 22:02:02 CEST wrote Giovanni Santini:
Il 27/05/2018 20:03, Andreas Schwab ha scritto:
On Mai 27 2018, Giovanni Santini itachi.sama.amaterasu@gmail.com wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago.
$ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3
Andreas.
You are correct, although the build for *telegram-desktop-dev* continues to be blocked...
I notice an extreme slow-down on our backend responsible for home projects atm.
Will need to debug where this is comming from suddenly, but most likely not today :/
Hi Giovanni,
On Montag, 28. Mai 2018 08:47:57 Adrian Schröter wrote:
On Sonntag, 27. Mai 2018, 22:02:02 CEST wrote Giovanni Santini:
Il 27/05/2018 20:03, Andreas Schwab ha scritto:
On Mai 27 2018, Giovanni Santini itachi.sama.amaterasu@gmail.com
wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago.
$ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3
Andreas.
You are correct, although the build for *telegram-desktop-dev* continues to be blocked...
I notice an extreme slow-down on our backend responsible for home projects atm.
Will need to debug where this is comming from suddenly, but most likely not today :/
Have a look here: https://build.opensuse.org/monitor (bottom of page)
As long as repo recalculation count keep above 500, OBS "feels" really slow. Symptom: package builds keep in finished state for a long time. Since RRC is way above 2000 since at least two days, OBS "feels" almost stalled.
This is a rare event: during the last year RRC was bigger only once (in January), where it exceeded 10000.
Let's hope, that things settle again in the next few days.
Cheers, Pete
On Montag, 28. Mai 2018, 12:30:51 CEST wrote Hans-Peter Jansen:
Hi Giovanni,
On Montag, 28. Mai 2018 08:47:57 Adrian Schröter wrote:
On Sonntag, 27. Mai 2018, 22:02:02 CEST wrote Giovanni Santini:
Il 27/05/2018 20:03, Andreas Schwab ha scritto:
On Mai 27 2018, Giovanni Santini itachi.sama.amaterasu@gmail.com
wrote:
I have noticed that builds for my repository are stuck to 2 days ago. Using `osc rbuild PACKAGE` doesn't work too.
My repository is: home:ItachiSan:archlinux
The builds finished only a few hours ago.
$ osc jobhist -l3 home:ItachiSan:archlinux standard x86_64 time package reason code build time worker 2018-05-27 14:31:06 webkit2gtk-mse new build failed 30m 24s lamb66:3 2018-05-27 14:36:08 ffmpeg meta change succeeded 35m 25s lamb27:7 2018-05-27 15:23:37 gecode meta change succeeded 50m 25s cloud124:3
Andreas.
You are correct, although the build for *telegram-desktop-dev* continues to be blocked...
I notice an extreme slow-down on our backend responsible for home projects atm.
Will need to debug where this is comming from suddenly, but most likely not today :/
Have a look here: https://build.opensuse.org/monitor (bottom of page)
As long as repo recalculation count keep above 500, OBS "feels" really slow. Symptom: package builds keep in finished state for a long time. Since RRC is way above 2000 since at least two days, OBS "feels" almost stalled.
This is a rare event: during the last year RRC was bigger only once (in January), where it exceeded 10000.
Let's hope, that things settle again in the next few days.
yes. JFYI, one issue got solved and the scheduler is a bit faster again. But has still a quite large backlog.
btw, this affects "only" x86_64 in home:* projects.
Please use local builds (osc build ..) if you feel slowed down, this is the fastest way to test or debug something as usual and just submit your working version. That one will build on the server eventually :)
Am 28.05.2018 um 12:40 schrieb Adrian Schröter:
yes. JFYI, one issue got solved and the scheduler is a bit faster again. But has still a quite large backlog.
btw, this affects "only" x86_64 in home:* projects.
Please use local builds (osc build ..) if you feel slowed down, this is the fastest way to test or debug something as usual and just submit your working version. That one will build on the server eventually:)
I think there is still something wrong. The builds are fine now but recalculation/publication does still not work.
On Dienstag, 29. Mai 2018, 08:08:24 CEST wrote Johannes Weberhofer:
Am 28.05.2018 um 12:40 schrieb Adrian Schröter:
yes. JFYI, one issue got solved and the scheduler is a bit faster again. But has still a quite large backlog.
btw, this affects "only" x86_64 in home:* projects.
Please use local builds (osc build ..) if you feel slowed down, this is the fastest way to test or debug something as usual and just submit your working version. That one will build on the server eventually:)
I think there is still something wrong. The builds are fine now but recalculation/publication does still not work.
It does work, but it takes forever .....
However, we solved a number of IO bottlenecks (network and disk) this morning.
It seems it has improved the situation, but I suspect it will take until tomorrow until we see the full effect. The caches on various workers and servers need to get filled again.
But we have a trend change in any case, the steady increasing number of to repositories which needs to be recalculate is decreasing now again (currently already <4000 repos on x86_64 home:* projects).
On Dienstag, 29. Mai 2018, 10:42:33 CEST wrote Adrian Schröter:
On Dienstag, 29. Mai 2018, 08:08:24 CEST wrote Johannes Weberhofer:
Am 28.05.2018 um 12:40 schrieb Adrian Schröter:
yes. JFYI, one issue got solved and the scheduler is a bit faster again. But has still a quite large backlog.
btw, this affects "only" x86_64 in home:* projects.
Please use local builds (osc build ..) if you feel slowed down, this is the fastest way to test or debug something as usual and just submit your working version. That one will build on the server eventually:)
I think there is still something wrong. The builds are fine now but recalculation/publication does still not work.
It does work, but it takes forever .....
However, we solved a number of IO bottlenecks (network and disk) this morning.
It seems it has improved the situation, but I suspect it will take until tomorrow until we see the full effect. The caches on various workers and servers need to get filled again.
But we have a trend change in any case, the steady increasing number of to repositories which needs to be recalculate is decreasing now again (currently already <4000 repos on x86_64 home:* projects).
JFYI, we down to 200 repos ... a usual value.
You can see the effect also on
https://build.opensuse.org/monitor
the drop of medium prio events for repo recalculation in the gfx on bottom left.
Just a bit higher number of waiting jobs, but we have also all workers running. So we should be fine again.
buildservice@lists.opensuse.org
-
Adrian Schröter -
Andreas Schwab -
Giovanni Santini -
Hans-Peter Jansen -
Johannes Weberhofer