[opensuse-buildservice] rpc timeouts: Is api.opensuse.org behaving at the moment?
My local OBS is not able to build anything at the moment. I notice that the scheduler logs are filling with 'rpc timeout' errors /srv/obs/log/scheduler_x86_64.log ... 2013-06-11 14:18:44: looking at high prio Desktop/SLE_11_SP2 (6/0/93/73/91) - Desktop/SLE_11_SP2 fetching remote repository state for openSUSE.org:SUSE:SLE-11:SP2/standard rpc timeout repository 'openSUSE.org:SUSE:SLE-11:SP2/standard' is unavailable This seems to be resulting in many (many!) packages being "broken" or "failed" in the webui monitor screen, and also, for some project/repository combinations: "The repository setup is broken, build not possible" Additionally, when deleting any <repository> ... </repository> block from the project meta, I get an error: "Uncaught exception: undefined method `repositories' for nil:NilClass" ... so I can't reduce the complexity of any projects to help me debug this. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Dienstag, 11. Juni 2013, 14:34:00 schrieb Magnus Lewis-Smith:
My local OBS is not able to build anything at the moment. I notice that the scheduler logs are filling with 'rpc timeout' errors
We see that as well and we saw this caused by a bunch of large requests eating our IO (storage and network). Michael is currently working on an improvement here to avoid that such requests can DoS us...
/srv/obs/log/scheduler_x86_64.log ... 2013-06-11 14:18:44: looking at high prio Desktop/SLE_11_SP2 (6/0/93/73/91) - Desktop/SLE_11_SP2 fetching remote repository state for openSUSE.org:SUSE:SLE-11:SP2/standard rpc timeout repository 'openSUSE.org:SUSE:SLE-11:SP2/standard' is unavailable
This seems to be resulting in many (many!) packages being "broken" or "failed" in the webui monitor screen, and also, for some project/repository combinations: "The repository setup is broken, build not possible"
Additionally, when deleting any <repository> ... </repository> block from the project meta, I get an error: "Uncaught exception: undefined method `repositories' for nil:NilClass" ... so I can't reduce the complexity of any projects to help me debug this. --
Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Jun 11, 2013 at 04:57:43PM +0200, Adrian Schröter wrote:
Am Dienstag, 11. Juni 2013, 14:34:00 schrieb Magnus Lewis-Smith:
My local OBS is not able to build anything at the moment. I notice that the scheduler logs are filling with 'rpc timeout' errors
We see that as well and we saw this caused by a bunch of large requests eating our IO (storage and network).
FYI: The cause of this is currently not known, maybe it has something to do with the migration of our storage space we did two weeks ago. Our storage expert is currently on vacation, though. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
I'm not like broken state. It's really annoying kicking manually rebuilds & fail state because of remote api delays.
I expect scheduler/workers continue build when upstream project appears.
What about delay state and freeze state for project in case remote api errors?
On 11.06.2013, at 18:47, Michael Schroeder
On Tue, Jun 11, 2013 at 04:57:43PM +0200, Adrian Schröter wrote:
Am Dienstag, 11. Juni 2013, 14:34:00 schrieb Magnus Lewis-Smith:
My local OBS is not able to build anything at the moment. I notice that the scheduler logs are filling with 'rpc timeout' errors
We see that as well and we saw this caused by a bunch of large requests eating our IO (storage and network).
FYI: The cause of this is currently not known, maybe it has something to do with the migration of our storage space we did two weeks ago. Our storage expert is currently on vacation, though.
Cheers, Michael.
-- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (4)
-
Adrian Schröter -
Kanstantsin Shautsou -
Magnus Lewis-Smith -
Michael Schroeder