[opensuse-buildservice] often recurring problem with project/repository "bad config"
Hi all, recently I quite often see certein repo's in projects in the state "The repository setup is broken, build not possible.". In the scheduler_* log I see for instance ==== vdrdevel/openSUSE_12.3: bad config (openSUSE.org:openSUSE:12.3:Update) ==== I see that on private build instances, which are using the OBS-repos using interconnect. Henne suggested on IRC, that this may be caused by network (connection) problems. I see that on three OBS instances, one of them uses a completely different path to build.openSUSE.org, yet I see the problems appearing around the same time (just some minutes apart). That hints, that - if this is caused by connection problems - the cause origins at the opensuse-end of the connection. I do not think that this is actually caused by "network" problems, but is caused by the load put on b.o.o. As this can be seen as an expected problem, I would like to have a solution for this problem. To overcome the blockade I usually restart the scheduler. I have seen the symptom disappearing by doing a submitrequest also, but I have several projects with packages linked to b.o.o and just having local patches, so I rather seldom submit packages. The scheduler should clear this "bad config" condition by retrying to retrieve the repo config after a certain time. My first question: does my observation make sense? How can I debug that behavior, or how can I be of help? The second: if my observation is correct, can there be a mechanism, which "self-heals" the system? If I can supply additional information, I am happy to supply everything needed. Thank you for your time, Stefan -- Stefan Botter Network Manager Jacobs University Bremen gGmbH Campus Ring 1 | 28759 Bremen | Germany Commercial registry: Amtsgericht Bremen, HRB 18117 President & Geschäftsführer: Prof. Dr. Heinz-Otto Peitgen Provost & Geschäftsführerin: Prof. Dr.-Ing. Katja Windt Chair Board of Governors: Prof. Dr. Karin Lochte
Stefan Botter <s.botter@jacobs-university.de> writes:
The second: if my observation is correct, can there be a mechanism, which "self-heals" the system?
I think obs_admin --deep-check-project can fix that up. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi Andreas, On Fri, 08 Nov 2013 11:48:26 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote:
Stefan Botter <s.botter@jacobs-university.de> writes:
The second: if my observation is correct, can there be a mechanism, which "self-heals" the system?
I think obs_admin --deep-check-project can fix that up.
Indeed, it does for that particular project. That would mean, I need to set up a watcher, which monitors the log file of the scheduler and invent a logic to recheck the project. Of course this will happen more often on a remote appliance than on the central build server, where there are check-ins every few seconds, but still it seems awkward, that the scheduler, which at least detects the problem with a project, just abandons this project, and never revisits it on its own. Can the scheduler upon detecting "bad config" initiate a job to deep-check the particular project?
Andreas.
Stefan -- Stefan Botter Network Manager Jacobs University Bremen gGmbH Campus Ring 1 | 28759 Bremen | Germany Commercial registry: Amtsgericht Bremen, HRB 18117 President & Geschäftsführer: Prof. Dr. Heinz-Otto Peitgen Provost & Geschäftsführerin: Prof. Dr.-Ing. Katja Windt Chair Board of Governors: Prof. Dr. Karin Lochte
participants (2)
-
Andreas Schwab -
Stefan Botter