[opensuse-buildservice] 335 builds failed today with 'No space left on device' / old-atreju
Dear OBS community, this morning I was flooded with 335 e-mails notifying me of build failures, all boiling down to variants of the following message: ------ Last lines of build log: /var/cache/obs/cache/47/471d708c4377c6f34a22d6377c9a439f.18970 write: No space left on device (worker was old-atreju1:9) ------ In case anyone is interested in all of the failures, I've exported them to https://people.osmocom.org/laforge/tmp/20200723-obs-fails.mbox All of those builds had 'old-atreju' in the name. Is that all just one physical machine? Or did dozens of machines all run out of space at the same time? I was a bit surprised that nobody had brought this up on the list yet, given that I'm sure our Osmocom packages are not the only ones with massive build-failure e-mail floods this morning. Thanks! Regards, Harald -- - Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Donnerstag, 23. Juli 2020, 17:51:17 CEST wrote Harald Welte:
Dear OBS community,
this morning I was flooded with 335 e-mails notifying me of build failures, all boiling down to variants of the following message:
------ Last lines of build log: /var/cache/obs/cache/47/471d708c4377c6f34a22d6377c9a439f.18970 write: No space left on device (worker was old-atreju1:9) ------
In case anyone is interested in all of the failures, I've exported them to https://people.osmocom.org/laforge/tmp/20200723-obs-fails.mbox
All of those builds had 'old-atreju' in the name. Is that all just one physical machine? Or did dozens of machines all run out of space at the same time?
I was a bit surprised that nobody had brought this up on the list yet, given that I'm sure our Osmocom packages are not the only ones with massive build-failure e-mail floods this morning.
Sorry, we had actually two problems here. For one that these workers were mis-configured at all to run out of disk space. Also we should catch such errors automatically and do not deliver failures for it, what did not happen in this particular situation. We fixed both on thursday morning however, but missed to restart the builds. So you need to retrigger them, but this error should not return anymore:) good morning adrian -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (2)
-
Adrian Schröter -
Harald Welte