[opensuse-buildservice] Blame machine enhancement
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello! Machine build09 (i586) is in funky mood again. That means it is throwing error messages like "Unable to connect to xend: Connection reset by peer. Is xend running?" or "Error: (1, 'Internal error', 'Could not obtain handle on privileged command interface (24 = Too many open files)')". Waiting queue for x86_64 is around 120 tasks, while i586 is empty. That's because build09 is always idle, so scheduler sends tasks to it and the build instantaneously fails. This situation is unfortunate, because it "pollutes" OBS with failed status and newly built projects have practically no chance to build at all. My proposal is to add machine list (maybe with hardware specs) to BuildService and add "Blame" button to each machine. When user clicks the button, BuildService will send an e-mail to OBS administrator, who can decide whether to restart the machine or not. This process can be semiautomated by parsing failed build logs and if there are 10+ known error messages (Is xend running?, Too many open files) for one machine, e-mail is sent as well. - -- Best Regards / S pozdravom, Pavol RUSNAK SUSE LINUX, s.r.o Package Maintainer Lihovarska 1060/12 PGP 0xA6917144 19000 Praha 9, CR prusnak[at]suse.cz http://www.suse.cz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFF1DNdASE5C6aRcUQRAhnDAKCzOWFTKFVHnTB0bQNqr2Cu/+KHdQCaAgbf xWdVYgwW8qistkB/tiU074U= =EeJd -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Donnerstag 15 Februar 2007 11:18:05 schrieb Pavol Rusnak:
Hello!
Machine build09 (i586) is in funky mood again. That means it is throwing error messages like "Unable to connect to xend: Connection reset by peer. Is xend running?" or "Error: (1, 'Internal error', 'Could not obtain handle on privileged command interface (24 = Too many open files)')".
Can you please report with an URL to a log file where you see that ?
Waiting queue for x86_64 is around 120 tasks, while i586 is empty. That's because build09 is always idle, so scheduler sends tasks to it and the build instantaneously fails. This situation is unfortunate, because it "pollutes" OBS with failed status and newly built projects have practically no chance to build at all. My proposal is to add machine list (maybe with hardware specs) to BuildService and add "Blame" button to each machine. When user clicks the button, BuildService will send an e-mail to OBS administrator, who can decide whether to restart the machine or not. This process can be semiautomated by parsing failed build logs and if there are 10+ known error messages (Is xend running?, Too many open files) for one machine, e-mail is sent as well.
We need to fix the issues and not to add workarounds. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Adrian Schröter wrote:
Can you please report with an URL to a log file where you see that ?
http://build.opensuse.org/project/monitor?project=X11%3Axfce Nearly all failures on i586 are because of the build09 machine. - -- Best Regards / S pozdravom, Pavol RUSNAK SUSE LINUX, s.r.o Package Maintainer Lihovarska 1060/12 PGP 0xA6917144 19000 Praha 9, CR prusnak[at]suse.cz http://www.suse.cz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFF1D5HASE5C6aRcUQRAldrAKDDucmZtaNDGlW6u5eWrCNnD0XNiACffKNh tN4YFsmdyRj6W0SLzCpp/8M= =qLxA -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Feb 15, 07 11:50:33 +0100, Adrian Schröter wrote:
Am Donnerstag 15 Februar 2007 11:18:05 schrieb Pavol Rusnak:
Hello!
Machine build09 (i586) is in funky mood again. That means it is throwing error messages like "Unable to connect to xend: Connection reset by peer. Is xend running?" or "Error: (1, 'Internal error', 'Could not obtain handle on privileged command interface (24 = Too many open files)')".
Can you please report with an URL to a log file where you see that ?
Yes. Xen is leaking filedescriptors. Xen bug. Example of a log: http://build.opensuse.org/package/live_build_log?package=XaraLX&arch=i586&repository=SUSE_Factory&project=home%3Ajnweiger cheers, Jw. -- o \ Juergen Weigert paint it green! __/ _=======.=======_ <V> | jw@suse.de wide open suse_/ _---|____________\/ \ | 0911 74053-508 (tm)__/ (____/ /\ (/) | __________________________/ _/ \_ vim:set sw=2 wm=8 SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Feb 15, 2007 at 11:18:05AM +0100, Pavol Rusnak wrote:
Machine build09 (i586) is in funky mood again. That means it is throwing error messages like "Unable to connect to xend: Connection reset by peer. Is xend running?" or "Error: (1, 'Internal error', 'Could not obtain handle on privileged command interface (24 = Too many open files)')".
Happens every 1000 builds or so, it's xend running out of file descriptors because it has a fd leak. Cheers, Michael. -- Michael Schroeder mls@suse.de main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Michael Schroeder wrote:
Happens every 1000 builds or so, it's xend running out of file descriptors because it has a fd leak.
On build09 it happens EVERY build. - -- Best Regards / S pozdravom, Pavol RUSNAK SUSE LINUX, s.r.o Package Maintainer Lihovarska 1060/12 PGP 0xA6917144 19000 Praha 9, CR prusnak[at]suse.cz http://www.suse.cz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFF1ELCASE5C6aRcUQRAqwUAKCgDhjoejw6Es17NsYq1eZZmHisTwCgjrkv ekf3S+kfQpikO4c80KTPr8c= =sfVS -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Donnerstag 15 Februar 2007 12:23:46 schrieb Pavol Rusnak:
Michael Schroeder wrote:
Happens every 1000 builds or so, it's xend running out of file descriptors because it has a fd leak.
On build09 it happens EVERY build.
No, it does not. All current 8 builds on that system seems to run fine atm. So please send a link to the log file, I will not reverse engineer where you have seen this problem. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday 15 February 2007 13:56:48 Adrian Schröter wrote:
So please send a link to the log file, I will not reverse engineer where you have seen this problem.
Build02/3 is taking 12 hours to compile something that should take 30 minutes to compile: http://build.opensuse.org/package/live_build_log?package=gdb&arch=x86_64&repository=Fedora_Extras_5&project=home%3Aamilcarlucas -- Amilcar Lucas Webmaster The KDevelop project --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Feb 15, 2007 at 01:56:48PM +0100, Adrian Schröter wrote:
Am Donnerstag 15 Februar 2007 12:23:46 schrieb Pavol Rusnak:
Michael Schroeder wrote:
Happens every 1000 builds or so, it's xend running out of file descriptors because it has a fd leak.
On build09 it happens EVERY build.
No, it does not. All current 8 builds on that system seems to run fine atm.
'Cause I did a 'xend restart'. M. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Feb 15, 2007 at 12:23:46PM +0100, Pavol Rusnak wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Michael Schroeder wrote:
Happens every 1000 builds or so, it's xend running out of file descriptors because it has a fd leak.
On build09 it happens EVERY build.
Yeah, bad wording. I meant it is broken after 1000 builds. M. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (5)
-
Adrian Schröter -
Amilcar do Carmo Lucas -
Juergen Weigert -
Michael Schroeder -
Pavol Rusnak