[yast-devel] Fail fast at Travis?
Hi all, I'd like to discuss one Travis feature here. Currently all Travis steps described in .travis.yml are executed regardless whether something fails or not. The problem is that if any step in the middle fails then it's quite hard to find the issue as the logs are usually very long. For example the yast2 log has over 4500 lines [1]. The question is whether we want to switch to the fail fast principle [2], so the Travis build would stop at the first failure and you could easily find it at the end of the log. (Travis does not support failing fast out of box, but there is a workaround for this [3].) Summary: Current state (always run all steps) + can find multiple issues in a single run - too long log for finding the issues - may report false positives for dependent steps (e.g. if compilation fails then obviously the tests fail too) - longer build times Fail fast + easy to find the failure + failed builds finish faster - you can only see/fix one issue at a time, more iterations for fixing all issues is needed (e.g. you fix a rubocop style issue, but then the testsuite does not pass, after fixing it the install step fails because of a missing change in Makefile... each fix means a new commit and a new build) What do you think about it? Should we change the Travis behavior? [1] https://travis-ci.org/yast/yast-yast2 [2] http://en.wikipedia.org/wiki/Fail-fast [3] https://github.com/travis-ci/travis-ci/issues/1066#issuecomment-32415453 -- Best Regards Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/ -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Thu, 22 Jan 2015 15:09:01 +0100 Ladislav Slezak <lslezak@suse.cz> wrote:
Hi all,
I'd like to discuss one Travis feature here.
Currently all Travis steps described in .travis.yml are executed regardless whether something fails or not.
The problem is that if any step in the middle fails then it's quite hard to find the issue as the logs are usually very long. For example the yast2 log has over 4500 lines [1].
Is possible to have it less verbose? all traversing dirs looks useless for me.
The question is whether we want to switch to the fail fast principle [2], so the Travis build would stop at the first failure and you could easily find it at the end of the log. (Travis does not support failing fast out of box, but there is a workaround for this [3].)
Summary:
Current state (always run all steps)
+ can find multiple issues in a single run
this is quite good with rubocop, as it is good to find problem in style and test together.
- too long log for finding the issues
I think in general we should find way how to reduce log length
- may report false positives for dependent steps (e.g. if compilation fails then obviously the tests fail too)
not much often in yast I think
- longer build times
Travis ones, so not a problem for us :)
Fail fast
+ easy to find the failure + failed builds finish faster - you can only see/fix one issue at a time, more iterations for fixing all issues is needed (e.g. you fix a rubocop style issue, but then the testsuite does not pass, after fixing it the install step fails because of a missing change in Makefile... each fix means a new commit and a new build)
Also setting set -x can have some side effects that is not expected. in general we should try to have short log if it succeed and long if failed. maybe it is just small tunning of commands?
What do you think about it? Should we change the Travis behavior?
In general I am fine with current behavior, just succeed command should not have so much log lines. Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 01/27/2015 11:48 AM, Josef Reidinger wrote:
On Thu, 22 Jan 2015 15:09:01 +0100 Ladislav Slezak <lslezak@suse.cz> wrote:
Hi all,
I'd like to discuss one Travis feature here.
Currently all Travis steps described in .travis.yml are executed regardless whether something fails or not.
The problem is that if any step in the middle fails then it's quite hard to find the issue as the logs are usually very long. For example the yast2 log has over 4500 lines [1].
Is possible to have it less verbose? all traversing dirs looks useless for me.
Well, what about just to listing the tests which succeed without any additional data and only including the details for failed tests in the log? Jiri
The question is whether we want to switch to the fail fast principle [2], so the Travis build would stop at the first failure and you could easily find it at the end of the log. (Travis does not support failing fast out of box, but there is a workaround for this [3].)
Summary:
Current state (always run all steps)
+ can find multiple issues in a single run
this is quite good with rubocop, as it is good to find problem in style and test together.
- too long log for finding the issues
I think in general we should find way how to reduce log length
- may report false positives for dependent steps (e.g. if compilation fails then obviously the tests fail too)
not much often in yast I think
- longer build times
Travis ones, so not a problem for us :)
Fail fast
+ easy to find the failure + failed builds finish faster - you can only see/fix one issue at a time, more iterations for fixing all issues is needed (e.g. you fix a rubocop style issue, but then the testsuite does not pass, after fixing it the install step fails because of a missing change in Makefile... each fix means a new commit and a new build)
Also setting set -x can have some side effects that is not expected. in general we should try to have short log if it succeed and long if failed. maybe it is just small tunning of commands?
What do you think about it? Should we change the Travis behavior?
In general I am fine with current behavior, just succeed command should not have so much log lines.
Josef
-- Regards, Jiri Srain Project Manager --------------------------------------------------------------------- SUSE LINUX, s.r.o. e-mail: jsrain@suse.com Lihovarska 1060/12 tel: +420 284 084 659 190 00 Praha 9 fax: +420 284 084 001 Czech Republic http://www.suse.com -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 27.1.2015 v 12:31 Jiri Srain napsal(a):
On 01/27/2015 11:48 AM, Josef Reidinger wrote: [...]
Is possible to have it less verbose? all traversing dirs looks useless for me.
Good idea, I did a test in yast2 package: - Removed "rake check:syntax" call (useless after adding rubocop check, the rubocop check fails when there is a syntax error so it's not needed any more). It prints status for each file, removing it saves ~250 lines in the output. - Added "-s" (silent) "make" option, it hides messages like "Entering directory ...", "Nothing to be done for ...", "Leaving directory ...", which are printed a lot. The result is really interesting, the log size has been decreased from ~4600 lines [1] to ~2500 [2], which is a huge difference (almost half of the original size!). I'll open a PR for that change...
Well, what about just to listing the tests which succeed without any additional data and only including the details for failed tests in the log?
Um, that would be nice, unfortunately it cannot be implemented easily :-( The problem is that Travis displays STDOUT + STDERR of the executed script, you would need to either change the make/rake behavior (huh...) or add some output filtering (that would be non-trivial and probably very sensitive to output format changes in rake/make so not really recommended...). [1] https://travis-ci.org/yast/yast-yast2/builds/48464147 [2] https://travis-ci.org/yast/yast-yast2/builds/48468523 -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 27.1.2015 13:25, Ladislav Slezak wrote:
Dne 27.1.2015 v 12:31 Jiri Srain napsal(a):
On 01/27/2015 11:48 AM, Josef Reidinger wrote: [...]
Is possible to have it less verbose? all traversing dirs looks useless for me. The result is really interesting, the log size has been decreased from ~4600 lines [1] to ~2500 [2], which is a huge difference (almost half of the original size!).
That's a very good progress, but still not enough. If it sounds too bad, sorry for that. It's definitely a job well done and we all appreciate it! :)
Well, what about just to listing the tests which succeed without any additional data and only including the details for failed tests in the log?
Um, that would be nice, unfortunately it cannot be implemented easily :-(
But still sounds like the best solution.
The problem is that Travis displays STDOUT + STDERR of the executed script, you would need to either change the make/rake behavior (huh...) or add some output filtering (that would be non-trivial and probably very sensitive to output format changes in rake/make so not really recommended...).
Just an idea... don't throw stones at me :) for every single test do: 1. Redirect stderr to stdout 2. Save stdout to a temporary file 3. Check the exit code, if it passed, show: X -> passed 4. If it failed, show all the details and x -> failed 5. Remove the tempfile Would it work? Lukas -- Lukas Ocilka, Systems Management (Yast) Team Leader SLE Department, SUSE Linux -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 27.1.2015 v 14:28 Lukas Ocilka napsal(a): [...]
for every single test do: 1. Redirect stderr to stdout 2. Save stdout to a temporary file 3. Check the exit code, if it passed, show: X -> passed 4. If it failed, show all the details and x -> failed 5. Remove the tempfile
Um, that would probably make sense only for the old testsuite, RSpec output can be used as a documentation too, see e.g. [1]. That output is useful even though there is no error, e.g. you can easily put a link to IRC describing the behavior of a class or method... [1] https://travis-ci.org/yast/yast-registration#L569 -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
participants (4)
-
Jiri Srain -
Josef Reidinger -
Ladislav Slezak -
Lukas Ocilka