please forgive my late answer. Better late than never, eh? ;)
On Wednesday, 2 December 2020 21.44.21 CET Martin Wilck wrote:
On Fri, 2020-11-27 at 20:52 +0100, Oliver Kurz wrote:
> To all users of openQA or anyone interested in running system-level
> tests in
> CI jobs,
> I would like to make you aware of a new approach for running system-
> tests using "isotovideo", the main backend of openQA tests, within
> integration (CI) services, for example github actions, travis CI,
> gitlab CI,
> Following an updated section
> it is possible to run "openQA tests" using a simple, convenient
> based on tests that can be provided for example in a git repository.
> An easy example is described in
Unless I'm missing something essential, the test step "Run isotovideo
against test code in happy-path scenario" returned 0, although the test
[2020-11-26T15:08:23.011 UTC] [debug] >>> testapi::_check_backend_response:
match=bootloader timed out after 30 (assert_screen)
[2020-11-26T15:08:23.092 UTC] [info] ::: basetest::runtest: # Test died: no
candidate needle with tag(s) 'bootloader' matched ...
1: EXIT 0
Yes, that is correct. This is discussed in
"EXIT 0" means that isotovideo could run the complete test from beginning to
end. Various test steps within could have still resulted in "failed".
I suppose the test fails because it's "one
simple test where you need to
the needles for". Forgive my ignorance - how would I do that? Can I
fetch them from somewhere?
In this example repository there are no needles because this is left as a
learning exercise to new users as described in
More importantly, how would I set up this test so that
it actually fails
the CI when it the test fails?
To make CI runs fail if any test modules fail within the isotovideo run one
should look for the "result" field in all testresults/result-*.json file like
for example this way:
jq '.result' testresults/result-*.json | grep -v 'ok' && echo
failed" && exit 1
Thanks for your feedback and good questions. I will try to clarify that within