Hi Martin, please forgive my late answer. Better late than never, eh? ;) On Wednesday, 2 December 2020 21.44.21 CET Martin Wilck wrote:
On Fri, 2020-11-27 at 20:52 +0100, Oliver Kurz wrote:
To all users of openQA or anyone interested in running system-level tests in CI jobs,
I would like to make you aware of a new approach for running system- level tests using "isotovideo", the main backend of openQA tests, within continuous integration (CI) services, for example github actions, travis CI, gitlab CI, etc.
Following an updated section https://github.com/os-autoinst/os-autoinst#getting-started it is possible to run "openQA tests" using a simple, convenient container based on tests that can be provided for example in a git repository.
An easy example is described in https://github.com/os-autoinst/os-autoinst-distri-example#local-testing-an d-ci-environment
Find the latest example CI run directly following https://github.com/os-autoinst/os-autoinst-distri-example/actions
Unless I'm missing something essential, the test step "Run isotovideo against test code in happy-path scenario" returned 0, although the test failed.
[2020-11-26T15:08:23.011 UTC] [debug] >>> testapi::_check_backend_response: match=bootloader timed out after 30 (assert_screen) [2020-11-26T15:08:23.092 UTC] [info] ::: basetest::runtest: # Test died: no candidate needle with tag(s) 'bootloader' matched ... 1: EXIT 0
Yes, that is correct. This is discussed in https://github.com/os-autoinst/os-autoinst-distri-example/issues/6 "EXIT 0" means that isotovideo could run the complete test from beginning to end. Various test steps within could have still resulted in "failed".
I suppose the test fails because it's "one simple test where you need to create the needles for". Forgive my ignorance - how would I do that? Can I fetch them from somewhere?
In this example repository there are no needles because this is left as a learning exercise to new users as described in https://github.com/os-autoinst/os-autoinst-distri-example#local-testing-and-...
More importantly, how would I set up this test so that it actually fails the CI when it the test fails?
To make CI runs fail if any test modules fail within the isotovideo run one should look for the "result" field in all testresults/result-*.json file like for example this way: ``` jq '.result' testresults/result-*.json | grep -v 'ok' && echo "Test modules failed" && exit 1 ``` Thanks for your feedback and good questions. I will try to clarify that within https://github.com/os-autoinst/os-autoinst-distri-example/ directly Have fun, Oliver