[opensuse-factory] Announcement: Run openQA tests with isotovideo in your CI pipelines
To all users of openQA or anyone interested in running system-level tests in CI jobs, I would like to make you aware of a new approach for running system-level tests using "isotovideo", the main backend of openQA tests, within continuous integration (CI) services, for example github actions, travis CI, gitlab CI, etc. Following an updated section https://github.com/os-autoinst/os-autoinst#getting-started it is possible to run "openQA tests" using a simple, convenient container based on tests that can be provided for example in a git repository. An easy example is described in https://github.com/os-autoinst/os-autoinst-distri-example#local-testing-and-... Find the latest example CI run directly following https://github.com/os-autoinst/os-autoinst-distri-example/actions I consider the above approach very suitable if one wants to run system level tests following the "openQA test API" in any project where direct feedback from CI jobs is appreciated but only a limited number of test scenarios are needed, no heavy visual tests are run where the openQA web UI would provide a more convenient interface, or when wanting to test changes to already existing openQA tests but in pull requests before they are merged. For more elaborate examples a complete openQA setup is more suitable. If you want to read more about openQA in general visit http://open.qa/ There is also a single HTML file document http://open.qa/docs/ as well as a PDF version http://open.qa/docs/current.pdf Have fun, Oliver
On Fri, 2020-11-27 at 20:52 +0100, Oliver Kurz wrote:
To all users of openQA or anyone interested in running system-level tests in CI jobs,
I would like to make you aware of a new approach for running system- level tests using "isotovideo", the main backend of openQA tests, within continuous integration (CI) services, for example github actions, travis CI, gitlab CI, etc.
Following an updated section https://github.com/os-autoinst/os-autoinst#getting-started it is possible to run "openQA tests" using a simple, convenient container based on tests that can be provided for example in a git repository.
An easy example is described in https://github.com/os-autoinst/os-autoinst-distri-example#local-testing-and-...
Find the latest example CI run directly following https://github.com/os-autoinst/os-autoinst-distri-example/actions
Unless I'm missing something essential, the test step "Run isotovideo against test code in happy-path scenario" returned 0, although the test failed. [2020-11-26T15:08:23.011 UTC] [debug] >>> testapi::_check_backend_response: match=bootloader timed out after 30 (assert_screen) [2020-11-26T15:08:23.092 UTC] [info] ::: basetest::runtest: # Test died: no candidate needle with tag(s) 'bootloader' matched ... 1: EXIT 0 I suppose the test fails because it's "one simple test where you need to create the needles for". Forgive my ignorance - how would I do that? Can I fetch them from somewhere? More importantly, how would I set up this test so that it actually fails the CI when it the test fails? Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Software Solutions Germany GmbH HRB 36809, AG Nürnberg GF: Felix Imendörffer
Hi Martin, please forgive my late answer. Better late than never, eh? ;) On Wednesday, 2 December 2020 21.44.21 CET Martin Wilck wrote:
On Fri, 2020-11-27 at 20:52 +0100, Oliver Kurz wrote:
To all users of openQA or anyone interested in running system-level tests in CI jobs,
I would like to make you aware of a new approach for running system- level tests using "isotovideo", the main backend of openQA tests, within continuous integration (CI) services, for example github actions, travis CI, gitlab CI, etc.
Following an updated section https://github.com/os-autoinst/os-autoinst#getting-started it is possible to run "openQA tests" using a simple, convenient container based on tests that can be provided for example in a git repository.
An easy example is described in https://github.com/os-autoinst/os-autoinst-distri-example#local-testing-an d-ci-environment
Find the latest example CI run directly following https://github.com/os-autoinst/os-autoinst-distri-example/actions
Unless I'm missing something essential, the test step "Run isotovideo against test code in happy-path scenario" returned 0, although the test failed.
[2020-11-26T15:08:23.011 UTC] [debug] >>> testapi::_check_backend_response: match=bootloader timed out after 30 (assert_screen) [2020-11-26T15:08:23.092 UTC] [info] ::: basetest::runtest: # Test died: no candidate needle with tag(s) 'bootloader' matched ... 1: EXIT 0
Yes, that is correct. This is discussed in https://github.com/os-autoinst/os-autoinst-distri-example/issues/6 "EXIT 0" means that isotovideo could run the complete test from beginning to end. Various test steps within could have still resulted in "failed".
I suppose the test fails because it's "one simple test where you need to create the needles for". Forgive my ignorance - how would I do that? Can I fetch them from somewhere?
In this example repository there are no needles because this is left as a learning exercise to new users as described in https://github.com/os-autoinst/os-autoinst-distri-example#local-testing-and-...
More importantly, how would I set up this test so that it actually fails the CI when it the test fails?
To make CI runs fail if any test modules fail within the isotovideo run one should look for the "result" field in all testresults/result-*.json file like for example this way: ``` jq '.result' testresults/result-*.json | grep -v 'ok' && echo "Test modules failed" && exit 1 ``` Thanks for your feedback and good questions. I will try to clarify that within https://github.com/os-autoinst/os-autoinst-distri-example/ directly Have fun, Oliver
participants (2)
-
Martin Wilck -
Oliver Kurz