OpenQA only does basic testing, you'll have to recognize, some things depend on people testing, and some use cases get none of that.
openQA does pretty damn advanced testing, and I challenge you to find something which openQA absolutely cannot test. Multi machine, multipath, HA, real hardware, you name it, openQA can do it, or we want to find a way to make it do it.
The biggest limitation is not the technology, but that someone has to care enough about an issue to write a test for it
That's not that hard to do - https://github.com/os-autoinst/openQA/blob/master/docs/WritingTests.asciidoc
And there is lots of examples to learn from - https://github.com/os-autoinst/os-autoinst-distri-opensuse/tree/master/tests
Even when there's the technical possibility of testing, it's not always done.
See what's happening with i586.
Wasting hardware testing hundreds of scenarios on an architecture a minority, a shrinking minority, of users use isn't an effective use of hardware..sure But we still are going to keep an i586 'canary' or two around to keep some attention on that old platform. If people would like to donate more testing hardware, they can contact donations@opensuse.org, we'll gladly take it and integrate it into the openQA system.
You really think if someone wrote a few thousand tests covering every workflow for the top 90% of users, it'd actually be run? There would probably not be enough resources to do that amount of testing.
We're running a few hundred different scenarios, each including 20-30 tests, so yes, we're already at the few thousand test mark..
And nobody did write that many tests. And no, it's not that easy. I read OpenQA test scripts and they read like chinese to me, and I consider myself quite capable of reading code in any language.
https://openqa.opensuse.org/tests/94150/modules/gedit/steps/1/src Really? it's not that hard..starting at line 10 I'll decode into plain english (the rest is boilerplate stuff, as described in the docs) 10 - load the X11 application called gedit 11 - check the screen is showing gedit launched 12 - type the string "If you can see this text gedit is working.\n" 13 - wait 2 seconds 14 - check the screen is showing the text 15 - wait 2 seconds 16 - press alt-f4 17 - wait 2 seconds 18 - press alt-w 19 - sleep 2 seconds Even the most complicated tests all share a common heritage with simple ones like this example..just as we want to delve a little deeper, things sometimes get a little bit more fanciful..but that's the same with all programming :) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org