On Mon, Oct 19, 2015 at 5:59 PM, Richard Brown
OpenQA only does basic testing, you'll have to recognize, some things depend on people testing, and some use cases get none of that.
openQA does pretty damn advanced testing, and I challenge you to find something which openQA absolutely cannot test. Multi machine, multipath, HA, real hardware, you name it, openQA can do it, or we want to find a way to make it do it.
The biggest limitation is not the technology, but that someone has to care enough about an issue to write a test for it
That's not that hard to do - https://github.com/os-autoinst/openQA/blob/master/docs/WritingTests.asciidoc
And there is lots of examples to learn from - https://github.com/os-autoinst/os-autoinst-distri-opensuse/tree/master/tests
Even when there's the technical possibility of testing, it's not always done.
See what's happening with i586.
Wasting hardware testing hundreds of scenarios on an architecture a minority, a shrinking minority, of users use isn't an effective use of hardware..sure
I agree, that's why I say OpenQA can only do limited (I said basic, but I should have used "limited") testing. The cost-benefit balance stops being positive after some point.
You really think if someone wrote a few thousand tests covering every workflow for the top 90% of users, it'd actually be run? There would probably not be enough resources to do that amount of testing.
We're running a few hundred different scenarios, each including 20-30 tests, so yes, we're already at the few thousand test mark..
And it's already at its limits, which is, as I understand, why i586 tests are being nixed.
And nobody did write that many tests. And no, it's not that easy. I read OpenQA test scripts and they read like chinese to me, and I consider myself quite capable of reading code in any language.
https://openqa.opensuse.org/tests/94150/modules/gedit/steps/1/src
Really? it's not that hard..starting at line 10 I'll decode into plain english (the rest is boilerplate stuff, as described in the docs)
10 - load the X11 application called gedit 11 - check the screen is showing gedit launched 12 - type the string "If you can see this text gedit is working.\n" 13 - wait 2 seconds 14 - check the screen is showing the text 15 - wait 2 seconds 16 - press alt-f4 17 - wait 2 seconds 18 - press alt-w 19 - sleep 2 seconds
Yeah, it's the needles that confuse me really. I may give it another try, if I find something worth testing that isn't being tested already. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org