Mailinglist Archive: opensuse-factory (1324 mails)

< Previous Next >
Re: [opensuse-factory] Will we have additional repositories for Leap as we have currently?
  • From: Claudio Freire <klaussfreire@xxxxxxxxx>
  • Date: Mon, 19 Oct 2015 19:02:51 -0300
  • Message-id: <CAGTBQpYafKRDHq+JOu=va-nm5vZ6fsQscz4coC6PE2KNh79LVA@mail.gmail.com>
On Mon, Oct 19, 2015 at 5:59 PM, Richard Brown <RBrownCCB@xxxxxxxxxxxx> wrote:
OpenQA only does
basic testing, you'll have to recognize, some things depend on people
testing, and some use cases get none of that.

openQA does pretty damn advanced testing, and I challenge you to find
something which openQA absolutely cannot test. Multi machine,
multipath, HA, real hardware, you name it, openQA can do it, or we
want to find a way to make it do it.

The biggest limitation is not the technology, but that someone has to
care enough about an issue to write a test for it

That's not that hard to do -
https://github.com/os-autoinst/openQA/blob/master/docs/WritingTests.asciidoc

And there is lots of examples to learn from -
https://github.com/os-autoinst/os-autoinst-distri-opensuse/tree/master/tests

Even when there's the technical possibility of testing, it's not always done.

See what's happening with i586.

Wasting hardware testing hundreds of scenarios on an architecture a
minority, a shrinking minority, of users use isn't an effective use of
hardware..sure

I agree, that's why I say OpenQA can only do limited (I said basic,
but I should have used "limited") testing.

The cost-benefit balance stops being positive after some point.

You really think if someone wrote a few thousand tests covering every
workflow for the top 90% of users, it'd actually be run? There would
probably not be enough resources to do that amount of testing.

We're running a few hundred different scenarios, each including 20-30
tests, so yes, we're already at the few thousand test mark..

And it's already at its limits, which is, as I understand, why i586
tests are being nixed.

And nobody did write that many tests. And no, it's not that easy. I
read OpenQA test scripts and they read like chinese to me, and I
consider myself quite capable of reading code in any language.

https://openqa.opensuse.org/tests/94150/modules/gedit/steps/1/src

Really? it's not that hard..starting at line 10 I'll decode into plain
english (the rest is boilerplate stuff, as described in the docs)

10 - load the X11 application called gedit
11 - check the screen is showing gedit launched
12 - type the string "If you can see this text gedit is working.\n"
13 - wait 2 seconds
14 - check the screen is showing the text
15 - wait 2 seconds
16 - press alt-f4
17 - wait 2 seconds
18 - press alt-w
19 - sleep 2 seconds

Yeah, it's the needles that confuse me really.

I may give it another try, if I find something worth testing that
isn't being tested already.
--
To unsubscribe, e-mail: opensuse-factory+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-factory+owner@xxxxxxxxxxxx

< Previous Next >
Follow Ups