Mailinglist Archive: yast-devel (63 mails)

< Previous Next >
Re: [yast-devel] An idea for integration tests
On Fri, 26 Sep 2014 15:24:13 +0200
Ancor Gonzalez Sosa <ancor@xxxxxxx> wrote:

Since I was tired of the Trello card I spent the whole workshop
working on and being inspired by Flavio's talk, I switched my mind
this morning to the integration tests card.

I had an idea for writing tests with much less stubbing (using a
virtual system) than I think is worth trying.

Explained (together with some braindumping) at
https://public.pad.fsfe.org/p/yast-integration-tests

Cheers.


Hi,
I answer to etherpad document here as I have more highlevel notes:

1) It does not make sense to me to separate two layers for integration
testing. Integration testing is whole stack test so if click on button
on top level do what is expected. Integration testing usually have
almost none mocking or testing on bariers ( where barier is parts out
of your control, which you cannot affect ).

2) Your proposal with scr_server is almost exactly what old testsuite
did - testing on SCR level. It have several significant drawbacks:

2a) It do not test whole stack. If there is bug in agent, you do not
catch it

2b) It also do not test integration with whole system, so if some
command stop working, like removing option or splitting package, or
if configuration file change syntax, so won't notice it.

2c) It is a lot of work and hard to maintain. Consider how often we
break our old testsuite just before we add some new SCR calls.

3) docker is for me not solution, as in docker do not run systemd and
almost all modules somehow manipulate with services. So I think only
way is full virtualization.


No let me comment your requirements:

rollbacks

yes, it make sense. When I play in past with cloud and kvm, it have
snapshotting ability

"inject" yast module with dependencies

I will be more strict here. Install yast package. It ensure that
installation works for package, that it have all required packages and
also test way which probably user use.

different initial systems

yes, make sense. Question is if we need it prepared or if we start with
same initial system and convert to target state like customer should do
it.

temporary system and catch output

Yes, it make sense for me.


What other requirements should be there

1) parallel run
integration tests are quite slow and we need to speed up as much as
possible

2) visible output
Common problem is that integration tests run over night, so next day
everyone should see what is broken and that we need to fix it.

3) easy debugging
If something break in over-night run, we need to have way how to get
what is broken

3) reduce fragility
Common problem of integration testing is fragility, as tests are often
broken. So we need easy way how to fix it or how to make it more robust



Few ideas I have about topic:

- using VNC library to work with UI -
https://code.google.com/p/ruby-vnc/

- use OCR to localize buttons with text and have pointers where to click

- use cloud for parallel build with its snapshotting ability

- create tree if requirements and how to get into such state like


installation with partitioning A -> snapshot A -> install + run module M1 in
scenario A
-> install + run module M2 in
scenario A

installation with partitioning B -> snapshot B -> install + run module M1 in
scenario B
-> install + run module M2 in
scenario B

installation with partitioning C -> snapshot C -> install + run module M1 in
scenario B


For quick install we can use autoyast and before we run test we should verify
that it is really expected state

- For released project use already known snapshot, for product before release
use latest succesful snapshot


Josef
--
To unsubscribe, e-mail: yast-devel+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: yast-devel+owner@xxxxxxxxxxxx

< Previous Next >
Follow Ups
References