On 09/29/2014 09:02 AM, Josef Reidinger wrote:
On Fri, 26 Sep 2014 15:24:13 +0200
Ancor Gonzalez Sosa <ancor(a)suse.de> wrote:
Since I was tired of the Trello card I spent the
working on and being inspired by Flavio's talk, I switched my mind
this morning to the integration tests card.
I had an idea for writing tests with much less stubbing (using a
virtual system) than I think is worth trying.
Explained (together with some braindumping) at
I answer to etherpad document here as I have more highlevel notes:
1) It does not make sense to me to separate two layers for integration
testing. Integration testing is whole stack test so if click on button
on top level do what is expected. Integration testing usually have
almost none mocking or testing on bariers ( where barier is parts out
of your control, which you cannot affect ).
My fault. I made an error in the title of the mail. I should have been
"an idea for functional tests" (I was sure I used "functional" until
read your reply). That's why the etherpad is titled "integration and
functional tests for Yast". I'm focusing first in the later because I'm
afraid there is no lightweight solution for the former.
2) Your proposal with scr_server is almost exactly
what old testsuite
did - testing on SCR level. It have several significant drawbacks:
2a) It do not test whole stack. If there is bug in agent, you do not
Not all, but will help to catch many of them, I think.
2b) It also do not test integration with whole
system, so if some
command stop working, like removing option or splitting package, or
if configuration file change syntax, so won't notice it.
As I said. The solution is not really for integration tests (wrong
mail's subject). Even though, I'm not sure if I get your point here. The
goal is not to test that a given SCR command was called (that's what we
already do with the current unit tests), but to check the status of the
target system. Nothing stops us to easily check if a service is running,
if a given file is there, etc.
2c) It is a lot of work and hard to maintain.
Consider how often we
break our old testsuite just before we add some new SCR calls.
I agree. Nevertheless, My impression was that agents do not change so
much nowadays. In any case, there are plenty of tools to automate
creation of VMs/containers out there.
3) docker is for me not solution, as in docker do not
run systemd and
almost all modules somehow manipulate with services. So I think only
way is full virtualization.
I was not aware of that drawback. I though it was possible to run
something "closer" to a full system inside a container. Well, time for a
closer look to Pennyworth then.
No let me comment your requirements:
yes, it make sense. When I play
in past with cloud and kvm, it have
"inject" yast module with dependencies
I will be more strict here. Install yast package. It ensure that
installation works for package, that it have all required packages and
also test way which probably user use.
different initial systems
yes, make sense.
Question is if we need it prepared or if we start with
same initial system and convert to target state like customer should do
I was wondering the same. We probably should have some hybrid system.
Some "recipes" to create the systems with some mechanisms to reuse the
results as long as they are reusable, to reduce the overhead.
temporary system and catch output
make sense for me.
What other requirements should be there
1) parallel run
integration tests are quite slow and we need to speed up as much as
2) visible output
Common problem is that integration tests run over night, so next day
everyone should see what is broken and that we need to fix it.
3) easy debugging
If something break in over-night run, we need to have way how to get
what is broken
3) reduce fragility
Common problem of integration testing is fragility, as tests are often
broken. So we need easy way how to fix it or how to make it more robust
Few ideas I have about topic:
- using VNC library to work with UI -
That's what openQA does nowadays.
- use OCR to localize buttons with text and have
pointers where to click
openQA have some preliminary support for this as well. But we found out
that the current image matching mechanisms does a good-enough job
(sometimes even better that OCR) in 99% of situations. We have not
invested much effort in OCR since then.
- use cloud for parallel build with its snapshotting
Sounds smart. openQA currently uses kvm for snapshots and we are trying
to get the cloud team involved to achieve distribution capabilities (all
workers must be in the same machine right now).
- create tree if requirements and how to get into such
installation with partitioning A -> snapshot A -> install + run module M1 in
-> install + run module M2 in
installation with partitioning B -> snapshot B -> install + run module M1 in
-> install + run module M2 in
installation with partitioning C -> snapshot C -> install + run module M1 in
For quick install we can use autoyast and before we run test we should verify that it
is really expected state
Sounds like ideas that has been considered for openQA (not implemented
because the lack of manpower).
- For released project use already known snapshot, for
product before release use latest succesful snapshot
As said before, I was focusing more in lightweight solutions for
functional tests. Your plans for comprehensive full integration tests
that runs overnight and test the installation process overlaps too much
with openQA, IMHO. I know that openQA sucks in many things (tests
written in dirty Perl not being the minor one), but a very high
percentage of your idea sounds like re-inventing a better openQA to me.
Ancor González Sosa
openSUSE Team at SUSE Linux GmbH
To unsubscribe, e-mail: yast-devel+unsubscribe(a)opensuse.org
To contact the owner, e-mail: yast-devel+owner(a)opensuse.org