Mailinglist Archive: yast-devel (144 mails)

< Previous Next >
[yast-devel] Requirements for Yast testing framework

Hi,

in a different thread we are discussing the possible frameworks for writing
tests.
Although it's nice to have some preferred framework and style, we should
actually
collect our requirements first to see what we need or expect from it.

I'd like to start a broader discussion because once we choose the framework
and start writing tests then it will hard to go back and use something else...

Look at the current existing Yast tests. What do you like? What you don't like?
Which features you miss or need?


Here are my ideas, please comment, add your expectations, requirements...


Testing Framework Requirements
------------------------------

- For programmers: The tests will be written by programmers and used by
programmers.

Therefore we do not need any human readable language in tests (like in
Cucumber
[1]). That only adds one level of complexity for developers without much gain.

The tests should be easy readable without strange syntax or rules. You
should quickly see what is being tested and how from the test code.

- Mocking Ruby calls: During tests we need to stub and mock some method calls
to avoid touching the real system. We need to provide testing data as
a part of tests.

We need to mock any Ruby call, the current Yast is able to mock only SCR calls
and that's limiting and makes troubles when using other modules.

- Unit tests: We do only unit testing [2] in Yast (we test modules/methods
separately without any context), so unit test support is mandatory.

The question is if we could do more and also use integration testing [3] in
the
future. Like the installer could setup partitioning and then check bootloader
and software proposal. That would probably need some UI support, but in some
cases it could be nice.

- Development tools integration: Currently we run "make check", the framework
should be easily used from make (or rake) from Yast code.

- OBS integration: We also run the tests during package build in OBS so we need
to
package the testing framework as a RPM package. It also needs to work in OBS
environment properly (e.g. there is no internet access, no outside connection
is possible).

- Code coverage: To have good tests we need to know the code coverage to easily
find
the missing parts.

- Sharing framework configuration, testing code, asserts...: It should be easy
to share common parts, asserts... to avoid code duplication and make
maintenance easier.

- Last but not least: it should be well maintained (SLE support!), stable,
well documented... (as usual with a good software...)


Thanks for your comments!


[1] http://cukes.info/
[2] http://en.wikipedia.org/wiki/Unit_testing
[3] http://en.wikipedia.org/wiki/Integration_testing

--

Ladislav Slez√°k
Appliance department / YaST Developer
Lihovarsk√° 1060/12
190 00 Prague 9 / Czech Republic
tel: +420 284 028 960
lslezak@xxxxxxxx
SUSE
--
To unsubscribe, e-mail: yast-devel+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: yast-devel+owner@xxxxxxxxxxxx

< Previous Next >