Mailinglist Archive: yast-devel (144 mails)

< Previous Next >
Re: [yast-devel] Requirements for Yast testing framework
Hi,

Yesterday I upgraded my RVM rubies (Ruby version manager, http://rvm.io/
) and got a bit surpriced after the 2.0.0-head has been installed and I
ran `gem list`: minitest was installed by default.

After that I have been googling a bit and here is the summary:
* minitest became part of ruby stdlib in version 1.9.2 [1]
* minitest kind of replaced some internals of test/unit lib
* minitest supports writing of traditional unit tests and also of
'specs' (Rspec DSL style). This would allow people to write their test
in prefered way and still to use single library [1]
* minitest beats rspec on complexity, some statistics are available here [2]
* minitest supports mocking and benchmarking
* minitest developer is a Ryan Davis aka zenspider, member of seattlerb
(Seattle Ruby Brigade), perhaps the most active ruby group worldwide [3]


The implications for us thinking about the right choice of testing
library for yast are: (please look below)


Dne 12.8.2013 16:04, Ladislav Slezak napsal(a):

Hi,


...

Look at
the current existing Yast tests. What do you like? What you don't like?
Which features you miss or need?


Here are my ideas, please comment, add your expectations, requirements...


Testing Framework Requirements
------------------------------

- For programmers: The tests will be written by programmers and used by
programmers.

Therefore we do not need any human readable language in tests (like in
Cucumber
[1]). That only adds one level of complexity for developers without much
gain.

The tests should be easy readable without strange syntax or rules. You
should quickly see what is being tested and how from the test code.
check

- Mocking Ruby calls: During tests we need to stub and mock some method calls
to avoid touching the real system. We need to provide testing data as
a part of tests.

We need to mock any Ruby call, the current Yast is able to mock only SCR
calls
and that's limiting and makes troubles when using other modules.
check

- Unit tests: We do only unit testing [2] in Yast (we test modules/methods
separately without any context), so unit test support is mandatory.

check
The question is if we could do more and also use integration testing [3] in
the
future. Like the installer could setup partitioning and then check
bootloader
and software proposal. That would probably need some UI support, but in some
cases it could be nice.

not sure about this
- Development tools integration: Currently we run "make check", the framework
should be easily used from make (or rake) from Yast code.

check if we consider using Rake
- OBS integration: We also run the tests during package build in OBS so we
need to
package the testing framework as a RPM package. It also needs to work in OBS
environment properly (e.g. there is no internet access, no outside
connection
is possible).
check as minitest is in ruby the default now

- Code coverage: To have good tests we need to know the code coverage to
easily find
the missing parts.
simplecov as the ruby standard coverage gem supports minitest

- Sharing framework configuration, testing code, asserts...: It should be easy
to share common parts, asserts... to avoid code duplication and make
maintenance easier.
I think this shouldn't be a problem with using a single library

- Last but not least: it should be well maintained (SLE support!), stable,
well documented... (as usual with a good software...)
check as mentioned above


[1] http://www.mikeperham.com/2012/09/25/minitest-ruby-1-9s-test-framework/
[2] https://gist.github.com/rjackson/5550653#file-output-txt
[3] http://www.seattlerb.org/

Vlado

< Previous Next >
References