Hi, in a different thread we are discussing the possible frameworks for writing tests. Although it's nice to have some preferred framework and style, we should actually collect our requirements first to see what we need or expect from it. I'd like to start a broader discussion because once we choose the framework and start writing tests then it will hard to go back and use something else... Look at the current existing Yast tests. What do you like? What you don't like? Which features you miss or need? Here are my ideas, please comment, add your expectations, requirements... Testing Framework Requirements ------------------------------ - For programmers: The tests will be written by programmers and used by programmers. Therefore we do not need any human readable language in tests (like in Cucumber [1]). That only adds one level of complexity for developers without much gain. The tests should be easy readable without strange syntax or rules. You should quickly see what is being tested and how from the test code. - Mocking Ruby calls: During tests we need to stub and mock some method calls to avoid touching the real system. We need to provide testing data as a part of tests. We need to mock any Ruby call, the current Yast is able to mock only SCR calls and that's limiting and makes troubles when using other modules. - Unit tests: We do only unit testing [2] in Yast (we test modules/methods separately without any context), so unit test support is mandatory. The question is if we could do more and also use integration testing [3] in the future. Like the installer could setup partitioning and then check bootloader and software proposal. That would probably need some UI support, but in some cases it could be nice. - Development tools integration: Currently we run "make check", the framework should be easily used from make (or rake) from Yast code. - OBS integration: We also run the tests during package build in OBS so we need to package the testing framework as a RPM package. It also needs to work in OBS environment properly (e.g. there is no internet access, no outside connection is possible). - Code coverage: To have good tests we need to know the code coverage to easily find the missing parts. - Sharing framework configuration, testing code, asserts...: It should be easy to share common parts, asserts... to avoid code duplication and make maintenance easier. - Last but not least: it should be well maintained (SLE support!), stable, well documented... (as usual with a good software...) Thanks for your comments! [1] http://cukes.info/ [2] http://en.wikipedia.org/wiki/Unit_testing [3] http://en.wikipedia.org/wiki/Integration_testing -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org