[yast-devel] Requirements for Yast testing framework
Hi, in a different thread we are discussing the possible frameworks for writing tests. Although it's nice to have some preferred framework and style, we should actually collect our requirements first to see what we need or expect from it. I'd like to start a broader discussion because once we choose the framework and start writing tests then it will hard to go back and use something else... Look at the current existing Yast tests. What do you like? What you don't like? Which features you miss or need? Here are my ideas, please comment, add your expectations, requirements... Testing Framework Requirements ------------------------------ - For programmers: The tests will be written by programmers and used by programmers. Therefore we do not need any human readable language in tests (like in Cucumber [1]). That only adds one level of complexity for developers without much gain. The tests should be easy readable without strange syntax or rules. You should quickly see what is being tested and how from the test code. - Mocking Ruby calls: During tests we need to stub and mock some method calls to avoid touching the real system. We need to provide testing data as a part of tests. We need to mock any Ruby call, the current Yast is able to mock only SCR calls and that's limiting and makes troubles when using other modules. - Unit tests: We do only unit testing [2] in Yast (we test modules/methods separately without any context), so unit test support is mandatory. The question is if we could do more and also use integration testing [3] in the future. Like the installer could setup partitioning and then check bootloader and software proposal. That would probably need some UI support, but in some cases it could be nice. - Development tools integration: Currently we run "make check", the framework should be easily used from make (or rake) from Yast code. - OBS integration: We also run the tests during package build in OBS so we need to package the testing framework as a RPM package. It also needs to work in OBS environment properly (e.g. there is no internet access, no outside connection is possible). - Code coverage: To have good tests we need to know the code coverage to easily find the missing parts. - Sharing framework configuration, testing code, asserts...: It should be easy to share common parts, asserts... to avoid code duplication and make maintenance easier. - Last but not least: it should be well maintained (SLE support!), stable, well documented... (as usual with a good software...) Thanks for your comments! [1] http://cukes.info/ [2] http://en.wikipedia.org/wiki/Unit_testing [3] http://en.wikipedia.org/wiki/Integration_testing -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 12.8.2013 16:04, Ladislav Slezak napsal(a):
- Last but not least: it should be well maintained (SLE support!), stable,
In this case "stable" is rather meant for the API. It should be possible to use the same tests in openSUSE and SLE after updating the framework in openSUSE. -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 08/12/2013 04:04 PM, Ladislav Slezak wrote:
- Unit tests: We do only unit testing [2] in Yast (we test modules/methods separately without any context), so unit test support is mandatory.
The question is if we could do more and also use integration testing [3] in the future. Like the installer could setup partitioning and then check bootloader and software proposal. That would probably need some UI support, but in some cases it could be nice.
Unit testing is definitely a must have. Continuous integration testing should be required for bigger projects, e.g., New Installer or AutoYast. I've been already working on such proposal. Bye Lukas -- Lukas Ocilka, Cloud & Systems Management Department SUSE LINUX s.r.o., Praha -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Hi, Yesterday I upgraded my RVM rubies (Ruby version manager, http://rvm.io/ ) and got a bit surpriced after the 2.0.0-head has been installed and I ran `gem list`: minitest was installed by default. After that I have been googling a bit and here is the summary: * minitest became part of ruby stdlib in version 1.9.2 [1] * minitest kind of replaced some internals of test/unit lib * minitest supports writing of traditional unit tests and also of 'specs' (Rspec DSL style). This would allow people to write their test in prefered way and still to use single library [1] * minitest beats rspec on complexity, some statistics are available here [2] * minitest supports mocking and benchmarking * minitest developer is a Ryan Davis aka zenspider, member of seattlerb (Seattle Ruby Brigade), perhaps the most active ruby group worldwide [3] The implications for us thinking about the right choice of testing library for yast are: (please look below) Dne 12.8.2013 16:04, Ladislav Slezak napsal(a):
Hi,
Look at
... the current existing Yast tests. What do you like? What you don't like?
Which features you miss or need?
Here are my ideas, please comment, add your expectations, requirements...
Testing Framework Requirements ------------------------------
- For programmers: The tests will be written by programmers and used by programmers.
Therefore we do not need any human readable language in tests (like in Cucumber [1]). That only adds one level of complexity for developers without much gain.
The tests should be easy readable without strange syntax or rules. You should quickly see what is being tested and how from the test code. check
- Mocking Ruby calls: During tests we need to stub and mock some method calls to avoid touching the real system. We need to provide testing data as a part of tests.
We need to mock any Ruby call, the current Yast is able to mock only SCR calls and that's limiting and makes troubles when using other modules. check
- Unit tests: We do only unit testing [2] in Yast (we test modules/methods separately without any context), so unit test support is mandatory.
check
The question is if we could do more and also use integration testing [3] in the future. Like the installer could setup partitioning and then check bootloader and software proposal. That would probably need some UI support, but in some cases it could be nice.
not sure about this
- Development tools integration: Currently we run "make check", the framework should be easily used from make (or rake) from Yast code.
check if we consider using Rake
- OBS integration: We also run the tests during package build in OBS so we need to package the testing framework as a RPM package. It also needs to work in OBS environment properly (e.g. there is no internet access, no outside connection is possible). check as minitest is in ruby the default now
- Code coverage: To have good tests we need to know the code coverage to easily find the missing parts. simplecov as the ruby standard coverage gem supports minitest
- Sharing framework configuration, testing code, asserts...: It should be easy to share common parts, asserts... to avoid code duplication and make maintenance easier. I think this shouldn't be a problem with using a single library
- Last but not least: it should be well maintained (SLE support!), stable, well documented... (as usual with a good software...) check as mentioned above
[1] http://www.mikeperham.com/2012/09/25/minitest-ruby-1-9s-test-framework/ [2] https://gist.github.com/rjackson/5550653#file-output-txt [3] http://www.seattlerb.org/ Vlado
# vmoravec@suse.com / 2013-08-14 10:06:54 +0200:
* minitest became part of ruby stdlib in version 1.9.2 [1] * minitest kind of replaced some internals of test/unit lib * minitest supports writing of traditional unit tests and also of 'specs' (Rspec DSL style). This would allow people to write their test in prefered way and still to use single library [1] * minitest beats rspec on complexity, some statistics are available here [2] * minitest supports mocking and benchmarking * minitest developer is a Ryan Davis aka zenspider, member of seattlerb (Seattle Ruby Brigade), perhaps the most active ruby group worldwide [3]
looks like we have a winner to me.
The question is if we could do more and also use integration testing [3] in the future. Like the installer could setup partitioning and then check bootloader and software proposal. That would probably need some UI support, but in some cases it could be nice.
not sure about this
if it can drive code, it can drive code. integration tests are not that different from unit tests. or am i missing something?
- Development tools integration: Currently we run "make check", the framework should be easily used from make (or rake) from Yast code.
check if we consider using Rake
does that mean it's hard to use outside of Rake?
[1] http://www.mikeperham.com/2012/09/25/minitest-ruby-1-9s-test-framework/ [2] https://gist.github.com/rjackson/5550653#file-output-txt [3] http://www.seattlerb.org/
-- roman -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 14.8.2013 11:22, Roman Neuhauser napsal(a):
# vmoravec@suse.com / 2013-08-14 10:06:54 +0200: ...
- Development tools integration: Currently we run "make check", the framework should be easily used from make (or rake) from Yast code.
check if we consider using Rake
does that mean it's hard to use outside of Rake?
As far as I can say from reading the docs and code, it's designed for ease of use and it does not expect rake to be used at all. Even the rake built-in Rake::TestTask designed for running tests is not recommended due to some pitfalls, for more look here [1]. It has so called autorun capability which runs all available tests if you have "require 'ministest/autorun" on top of the test_helper file. To run a single test file it's enough to put "ruby -r 'minitest/autorun' simple_spec.rb" into your terminal. For more just follow the link below. [1] http://crashruby.com/2013/05/10/running-a-minitest-suite/ vlado
On Wed, 14 Aug 2013 11:22:21 +0200 Roman Neuhauser <rneuhauser@suse.cz> wrote:
# vmoravec@suse.com / 2013-08-14 10:06:54 +0200:
* minitest became part of ruby stdlib in version 1.9.2 [1] * minitest kind of replaced some internals of test/unit lib * minitest supports writing of traditional unit tests and also of 'specs' (Rspec DSL style). This would allow people to write their test in prefered way and still to use single library [1] * minitest beats rspec on complexity, some statistics are available here [2] * minitest supports mocking and benchmarking * minitest developer is a Ryan Davis aka zenspider, member of seattlerb (Seattle Ruby Brigade), perhaps the most active ruby group worldwide [3]
looks like we have a winner to me.
The question is if we could do more and also use integration testing [3] in the future. Like the installer could setup partitioning and then check bootloader and software proposal. That would probably need some UI support, but in some cases it could be nice.
not sure about this
if it can drive code, it can drive code. integration tests are not that different from unit tests. or am i missing something?
There is one big difference. Good unit test have minimum of mocking. But integration test for yast means increadible bunch of mocking as you usually don't want to break testing machine. For me integration testing is better in simulated environment and I think it is better to use completelly different tool like openqa or cucumber in virtualized environment.
- Development tools integration: Currently we run "make check", the framework should be easily used from make (or rake) from Yast code.
check if we consider using Rake
does that mean it's hard to use outside of Rake?
Actually for me it is almost must have ability to run one test or at least one test file as usually testing take some time and when I detecting problem I want isolated run of test. But almost all frameworks support it, but sometimes it needs small help. Josef
[1] http://www.mikeperham.com/2012/09/25/minitest-ruby-1-9s-test-framework/ [2] https://gist.github.com/rjackson/5550653#file-output-txt [3] http://www.seattlerb.org/
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
# jreidinger@suse.cz / 2013-08-16 11:44:39 +0200:
On Wed, 14 Aug 2013 11:22:21 +0200 Roman Neuhauser <rneuhauser@suse.cz> wrote:
# vmoravec@suse.com / 2013-08-14 10:06:54 +0200:
The question is if we could do more and also use integration testing [3] in the future. Like the installer could setup partitioning and then check bootloader and software proposal. That would probably need some UI support, but in some cases it could be nice.
not sure about this
if it can drive code, it can drive code. integration tests are not that different from unit tests. or am i missing something?
There is one big difference. Good unit test have minimum of mocking. But integration test for yast means increadible bunch of mocking as you usually don't want to break testing machine.
I'd say both contexts require this capability, period. See eg. yast-installation/src/modules/SystemFilesCopy.rb. OTOH...
For me integration testing is better in simulated environment and I think it is better to use completelly different tool like openqa or cucumber in virtualized environment.
... yeah, this is a different kind and scale of "mocking".
Actually for me it is almost must have ability to run one test or at least one test file as usually testing take some time and when I detecting problem I want isolated run of test. But almost all frameworks support it, but sometimes it needs small help.
Agreed, the ability to choose a subset of tests to run on the command line is important. -- roman -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 14.8.2013 10:06, Vladimir Moravec napsal(a):
Hi,
Yesterday I upgraded my RVM rubies (Ruby version manager, http://rvm.io/ ) and got a bit surpriced after the 2.0.0-head has been installed and I ran `gem list`: minitest was installed by default.
I did some more research regarding to minitest and found this: The official documentation: http://www.ruby-doc.org/stdlib-2.0/libdoc/minitest/rdoc/index.html A nice overview with both rspec-like and testunit-like examples, including mocking: http://www.mattsears.com/articles/2011/12/10/minitest-quick-reference Mocking examples: http://stackoverflow.com/questions/10465350/ruby-mocking-a-class-method-with... http://tatey.com/2012/02/07/mocking-with-minitest-mock-and-simple-delegator/ http://stackoverflow.com/questions/7211086/how-do-i-stub-things-in-minitest The mocking API does not look as powerfull as in 'mocha' gem[1], but should be enough I guess. If it would not be sufficient then using mocha is possible, it supports both minitest and testunit. [1] https://github.com/freerange/mocha -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
# lslezak@suse.cz / 2013-08-15 11:24:53 +0200:
I did some more research regarding to minitest and found this:
The official documentation: http://www.ruby-doc.org/stdlib-2.0/libdoc/minitest/rdoc/index.html
A nice overview with both rspec-like and testunit-like examples, including mocking: http://www.mattsears.com/articles/2011/12/10/minitest-quick-reference
Mocking examples: http://stackoverflow.com/questions/10465350/ruby-mocking-a-class-method-with... http://tatey.com/2012/02/07/mocking-with-minitest-mock-and-simple-delegator/ http://stackoverflow.com/questions/7211086/how-do-i-stub-things-in-minitest
The mocking API does not look as powerfull as in 'mocha' gem[1], but should be enough I guess. If it would not be sufficient then using mocha is possible, it supports both minitest and testunit.
i just looked at the minitest code[1], and it can apparently verify calls including argument values. you can't say "expect a call to #foo with any arguments" or "expect any number of calls to #bar", but i'd say that's a good thing, at least in the long run. [1] https://github.com/seattlerb/minitest/blob/master/lib/minitest/mock.rb -- roman -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 12.8.2013 16:04, Ladislav Slezak napsal(a):
in a different thread we are discussing the possible frameworks for writing tests. Although it's nice to have some preferred framework and style, we should actually collect our requirements first to see what we need or expect from it.
[...]
I think there is one decision missing which would translate into a requirement: What style do we want to write the tests in? In Ruby world, there are currently two dominant styles: * Test::Unit-like * RSpec-like The most important distinction is that Test::Unit-like frameworks use regular classes and methods to write tests, RSpec-like frameworks use structured blocks. I think we should choose one style, as mixing both in one codebase would only lead to confusion and inconsistency. I also think that we should choose RSpec-like style, as it leads to better structured and documented tests. In the next paragraphs, I will assume that RSpec-like style is the preferred one and thus its support is a requirement. If someone disagrees with this assumption, please say so. Now, a second requirement I feel is missing becomes important: If possible, use what other Ruby developers use. This is extremely important. Choosing a widely used framework means it will be more stable, supported, documented, etc. than the other alternatives. There will be a large body of knowledge available when we get into trouble. Its development will also probably progress faster. Most importantly of all, using a widely used framework will make it easy for outside contributors to hack YaST becaue they won't need to learn a new thing (most likely). I think this is in line with the philosophy which was also behind the Ruby translation. ----- If one looks at the existing frameworks, it's clear that for RSpec-like style the main contenders are minitest and RSpec. For minitest, I see one advantage -- it is being bundled with Ruby. While it is used by many projects, I have yet to see one where its RSpec-like way of writing tests is used. To me, it looks like people treat it mostly as a replacement of Test::Unit. This is a disadvantage. RSpec seems to be the default choice in the Ruby world at the moment and for me, it is a clear winner in the "use what other Ruby developers use" category. It also offers *only* RSpec-like tests, which means they are treated in a first-class way. It is not just one of several offered options like in case of minitest. I see that as a an advantage (would you rather buy your bike in Tesco, or in a specialized bike shop?). My conclusion is: Both minitest and RSpec seem good choices, but for the reasons stated above, I think RSpec is a better one. ----- BTW, we should get to a firm decision soon, as the testing framework decision blocks new development. If the question isn't resolved here soon, it will get resolved "organically" (= everyone doing what he wants). I don't think anybody wants that. -- David Majda SUSE Studio developer http://susestudio.com/ -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 08/26/2013 10:57 AM, David Majda wrote:
In Ruby world, there are currently two dominant styles:
* Test::Unit-like * RSpec-like
OK, as there was no decision so far and the discussion has apparently died although it's blocking the development, *I suggest to use RSpec* for all the new tests. There are just a few test in, e.g., Services Manager that might need partial rewriting because they use Test::Unit but that's it. Is anybody against this suggestion? Please write also if you agree with it. Thanks in advance Lukas -- Lukas Ocilka, Cloud & Systems Management Department SUSE LINUX s.r.o., Praha -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Wed, Sep 11, 2013 at 04:02:57PM +0200, Lukas Ocilka wrote:
OK, as there was no decision so far and the discussion has apparently died although it's blocking the development, *I suggest to use RSpec* for all the new tests. There are just a few test in, e.g., Services Manager that might need partial rewriting because they use Test::Unit but that's it.
Is anybody against this suggestion? Please write also if you agree with it.
Although I have little actual experience (I only have a rather messy Test::Unit suite in ruby-dbus), I am for: RSpec From David's arguments, the points important to me are - superior ability to structure the specs - wide acceptance in the community -- Martin Vidner, Cloud & Systems Management Team http://en.opensuse.org/User:Mvidner Kuracke oddeleni v restauraci je jako fekalni oddeleni v bazenu
participants (7)
-
David Majda -
Josef Reidinger -
Ladislav Slezak -
Lukas Ocilka -
Martin Vidner -
Roman Neuhauser -
Vladimir Moravec