[yast-devel] unit tests for yast modules written in ruby
hola! vlado brought up an interesting topic, namely the choice of a unit-testing framework. there's a bunch of them written for ruby code, some of the most popular ones (presented in no particular order) are: * minitest http://rubydoc.info/gems/minitest/frames * rspec http://rspec.info/ * shoulda http://rubydoc.info/gems/shoulda/frames * Test::Unit http://www.ruby-doc.org/stdlib-1.9.3/libdoc/test/unit/rdoc/Test/Unit.html see https://en.wikipedia.org/wiki/Unit_testing_frameworks_for_Ruby for a pretty good overview. i have no strong preference or dislike for any of them, with the exception of rspec, which creates methods on the CUT (to be clear: i don't like that). but before we dive into a massive flamewar re: which one is the best, there's another topic to chew on: do we need to standardize this at all? as far as i understand the situation, yast modules are mostly silos with one or two people ever hacking on any given module. this arrangement would lend weight to the argument that no, we don't need to standardize on a single unit-testing framework, each maintainer is free to choose whatever they like most. OTOH, strong code-ownership has its downsides, so maybe, unless we want to make the segregation of interests stronger, we should not move the code further apart. a standard test framework would make it easier to move code between modules, eg. utility functions initially written for a specific yast module getting promoted to ruby-bindings or something. opinions? -- roman -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
# rneuhauser@suse.cz / 2013-08-08 17:37:51 +0200:
i have no strong preference or dislike for any of them, with the exception of rspec, which creates methods on the CUT (to be clear: i don't like that).
i see in archives recent discussion of unit tests: http://lists.opensuse.org/yast-devel/2013-08/msg00000.html http://lists.opensuse.org/yast-devel/2013-08/msg00014.html vlado points out that Rspec is moving away from clobbering the CUT, which neuters my complaint. -- roman -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Fri, Aug 09, 2013 at 07:34:20PM +0200, Roman Neuhauser wrote:
# rneuhauser@suse.cz / 2013-08-08 17:37:51 +0200:
i have no strong preference or dislike for any of them, with the exception of rspec, which creates methods on the CUT (to be clear: i don't like that).
i see in archives recent discussion of unit tests:
http://lists.opensuse.org/yast-devel/2013-08/msg00000.html http://lists.opensuse.org/yast-devel/2013-08/msg00014.html
vlado points out that Rspec is moving away from clobbering the CUT, which neuters my complaint.
What is CUT? ciao Arvin -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
# aschnell@suse.de / 2013-08-09 19:38:43 +0200:
On Fri, Aug 09, 2013 at 07:34:20PM +0200, Roman Neuhauser wrote:
# rneuhauser@suse.cz / 2013-08-08 17:37:51 +0200: vlado points out that Rspec is moving away from clobbering the CUT, which neuters my complaint.
What is CUT?
Code Under Test, this is a standard TLA (Three-Letter Acronym) in the testing context. You may also see SUT (System Under Test). -- roman -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 8.8.2013 17:37, Roman Neuhauser napsal(a):
hola!
vlado brought up an interesting topic, namely the choice of a unit-testing framework. there's a bunch of them written for ruby code, some of the most popular ones (presented in no particular order) are:
* minitest http://rubydoc.info/gems/minitest/frames * rspec http://rspec.info/ * shoulda http://rubydoc.info/gems/shoulda/frames * Test::Unit http://www.ruby-doc.org/stdlib-1.9.3/libdoc/test/unit/rdoc/Test/Unit.html
see https://en.wikipedia.org/wiki/Unit_testing_frameworks_for_Ruby for a pretty good overview.
i have no strong preference or dislike for any of them, with the exception of rspec, which creates methods on the CUT (to be clear: i don't like that).
I do not have any strong preference, I only know the test-unit + mocha which we already use in Webyast and SLMS. (SLMS in addition to this uses factory_girl, but Yast does not need any DB mocking so it's irrelevant here...)
but before we dive into a massive flamewar re: which one is the best,
I don't think there will be a flamewar as Yast developers do not know any of them, and the legacy Yast testing framework (expects + dejagnu) works completely differently (and looks weird from Ruby perspective)...
there's another topic to chew on: do we need to standardize this at all?
Having a common standard in all Yast packages should be preferred. Yast maintainers change quite often, some modules are moved or changed between maintainers etc.. More frameworks also mean more maintenance effort (to package them to RPMS). And if you want to send a patch to someone else's module you don't need to learn a new testing framework just because of adding a single test case. Also this would make automatic checks easier (e.g. in Jenkins you don't have to care which gem/packages are needed, how to run the tests in a particular Yast package etc...) But if a different framework is more suitable in a special case it should be fine to use it, there should be no dogma. -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 12.8.2013 14:47, Ladislav Slezak napsal(a): ...
I do not have any strong preference, I only know the test-unit + mocha which we already use in Webyast and SLMS. (SLMS in addition to this uses factory_girl, but Yast does not need any DB mocking so it's irrelevant here...)
but before we dive into a massive flamewar re: which one is the best,
I don't think there will be a flamewar as Yast developers do not know any of them, and the legacy Yast testing framework (expects + dejagnu) works completely differently (and looks weird from Ruby perspective)...
there's another topic to chew on: do we need to standardize this at all?
Having a common standard in all Yast packages should be preferred. Yast maintainers change quite often, some modules are moved or changed between maintainers etc..
This is the for me the strongest argument to use a single test framework. More frameworks also mean more maintenance effort
(to package them to RPMS).
Yep, and do not forget the dependencies and support accross several sles and opensuse releases. I would prefer to use the potential of ruby standard library and vote for test/unit but it lacks support for mocking and stubbing which brings additional dependency on another library, e.g. [1]. On the other hand there are rspec and minitest that have built-in mocking support and a nice testing DSL; and both libs have good maintainers and tons of extensions (if needed). The advantage of using minitest is that it provides the test/unit style of writing unit tests. Mayge we might have to make the effort to try them out on some translated yast code before we make the final decision and rewrite the tests afterwards when the experience proves unsatisfactory. vlado [1] https://github.com/freerange/mocha
On Mon, 12 Aug 2013 15:52:39 +0200 Vladimir Moravec <vmoravec@suse.com> wrote:
Dne 12.8.2013 14:47, Ladislav Slezak napsal(a): ...
I do not have any strong preference, I only know the test-unit + mocha which we already use in Webyast and SLMS. (SLMS in addition to this uses factory_girl, but Yast does not need any DB mocking so it's irrelevant here...)
but before we dive into a massive flamewar re: which one is the best,
I don't think there will be a flamewar as Yast developers do not know any of them, and the legacy Yast testing framework (expects + dejagnu) works completely differently (and looks weird from Ruby perspective)...
there's another topic to chew on: do we need to standardize this at all?
Having a common standard in all Yast packages should be preferred. Yast maintainers change quite often, some modules are moved or changed between maintainers etc..
This is the for me the strongest argument to use a single test framework.
More frameworks also mean more maintenance effort
(to package them to RPMS).
Yep, and do not forget the dependencies and support accross several sles and opensuse releases.
Well, it is not needed if we do unit testing only in jenkins and not during runtime, then you don't need to support it. It allows us to have more freedom when we choose testing framework.
I would prefer to use the potential of ruby standard library and vote for test/unit but it lacks support for mocking and stubbing which brings additional dependency on another library, e.g. [1].
On the other hand there are rspec and minitest that have built-in mocking support and a nice testing DSL; and both libs have good maintainers and tons of extensions (if needed). The advantage of using minitest is that it provides the test/unit style of writing unit tests.
I personally like how rspec write its test description as it allows nice orientation in tests, especially when it starting grown. But even simple test/unit style is fine for me.
Mayge we might have to make the effort to try them out on some translated yast code before we make the final decision and rewrite the tests afterwards when the experience proves unsatisfactory.
well, I think it more depends on taste of developer and we already have bunch of code where we can check how it looks. E.g. SLMS and webyast for test/unit style e.g. [1] and y2r for rspec style e.g. [2]. Ruby bindings use also test unit [3] [1] https://github.com/webyast/webyast/blob/master/webyast/test/unit/basesystem_... [2] https://github.com/yast/y2r/blob/master/spec/y2r/parser_spec.rb [3] https://github.com/yast/yast-ruby-bindings/blob/master/tests/ruby/builtins_t... Josef
vlado
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 16.8.2013 11:08, Josef Reidinger napsal(a):
On Mon, 12 Aug 2013 15:52:39 +0200 Vladimir Moravec <vmoravec@suse.com> wrote: [...]
Yep, and do not forget the dependencies and support accross several sles and opensuse releases.
Well, it is not needed if we do unit testing only in jenkins and not during runtime, then you don't need to support it. It allows us to have more freedom when we choose testing framework.
We run the tests also during package build in OBS (or IBS), that means we need the framework to be packaged in RPMs. (And also our customers should be able to rebuild packages so the RPMs should be included in SLE SDK.) Jenkins only handles Git master branch, having tests in RPM build is currently a must have for SLES maintenance. -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Mon, 26 Aug 2013 16:28:44 +0200 Ladislav Slezak <lslezak@suse.cz> wrote:
Dne 16.8.2013 11:08, Josef Reidinger napsal(a):
On Mon, 12 Aug 2013 15:52:39 +0200 Vladimir Moravec <vmoravec@suse.com> wrote: [...]
Yep, and do not forget the dependencies and support accross several sles and opensuse releases.
Well, it is not needed if we do unit testing only in jenkins and not during runtime, then you don't need to support it. It allows us to have more freedom when we choose testing framework.
We run the tests also during package build in OBS (or IBS), that means we need the framework to be packaged in RPMs. (And also our customers should be able to rebuild packages so the RPMs should be included in SLE SDK.)
Jenkins only handles Git master branch, having tests in RPM build is currently a must have for SLES maintenance.
Well, it is not problem to extend jenkins to handle also different branch ( especially if we simplify it ). But I agree, that it needs some work and having tests in rpms can discovery quickly changes in underlaying libraries, that is updated in BS. Josef
--
Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, is there a decision about the prefered test framework yet? Reading this tread and the thread about Requirements for YaST testing framework, for me it seems that 'minitest' has slight advantage. And I think it is agreed that all YaST modules should use the same frame work (as far as possible). Greetings Gabi Am 08.08.2013 17:37, schrieb Roman Neuhauser:
hola!
vlado brought up an interesting topic, namely the choice of a unit-testing framework. there's a bunch of them written for ruby code, some of the most popular ones (presented in no particular order) are:
* minitest http://rubydoc.info/gems/minitest/frames * rspec http://rspec.info/ * shoulda http://rubydoc.info/gems/shoulda/frames * Test::Unit http://www.ruby-doc.org/stdlib-1.9.3/libdoc/test/unit/rdoc/Test/Unit.html
see https://en.wikipedia.org/wiki/Unit_testing_frameworks_for_Ruby for a pretty good overview.
i have no strong preference or dislike for any of them, with the exception of rspec, which creates methods on the CUT (to be clear: i don't like that).
but before we dive into a massive flamewar re: which one is the best, there's another topic to chew on: do we need to standardize this at all?
as far as i understand the situation, yast modules are mostly silos with one or two people ever hacking on any given module. this arrangement would lend weight to the argument that no, we don't need to standardize on a single unit-testing framework, each maintainer is free to choose whatever they like most. OTOH, strong code-ownership has its downsides, so maybe, unless we want to make the segregation of interests stronger, we should not move the code further apart. a standard test framework would make it easier to move code between modules, eg. utility functions initially written for a specific yast module getting promoted to ruby-bindings or something.
opinions?
- -- Gabriele Mohr SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstr. 5 Tel: +49 911 740 53 362 90409 Nürnberg Email: gs@suse.de - ----------------------------------------------------------------- -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iD8DBQFSJGI0zwhO63ql6h0RAhNsAJ4qc3OzD1PRh32turuVRMv6pGvDowCdGPuH KLH10Cc+wYVPRAfVSVmS3ww= =JEa0 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Mon, 02 Sep 2013 12:02:28 +0200 Gabriele Mohr <gs@suse.de> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
is there a decision about the prefered test framework yet?
Reading this tread and the thread about Requirements for YaST testing framework, for me it seems that 'minitest' has slight advantage. And I think it is agreed that all YaST modules should use the same frame work (as far as possible).
Greetings Gabi
Well, I play a bit with minitest and must say that it contain bunch of functionality, but there is two really annoying things about it. 1) It mocking mechanism - it is limited to be used only in block, so it cannot be used in before stage 2) output of spec-style writing. In fact it just translate it into test/unit tests and output is quite horrible compared to rspec. Question is how big is these two pains and how hard will be to use rspec instead ( it in not part of ruby core ). Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 2.9.2013 12:29, Josef Reidinger napsal(a):
Well, I play a bit with minitest and must say that it contain bunch of functionality, but there is two really annoying things about it.
1) It mocking mechanism - it is limited to be used only in block, so it cannot be used in before stage
To clarify the impact: This means that tests with mocks would contain duplicate code. Of course, this could be avoided by using some custom mocking framework, like Mocha [1], but: 1. There can be integration issues between Mocha and minitest, so this combo would need to be tested first. 2. The main advantage of minitest -- being included in Ruby stdlib -- would be lost, as we would be dependent on Mocha too. I should add that RSpec already contains a mocking framework, very similar to Mocha. It works well in my experience (I used it on multiple projects).
2) output of spec-style writing. In fact it just translate it into test/unit tests and output is quite horrible compared to rspec.
Could you provide an example output? One thing I value on RSpec is the --format=documentation output: http://ascii.io/a/5272 This allows one to see specs in structured way, which makes it easier to spot organization & clarity issues, refactor & restructure the specs, etc. And it serves as a documentation of capabilities of the tested classes (if one writes the spec descriptions well). [1] http://gofreerange.com/mocha/docs/ -- David Majda SUSE Studio developer http://susestudio.com/ -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Mon, 02 Sep 2013 13:01:20 +0200 David Majda <dmajda@suse.cz> wrote:
Dne 2.9.2013 12:29, Josef Reidinger napsal(a):
Well, I play a bit with minitest and must say that it contain bunch of functionality, but there is two really annoying things about it.
1) It mocking mechanism - it is limited to be used only in block, so it cannot be used in before stage
To clarify the impact: This means that tests with mocks would contain duplicate code. Of course, this could be avoided by using some custom mocking framework, like Mocha [1], but:
1. There can be integration issues between Mocha and minitest, so this combo would need to be tested first.
2. The main advantage of minitest -- being included in Ruby stdlib -- would be lost, as we would be dependent on Mocha too.
I should add that RSpec already contains a mocking framework, very similar to Mocha. It works well in my experience (I used it on multiple projects).
2) output of spec-style writing. In fact it just translate it into test/unit tests and output is quite horrible compared to rspec.
Could you provide an example output?
yes. it is at http://pastebin.com/W3Hyjn0k . To describe failing test, It in `describe "Yast::IP"` in `describe "when told to check IPv6"` and test has generated name ( I generate it, because I have there set of various ipv6 addresses ) `it "return true for valid IPv6 '#{valid_ip6}'" do` and valid_ipv6 has changed value ( by my, to really fail - "1:2:3:4:5:68888888:127.0.0.1". Code is at [1] BTW This is actually work-around. I like to have return true for valid ipv6 and have there list of various ipv6, but then it doesn't write value of ipv6 and return only `expect true, returned false`, which is useless when test fail. [1] https://github.com/yast/yast-yast2/blob/master/library/types/test/ip_test.rb...
One thing I value on RSpec is the --format=documentation output:
This allows one to see specs in structured way, which makes it easier to spot organization & clarity issues, refactor & restructure the specs, etc. And it serves as a documentation of capabilities of the tested classes (if one writes the spec descriptions well).
yes, looks quite nice. Josef
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
participants (7)
-
Arvin Schnell -
David Majda -
Gabriele Mohr -
Josef Reidinger -
Ladislav Slezak -
Roman Neuhauser -
Vladimir Moravec