On Fri, Aug 09, 2013 at 07:34:20PM +0200, Roman Neuhauser wrote:

# rneuhauser@suse.cz / 2013-08-08 17:37:51 +0200:

...

i have no strong preference or dislike for any of them, with the exception of rspec, which creates methods on the CUT (to be clear: i don't like that).

i see in archives recent discussion of unit tests:

http://lists.opensuse.org/yast-devel/2013-08/msg00000.html http://lists.opensuse.org/yast-devel/2013-08/msg00014.html

vlado points out that Rspec is moving away from clobbering the CUT, which neuters my complaint.

What is CUT? ciao Arvin -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

On Mon, 12 Aug 2013 15:52:39 +0200 Vladimir Moravec wrote:

Dne 12.8.2013 14:47, Ladislav Slezak napsal(a): ...

...

I do not have any strong preference, I only know the test-unit + mocha which we already use in Webyast and SLMS. (SLMS in addition to this uses factory_girl, but Yast does not need any DB mocking so it's irrelevant here...)

...

but before we dive into a massive flamewar re: which one is the best,

I don't think there will be a flamewar as Yast developers do not know any of them, and the legacy Yast testing framework (expects + dejagnu) works completely differently (and looks weird from Ruby perspective)...

...

there's another topic to chew on: do we need to standardize this at all?

Having a common standard in all Yast packages should be preferred. Yast maintainers change quite often, some modules are moved or changed between maintainers etc..

This is the for me the strongest argument to use a single test framework.

More frameworks also mean more maintenance effort

...

(to package them to RPMS).

Yep, and do not forget the dependencies and support accross several sles and opensuse releases.

Well, it is not needed if we do unit testing only in jenkins and not during runtime, then you don't need to support it. It allows us to have more freedom when we choose testing framework.

I would prefer to use the potential of ruby standard library and vote for test/unit but it lacks support for mocking and stubbing which brings additional dependency on another library, e.g. [1].

On the other hand there are rspec and minitest that have built-in mocking support and a nice testing DSL; and both libs have good maintainers and tons of extensions (if needed). The advantage of using minitest is that it provides the test/unit style of writing unit tests.

I personally like how rspec write its test description as it allows nice orientation in tests, especially when it starting grown. But even simple test/unit style is fine for me.

Mayge we might have to make the effort to try them out on some translated yast code before we make the final decision and rewrite the tests afterwards when the experience proves unsatisfactory.

well, I think it more depends on taste of developer and we already have bunch of code where we can check how it looks. E.g. SLMS and webyast for test/unit style e.g. [1] and y2r for rspec style e.g. [2]. Ruby bindings use also test unit [3] [1] https://github.com/webyast/webyast/blob/master/webyast/test/unit/basesystem_... [2] https://github.com/yast/y2r/blob/master/spec/y2r/parser_spec.rb [3] https://github.com/yast/yast-ruby-bindings/blob/master/tests/ruby/builtins_t... Josef

vlado

[1] https://github.com/freerange/mocha

-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

On Mon, 26 Aug 2013 16:28:44 +0200 Ladislav Slezak wrote:

Dne 16.8.2013 11:08, Josef Reidinger napsal(a):

...

On Mon, 12 Aug 2013 15:52:39 +0200 Vladimir Moravec wrote: [...]

...

Yep, and do not forget the dependencies and support accross several sles and opensuse releases.

Well, it is not needed if we do unit testing only in jenkins and not during runtime, then you don't need to support it. It allows us to have more freedom when we choose testing framework.

We run the tests also during package build in OBS (or IBS), that means we need the framework to be packaged in RPMs. (And also our customers should be able to rebuild packages so the RPMs should be included in SLE SDK.)

Jenkins only handles Git master branch, having tests in RPM build is currently a must have for SLES maintenance.

Well, it is not problem to extend jenkins to handle also different branch ( especially if we simplify it ). But I agree, that it needs some work and having tests in rpms can discovery quickly changes in underlaying libraries, that is updated in BS. Josef

--

Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE

-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

On Mon, 02 Sep 2013 12:02:28 +0200 Gabriele Mohr wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Hi,

is there a decision about the prefered test framework yet?

Reading this tread and the thread about Requirements for YaST testing framework, for me it seems that 'minitest' has slight advantage. And I think it is agreed that all YaST modules should use the same frame work (as far as possible).

Greetings Gabi

Well, I play a bit with minitest and must say that it contain bunch of functionality, but there is two really annoying things about it. 1) It mocking mechanism - it is limited to be used only in block, so it cannot be used in before stage 2) output of spec-style writing. In fact it just translate it into test/unit tests and output is quite horrible compared to rspec. Question is how big is these two pains and how hard will be to use rspec instead ( it in not part of ruby core ). Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

On Mon, 02 Sep 2013 13:01:20 +0200 David Majda wrote:

Dne 2.9.2013 12:29, Josef Reidinger napsal(a):

...

Well, I play a bit with minitest and must say that it contain bunch of functionality, but there is two really annoying things about it.

1) It mocking mechanism - it is limited to be used only in block, so it cannot be used in before stage

To clarify the impact: This means that tests with mocks would contain duplicate code. Of course, this could be avoided by using some custom mocking framework, like Mocha [1], but:

1. There can be integration issues between Mocha and minitest, so this combo would need to be tested first.

2. The main advantage of minitest -- being included in Ruby stdlib -- would be lost, as we would be dependent on Mocha too.

I should add that RSpec already contains a mocking framework, very similar to Mocha. It works well in my experience (I used it on multiple projects).

...

2) output of spec-style writing. In fact it just translate it into test/unit tests and output is quite horrible compared to rspec.

Could you provide an example output?

yes. it is at http://pastebin.com/W3Hyjn0k . To describe failing test, It in `describe "Yast::IP"` in `describe "when told to check IPv6"` and test has generated name ( I generate it, because I have there set of various ipv6 addresses ) `it "return true for valid IPv6 '#{valid_ip6}'" do` and valid_ipv6 has changed value ( by my, to really fail - "1:2:3:4:5:68888888:127.0.0.1". Code is at [1] BTW This is actually work-around. I like to have return true for valid ipv6 and have there list of various ipv6, but then it doesn't write value of ipv6 and return only `expect true, returned false`, which is useless when test fail. [1] https://github.com/yast/yast-yast2/blob/master/library/types/test/ip_test.rb...

One thing I value on RSpec is the --format=documentation output:

http://ascii.io/a/5272

This allows one to see specs in structured way, which makes it easier to spot organization & clarity issues, refactor & restructure the specs, etc. And it serves as a documentation of capabilities of the tested classes (if one writes the spec descriptions well).

yes, looks quite nice. Josef

[1] http://gofreerange.com/mocha/docs/

-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org