[yast-devel] PoC: Running integration tests / smoke tests at Travis
Hi all, Short Background ----------------- I fixed a trivial bug in the Rear YaST module [1]. The module did not work at all and crashed with "Internal error" right at the start because some files were missing in the RPM package. The nasty thing is that the bug was unnoticed for 6(!!) months. And we got a bug report just after the SP3 was released... :-( Improving it ------------ So I was thinking how to improve the situation, if anybody did a smoke test and just try to run the module the bug would have been found. That means we would need an integration test. openQA was my first idea but it's hard to maintain and we would get the result too late. But since we use Docker at Travis and we already build the YaST packages there then it should be easy to install the package and run it. Proof of Concept ---------------- I have implemented a trivial openQA equivalent using the "tmux" package which is similar to "screen". That means the tests are run using the ncurses UI which is lightweight and should work fast in Travis. Currently I can dump the screen to the Travis log (see the examples below) check what's on the screen, send keyboard presses and navigate in the workflow. For the rear module I can verify that the module starts properly, I can even change a value in the configuration, save it and then start the module again to verify it has been saved properly. This initial run shows that the module is broken [1] and displays the "Internal error" popup as reported in bugzilla. After merging the fix [2] it runs properly and runs the save and load workflow as expected, check the screen dumps in the log [3]. The integration test is implemented in a shell script [4]. It's still just a proof of concept, there are a lot of TODOs, the structure does not look nice, some functions should be moved to a separate shared file, etc... But it really shows that it could improve the test automation a lot. Advantages ---------- - Runs at Travis, the tests are run early, even _before_ we actually merge the change - Uses ncurses UI and simple "grep", no fragile screenshot comparing (It's still not ideal, but should be good enough at this point.) - You can run the same tests locally using the same Docker image => easy debugging, stable environment (more or less) - Writing smoke tests is easy even for legacy modules Disadvantages ------------- - Cannot run full installation, only modules in the installed system - So AutoYaST cannot be tested as well. On the other hand AY allows system cloning and can apply a profile to the running system so we could at least partly test the AY functionality at Travis. And many modules support interactive "configuration" mode. - Docker is container based, no HW virtualization provided, that means it cannot test HW related modules. But we can still run at least simple smoke tests that verify the module starts and displays "No printer detected" or something like that. That still makes sense. TODO/Ideas ---------- - The common functions should be shared (in yast2-devtools?), add helpers for common testing scenarios. - A shell script might not be the best tool, maybe rspec or serverspec [5] might be better. On the other hand a simple shell script has less dependencies and could be easily shared with openQA. It could run the same tests to ensure YaST still works when the external dependencies are changed. - For mocking the external tools in complex scenarios we could use shellmock [6]. What do you think about it? Any ideas, comments? Thank you in advance! [1] https://travis-ci.org/yast/yast-rear/builds/259803414#L681 [2] https://github.com/yast/yast-rear/pull/7 [3] https://travis-ci.org/yast/yast-rear/builds/260086828#L670 [4] https://github.com/yast/yast-rear/pull/8 [5] http://serverspec.org/ [6] https://github.com/gooddata/shellmock -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Hi, I will top post as it is quite long email, but keeping it to have referece. So I think at first we should agreed what is goal of such tests. If it is smoke test, then it should run almost automatic without writting too much ( ideally nothing, just run ncurses check if there is not error and exit ). If goal is to run integration tests, then we already have tools around and question is why we should use it and the most important decide what to use, as having three various integration tests tools is very confusing and time consuming. So e.g. for rear example, smoke test can find problem. Of course if we write integration test it will also detect it, but question is why write new tool for it when we can have such test even in openQA which we currently using? So to sum it up, if we use it as smoke test, then it have to run without modification of target module git repo and run it automatic for given repo ( e.g. take command from desktop file, start it in ncurses, check for error and then abort ). If we want to use it as integration test, then it have to be properly discussed why to use this and not openQA as having multiple tools for same purpose does not look like good idea for me. Josef On Wed, 2 Aug 2017 12:59:19 +0200 Ladislav Slezak <lslezak@suse.cz> wrote:
Hi all,
Short Background -----------------
I fixed a trivial bug in the Rear YaST module [1]. The module did not work at all and crashed with "Internal error" right at the start because some files were missing in the RPM package.
The nasty thing is that the bug was unnoticed for 6(!!) months. And we got a bug report just after the SP3 was released... :-(
Improving it ------------
So I was thinking how to improve the situation, if anybody did a smoke test and just try to run the module the bug would have been found.
That means we would need an integration test. openQA was my first idea but it's hard to maintain and we would get the result too late.
But since we use Docker at Travis and we already build the YaST packages there then it should be easy to install the package and run it.
Proof of Concept ----------------
I have implemented a trivial openQA equivalent using the "tmux" package which is similar to "screen". That means the tests are run using the ncurses UI which is lightweight and should work fast in Travis.
Currently I can dump the screen to the Travis log (see the examples below) check what's on the screen, send keyboard presses and navigate in the workflow.
For the rear module I can verify that the module starts properly, I can even change a value in the configuration, save it and then start the module again to verify it has been saved properly.
This initial run shows that the module is broken [1] and displays the "Internal error" popup as reported in bugzilla. After merging the fix [2] it runs properly and runs the save and load workflow as expected, check the screen dumps in the log [3].
The integration test is implemented in a shell script [4]. It's still just a proof of concept, there are a lot of TODOs, the structure does not look nice, some functions should be moved to a separate shared file, etc...
But it really shows that it could improve the test automation a lot.
Advantages ----------
- Runs at Travis, the tests are run early, even _before_ we actually merge the change
- Uses ncurses UI and simple "grep", no fragile screenshot comparing (It's still not ideal, but should be good enough at this point.)
- You can run the same tests locally using the same Docker image => easy debugging, stable environment (more or less)
- Writing smoke tests is easy even for legacy modules
Disadvantages -------------
- Cannot run full installation, only modules in the installed system
- So AutoYaST cannot be tested as well. On the other hand AY allows system cloning and can apply a profile to the running system so we could at least partly test the AY functionality at Travis. And many modules support interactive "configuration" mode.
- Docker is container based, no HW virtualization provided, that means it cannot test HW related modules.
But we can still run at least simple smoke tests that verify the module starts and displays "No printer detected" or something like that. That still makes sense.
TODO/Ideas ----------
- The common functions should be shared (in yast2-devtools?), add helpers for common testing scenarios.
- A shell script might not be the best tool, maybe rspec or serverspec [5] might be better. On the other hand a simple shell script has less dependencies and could be easily shared with openQA. It could run the same tests to ensure YaST still works when the external dependencies are changed.
- For mocking the external tools in complex scenarios we could use shellmock [6].
What do you think about it? Any ideas, comments? Thank you in advance!
[1] https://travis-ci.org/yast/yast-rear/builds/259803414#L681 [2] https://github.com/yast/yast-rear/pull/7 [3] https://travis-ci.org/yast/yast-rear/builds/260086828#L670 [4] https://github.com/yast/yast-rear/pull/8 [5] http://serverspec.org/ [6] https://github.com/gooddata/shellmock
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 2.8.2017 v 13:10 Josef Reidinger napsal(a):
So I think at first we should agreed what is goal of such tests. If it is smoke test, then it should run almost automatic without writting too much ( ideally nothing, just run ncurses check if there is not error and exit ).
Um, I'm not sure if we can make such automatic check generic enough. I mean some modules display a popup at start, some want to install additional packages, etc. How can I "exit" safely and reliably? Maybe there is "Abort" button, maybe "Cancel", maybe it's "OK", maybe you have to "Cancel" the popup first then "Abort" the main dialog... Killing running YaST also does not look nice to me. But yes, having such an automatic check without need to touch the sources would be nice.
If goal is to run integration tests, then we already have tools around and question is why we should use it and the most important decide what to use, as having three various integration tests tools is very confusing and time consuming.
So e.g. for rear example, smoke test can find problem. Of course if we write integration test it will also detect it, but question is why write new tool for it when we can have such test even in openQA which we currently using?
Um, there is *no* openQA test for Rear (otherwise it would have found the issue). Do you really want to write and maintain openQA tests? IIRC we discussed it several times and the response was always "no". Also running openQA is not trivial, how I can run an openQA test on my machine before I open a pull request? What about the community developers who open a pull request from a fork? Travis works out of box with very small effort on our side...
So to sum it up, if we use it as smoke test, then it have to run without modification of target module git repo and run it automatic for given repo ( e.g. take command from desktop file, start it in ncurses, check for error and then abort ). If we want to use it as integration test, then it have to be properly discussed why to use this and not openQA as having multiple tools for same purpose does not look like good idea for me.
Um, personally I do not like openQA as writing and maintaining the tests is difficult and running them locally is almost impossible (at least for me). Look at the storage-ng case, Christopher is fighting with the openQA and OBS all the time and it requires really high effort just to keep it working. With this lightweight approach the effort should be much lower. (But don't get we wrong, openQA still has a great value as it can test the whole installation, test several HW configurations, etc...) And the most importantly: And as I mentioned, if we use a simple solution (e.g. a simple shell script) openQA could run the very same script. So there would be actually no duplication and we could even save some work for the openQA team. My proposal is: - Use unit tests whenever you can - Use Travis for smoke tests and integration testing with external tools (simple scenarios where it's possible) - For the rest use openQA - Run the Travis integration test also in openQA So far we have unit test (small scope, quick, simple to maintain) and openQA (full distro scope, slow, difficult to maintain), IMHO it makes sense to have something in the middle with the advantages of the both systems. -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Hello, Am Mittwoch, 2. August 2017, 14:29:19 CEST schrieb Ladislav Slezak:
Dne 2.8.2017 v 13:10 Josef Reidinger napsal(a):
So I think at first we should agreed what is goal of such tests. If it is smoke test, then it should run almost automatic without writting too much ( ideally nothing, just run ncurses check if there is not error and exit ). Um, I'm not sure if we can make such automatic check generic enough. I mean some modules display a popup at start, some want to install additional packages, etc.
How can I "exit" safely and reliably? Maybe there is "Abort" button, maybe "Cancel", maybe it's "OK", maybe you have to "Cancel" the popup first then "Abort" the main dialog... Killing running YaST also does not look nice to me.
But yes, having such an automatic check without need to touch the sources would be nice.
Maybe I have an idea ;-) yast $module help should print some help text and exit for all YaST modules. It seems "longhelp" and "xmlhelp" are also expected to work in every YaST module. I know this discussion is not really a place for a bugreport, so please take everything below as an argument why such a check would make sense, and tell me if you want bugreports for the affected modules. You remember the "should print some help text and exit" above? "Should", because not all modules behave that way, which means you can even start them with yast $module <tab><tab> ;-) I reported about 40 bugs for this 10 years ago - see boo#266892, which also contains a possible test script in comment #3, or search for bug reports from me with "yast help" in the summary. In the meantime, there are some (new?) modules with this behaviour, which means the automated check promised in the bugreport isn't in place yet or anymore. For the records: On my system, the following YaST modules don't support the help option and get started instead: auth-client fonts journal ldapkrb virt-install vpn yast $module longhelp and yast $module xmlhelp has some more fallout.: all modules listed above, additionally: registration scc sw_single view_anymsg There might be more, I don't have all YaST modules installed. Tell me if you want a bugreport for each of them ;-) Regards, Christian Boltz -- Aus der Beschreibung entnehme ich, daß deine Fonts nach Typ 3 konvertiert werden (Finger im Hals) und deine Bilder auf Screen- Qualität (Fuß zum Finger dazusteck...) [Ratti in suse-linux] -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 2.8.2017 v 12:59 Ladislav Slezak napsal(a):
After merging the fix [2] it runs properly and runs the save and load workflow as expected, check the screen dumps in the log [3].
BTW I tried using color screenshots but Travis does not render that well. What a shame, that would be cool... (https://travis-ci.org/yast/yast-rear/builds/259785540#L668) -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
participants (3)
-
Christian Boltz -
Josef Reidinger -
Ladislav Slezak