Thanks for the reply, I decided to reply to this e-mail and not to previous one to keep answers in one place. On 10/4/18 9:15 AM, Josef Reidinger wrote:
On Thu, 4 Oct 2018 09:02:46 +0200 Rodion Iafarov
wrote: -------- Forwarded Message -------- Subject: YaST GUI testing framework Date: Mon, 24 Sep 2018 18:37:18 +0200 From: Rodion Iafarov
To: yast-devel@opensuse.org Dear YaSTees,
I would like to kindly ask for your help. In order to make next game changing step for YaST GUI testing, we need to improve macro player functionality. My first question would be: is the macro player good enough? If we want to invest some time in this area, maybe we could use something else. Yes, as for the recorder it could speed up test development a bit, but in general we just need player. In case we cannot get enough support and development is too heavy, we may use squish or ldtp, which have certain limitations, but will server the purpose. I have spent two hackweeks developing an automatic YaST UI testing
Dne 02. 10. 18 v 11:18 Lukas Ocilka napsal(a): framework. It's based on a REST API (HTTP with JSON format), it can not only send the user input actions but can also test the content of the current dialog.
The advantage is that the REST API is not bound to any programming language, you can write a client driving an YaST installation in any language. With help of "curl" and "jq" it would be possible even in shell... (but I'd avoid it)
The YaST UI actions which can be done by user are quite limited (e.g. you can press a button, select a checkbox, write something to an input field... and that's basically it). So made a simple Cucumber wrapper which defines such steps. Unfortunately, this was initial reason why we still on current state of
On 10/3/18 11:00 AM, Ladislav Slezak wrote: things. Even with simplest yast module (e.g. hostnames), we need interaction with tables, selecting rows, checking values in the cells, etc. So this is a must to proceed. That's also the reason why we can consider other tools which can already operate on qt. Problem of tools operating only on qt is that we really have to test also ncurses as many corporate customers uses ncurses ( info we get from support and sale guys ), so it is also very vital to test it ( like that everything fits into screen, all is accessible, no trash in output and so on ). That's true, but we need to start somewhere. If we cover qt for functional, we can at least limit coverage with ncurses, as performing same action will trigger same code underneath.
Of course, if you do not like Cucumber you can use any other framework, you could extend the current openQA or start something new from scratch. That's a big advantage of the generic REST API, if the testing frontend does not fit any more you can easily change it without affecting YaST itself. We could even use a different frontend in our CI (Travis/Jenkins) and different one in openQA built on top of the same REST backend.
See more details about the REST API in my hackweek blog posts [1], [2].
If you want to give it a try I have prepared RPM packages for SLE15/Leap15 in [3], how to install the packages is described in the Cucumber wrapper at GitHub [4].
(Note: I built the packages several months ago, if something does not work now just ping me.)
This project is far from complete, there are still some missing parts like support for some specific widgets, security improvements (authentication and encryption, isolating to a separate plugin so the REST API can be installed only when needed, not included in the default installation, etc...).
But the prototype is already able to test the default openSUSE/SLE installation well, (see the screencasts in the blog) so I think it would make sense to at least think about this possibility as an alternative to the UI macro player.
If you have any questions about this project just ask me. I've actually seen those two projects and they look really promising, so I'm with Lukas here, seems like a good solution.
However, I started this discussion to confirm preferable way, so we could collaborate. I think first step can be to write down your use cases and also your requirements, so hopefully we can somehow prioritize work on it.
These is combined list for the needs for your previous email: 1. Operating on all existing widgets, including tables, trees, comboboxes, tabs. Being able to perform same set of actions like user can do normally. 2. Possibility to assess widgets properties e.g. values in the table cells, labels text, UI properties (if control is visible/disabled) 3. Some way of integration with openQA (that's important for QA team, as we have to use it, but it should not be an issue as we can simply create file with results and parse it) 4. There is not to complex way to launch tests when running installation (e.g. booting from installer DVD and triggering test) 5. There is a way to run tests with local changes without configuring any kind of complex setup 6. Solution can be used to run CI per branch 7. Can be used for ncurses and qt GUIs 8. Running tests minimizes influence on SUT (we should keep difference of real installation vs integration one as little as possible not installing hundreds of the dependencies). As we cannot get it all, my proposal would be to start with something more simple, e.g. yast module instead of installer. But even for simplest ones e.g. timezones and hostnames, we lack functionality at the moment. Also, I would like to mention, we as QA department are really interested in the development and considered driving it with the support of the YaST team, as I fully understand lack of resources for it. But we need your support to decide on a solution and helping to create tasks so we could pick them up.
Josef
Note: Because this REST API is available for any libyui application and is not YaST specific we could possibly cooperate with the Mageia (formerly Mandriva, formerly Mandrake) developers, their management tools also use the libyui [5] and having a testing framework would probably help them as well...
Ladislav
[1] https://blog.ladslezak.cz/2018/07/13/hackweek-17/ [2] https://blog.ladslezak.cz/2017/12/06/hackweek-16-yast-ui-rest-api/ [3] https://build.opensuse.org/project/show/home:lslezak:libyui-rest-api [4] https://github.com/lslezak/cucumber-yast [5] https://madb.mageia.org/package/show/name/manatools
--
Best Regards
Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/
--
Kind regards,
Rodion Iafarov