[yast-devel] YaST GUI testing framework
-------- Forwarded Message --------
Subject: YaST GUI testing framework
Date: Mon, 24 Sep 2018 18:37:18 +0200
From: Rodion Iafarov
On 10/02/2018 11:18 AM, Lukas Ocilka wrote:
-------- Forwarded Message -------- Subject: YaST GUI testing framework Date: Mon, 24 Sep 2018 18:37:18 +0200 From: Rodion Iafarov
To: yast-devel@opensuse.org Dear YaSTees,
I would like to kindly ask for your help. In order to make next game changing step for YaST GUI testing, we need to improve macro player functionality.
After a nice discussion with Stefan Hundhammer and Chistopher Hofmann, we got clear that we cannot simply resurrect functionality which was there already as it's based on YCP, which YaST team wants to replace for quite a while. Our QA team feels fancy to help with the progress in order to get scalable framework for YaST GUI testing, including installer. However, as you have better insights, we need your help to decide steps to be done and split them to the items we can deliver during the sprint. This will allow us slowly moving forward.
Questions to clarify before we start:
1) Will YCP be dropped in foreseeable future, if yes what shall be used instead? yaml+ruby?
Do you mean the programming language called YCP or the components system called YCP? I'm asking because, although we usually say "YCP" to refer to that perl-ish language we are indeed dropping, YCP actually stands for YaST Communication Protocol and I would say is still the main method to communicate our Ruby code with libYUI and with WFM. Isn't it? For more fun with acronyms, check https://yastgithubio.readthedocs.io/en/latest/architecture/
2) Where to start? We could have a fork of libyui which is used for testing, or our implementation has to be secure not to introduce any new flaws
Using a separate "feature branch" sounds good as long as it does not run for two years before being merged back to master, like storage-ng. :-)
In case we cannot get enough support and development is too heavy, we may use squish or ldtp, which have certain limitations, but will server the purpose.
Looking forward to your reply.
Cheers. -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 10/3/18 10:27 AM, Ancor Gonzalez Sosa wrote:
On 10/02/2018 11:18 AM, Lukas Ocilka wrote:
-------- Forwarded Message -------- Subject: YaST GUI testing framework Date: Mon, 24 Sep 2018 18:37:18 +0200 From: Rodion Iafarov
To: yast-devel@opensuse.org Dear YaSTees,
I would like to kindly ask for your help. In order to make next game changing step for YaST GUI testing, we need to improve macro player functionality.
After a nice discussion with Stefan Hundhammer and Chistopher Hofmann, we got clear that we cannot simply resurrect functionality which was there already as it's based on YCP, which YaST team wants to replace for quite a while. Our QA team feels fancy to help with the progress in order to get scalable framework for YaST GUI testing, including installer. However, as you have better insights, we need your help to decide steps to be done and split them to the items we can deliver during the sprint. This will allow us slowly moving forward.
Questions to clarify before we start:
1) Will YCP be dropped in foreseeable future, if yes what shall be used instead? yaml+ruby? Do you mean the programming language called YCP or the components system called YCP?
I actually meant both, as my understanding was that Communication Protocol is also a subject to change. In general, it doesn't matter what will it be, my main motivation is to find solution which fits all our needs.
I'm asking because, although we usually say "YCP" to refer to that perl-ish language we are indeed dropping, YCP actually stands for YaST Communication Protocol and I would say is still the main method to communicate our Ruby code with libYUI and with WFM. Isn't it?
For more fun with acronyms, check https://yastgithubio.readthedocs.io/en/latest/architecture/
2) Where to start? We could have a fork of libyui which is used for testing, or our implementation has to be secure not to introduce any new flaws Using a separate "feature branch" sounds good as long as it does not run for two years before being merged back to master, like storage-ng. :-)
In case we cannot get enough support and development is too heavy, we may use squish or ldtp, which have certain limitations, but will server the purpose.
Looking forward to your reply. Cheers.
--
Kind regards,
Rodion Iafarov
On Thu, 4 Oct 2018 09:06:19 +0200
Rodion Iafarov
On 10/3/18 10:27 AM, Ancor Gonzalez Sosa wrote:
On 10/02/2018 11:18 AM, Lukas Ocilka wrote:
-------- Forwarded Message -------- Subject: YaST GUI testing framework Date: Mon, 24 Sep 2018 18:37:18 +0200 From: Rodion Iafarov
To: yast-devel@opensuse.org Dear YaSTees,
I would like to kindly ask for your help. In order to make next game changing step for YaST GUI testing, we need to improve macro player functionality.
After a nice discussion with Stefan Hundhammer and Chistopher Hofmann, we got clear that we cannot simply resurrect functionality which was there already as it's based on YCP, which YaST team wants to replace for quite a while. Our QA team feels fancy to help with the progress in order to get scalable framework for YaST GUI testing, including installer. However, as you have better insights, we need your help to decide steps to be done and split them to the items we can deliver during the sprint. This will allow us slowly moving forward.
Questions to clarify before we start:
1) Will YCP be dropped in foreseeable future, if yes what shall be used instead? yaml+ruby? Do you mean the programming language called YCP or the components system called YCP?
I actually meant both, as my understanding was that Communication Protocol is also a subject to change.
Well, for communication protocol I worry it is very very long term goal, as it is just data structures which can be in future replaced by direct ruby structures or something like DBus which has own data structures that is language agnostic. ( or rest API as lada mention ). So drop of language is actually almost done, but drop of communication protocol is in very far future.
In general, it doesn't matter what will it be, my main motivation is to find solution which fits all our needs.
Can you write down all your needs? It would be good to have it written down with some use cases, so we can document it and also regularly check if it is still true and if we are heading towards it? Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 02. 10. 18 v 11:18 Lukas Ocilka napsal(a):
-------- Forwarded Message -------- Subject: YaST GUI testing framework Date: Mon, 24 Sep 2018 18:37:18 +0200 From: Rodion Iafarov
To: yast-devel@opensuse.org Dear YaSTees,
I would like to kindly ask for your help. In order to make next game changing step for YaST GUI testing, we need to improve macro player functionality.
My first question would be: is the macro player good enough? If we want to invest some time in this area, maybe we could use something else.
In case we cannot get enough support and development is too heavy, we may use squish or ldtp, which have certain limitations, but will server the purpose.
I have spent two hackweeks developing an automatic YaST UI testing framework. It's based on a REST API (HTTP with JSON format), it can not only send the user input actions but can also test the content of the current dialog. The advantage is that the REST API is not bound to any programming language, you can write a client driving an YaST installation in any language. With help of "curl" and "jq" it would be possible even in shell... (but I'd avoid it) The YaST UI actions which can be done by user are quite limited (e.g. you can press a button, select a checkbox, write something to an input field... and that's basically it). So made a simple Cucumber wrapper which defines such steps. Of course, if you do not like Cucumber you can use any other framework, you could extend the current openQA or start something new from scratch. That's a big advantage of the generic REST API, if the testing frontend does not fit any more you can easily change it without affecting YaST itself. We could even use a different frontend in our CI (Travis/Jenkins) and different one in openQA built on top of the same REST backend. See more details about the REST API in my hackweek blog posts [1], [2]. If you want to give it a try I have prepared RPM packages for SLE15/Leap15 in [3], how to install the packages is described in the Cucumber wrapper at GitHub [4]. (Note: I built the packages several months ago, if something does not work now just ping me.) This project is far from complete, there are still some missing parts like support for some specific widgets, security improvements (authentication and encryption, isolating to a separate plugin so the REST API can be installed only when needed, not included in the default installation, etc...). But the prototype is already able to test the default openSUSE/SLE installation well, (see the screencasts in the blog) so I think it would make sense to at least think about this possibility as an alternative to the UI macro player. If you have any questions about this project just ask me. Note: Because this REST API is available for any libyui application and is not YaST specific we could possibly cooperate with the Mageia (formerly Mandriva, formerly Mandrake) developers, their management tools also use the libyui [5] and having a testing framework would probably help them as well... Ladislav [1] https://blog.ladslezak.cz/2018/07/13/hackweek-17/ [2] https://blog.ladslezak.cz/2017/12/06/hackweek-16-yast-ui-rest-api/ [3] https://build.opensuse.org/project/show/home:lslezak:libyui-rest-api [4] https://github.com/lslezak/cucumber-yast [5] https://madb.mageia.org/package/show/name/manatools -- Best Regards Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/ -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 03. 10. 18 v 11:00 Ladislav Slezak napsal(a):
I have spent two hackweeks developing an automatic YaST UI testing framework. It's based on a REST API (HTTP with JSON format), it can not only send the user input actions but can also test the content of the current dialog.
The advantage is that the REST API is not bound to any programming language, you can write a client driving an YaST installation in any language. With help of "curl" and "jq" it would be possible even in shell... (but I'd avoid it)
I'll definitely support THIS way of development. It solves more things at once. Not only automated testing. Can it run the "UI" actually WITHOUT the UI, just the libyui layer, but without qt/ncurses called? E.g., libyui-rest-api?
The YaST UI actions which can be done by user are quite limited (e.g. you can press a button, select a checkbox, write something to an input field... and that's basically it). So made a simple Cucumber wrapper which defines such steps.
Yes, select a radio button, edit (editable) combo-box, but in general, it's true.
Of course, if you do not like Cucumber you can use any other framework, you could extend the current openQA or start something new from scratch. That's a big advantage of the generic REST API, if the testing frontend does not fit any more you can easily change it without affecting YaST itself. We could even use a different frontend in our CI (Travis/Jenkins) and different one in openQA built on top of the same REST backend.
I'm wondering why I haven't asked for continuing on the project out-of-hackweek? Probably bugs and features are the reason.
See more details about the REST API in my hackweek blog posts [1], [2].
If you want to give it a try I have prepared RPM packages for SLE15/Leap15 in [3], how to install the packages is described in the Cucumber wrapper at GitHub [4].
(Note: I built the packages several months ago, if something does not work now just ping me.)
This project is far from complete, there are still some missing parts like support for some specific widgets, security improvements (authentication and encryption, isolating to a separate plugin so the REST API can be installed only when needed, not included in the default installation, etc...).
And it can bring what people wanted long time ago: remote management, mass-management, ... it HAS TO BE a plugin only, though.
But the prototype is already able to test the default openSUSE/SLE installation well, (see the screencasts in the blog) so I think it would make sense to at least think about this possibility as an alternative to the UI macro player.
If you have any questions about this project just ask me.
When you can finish that :)?
Note: Because this REST API is available for any libyui application and is not YaST specific we could possibly cooperate with the Mageia (formerly Mandriva, formerly Mandrake) developers, their management tools also use the libyui [5] and having a testing framework would probably help them as well...
Cool stuff, cool ideas, I must have been blind that I haven't seen the potential as I see it now :) Thanks! Lukas -- Lukas Ocilka, Systems Management Team Leader & YaST Product Owner SLE Department, SUSE Linux 🌲 Please consider the environment before printing this e-mail ☂ Handle with care - Your reply can be stored in the cloud ⚠ IMPORTANT: The contents of this email and any attachments are confidential. They are intended for the named recipient(s) only. If you have received this email by mistake, please, notify the sender immediately and do not disclose the contents to anyone or make copies of thereof. -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 03. 10. 18 v 12:15 Lukas Ocilka napsal(a):
Dne 03. 10. 18 v 11:00 Ladislav Slezak napsal(a):
I'll definitely support THIS way of development. It solves more things at once. Not only automated testing. Can it run the "UI" actually WITHOUT the UI, just the libyui layer, but without qt/ncurses called? E.g., libyui-rest-api?
Unfortunately not, the REST API needs to be hooked deep into the respective UI implementation (libyui-qt/libyui-ncurses), it cannot work on the shared level (libyui). You would need some new dummy UI implementation for that. But we already have one dummy UI for running the old tests but that's "dumb dummy" UI, it does not track the UI state (widget values) but always returns nil for what ever you ask it. You would need to add code to keep the widget state. But I think the easier solution would be to simply use some fake the environment for the qt or ncurses UI. Like using xvfb/xvnc for Qt or screen/tmux for ncurses UI. You would run the YaST modules with regular UI but hidden in background.
The YaST UI actions which can be done by user are quite limited (e.g. you can press a button, select a checkbox, write something to an input field... and that's basically it). So made a simple Cucumber wrapper which defines such steps.
Yes, select a radio button, edit (editable) combo-box, but in general, it's true.
The main point is that the list of actions and results is limited and is very small. With unit tests the actions and expected results are basically unlimited.
I'm wondering why I haven't asked for continuing on the project out-of-hackweek? Probably bugs and features are the reason.
As usually... The problem is that we are developing several products in parallel (SLE15-SP1/Leap15.1, SLE11-SP4, Kubic/CaaSP) so there less time for some long running projects like this.
If you have any questions about this project just ask me.
When you can finish that :)?
Um, it's mostly working, but you know the 80/20 rule, don't you? :-)
Note: Because this REST API is available for any libyui application and is not YaST specific we could possibly cooperate with the Mageia (formerly Mandriva, formerly Mandrake) developers, their management tools also use the libyui [5] and having a testing framework would probably help them as well...
Cool stuff, cool ideas, I must have been blind that I haven't seen the potential as I see it now :)
I hope spreading the word about this project could attract some external contributors. -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
-------- Forwarded Message -------- Subject: YaST GUI testing framework Date: Mon, 24 Sep 2018 18:37:18 +0200 From: Rodion Iafarov
To: yast-devel@opensuse.org Dear YaSTees,
I would like to kindly ask for your help. In order to make next game changing step for YaST GUI testing, we need to improve macro player functionality. My first question would be: is the macro player good enough? If we want to invest some time in this area, maybe we could use something else. Yes, as for the recorder it could speed up test development a bit, but in general we just need player. In case we cannot get enough support and development is too heavy, we may use squish or ldtp, which have certain limitations, but will server the purpose. I have spent two hackweeks developing an automatic YaST UI testing
Dne 02. 10. 18 v 11:18 Lukas Ocilka napsal(a): framework. It's based on a REST API (HTTP with JSON format), it can not only send the user input actions but can also test the content of the current dialog.
The advantage is that the REST API is not bound to any programming language, you can write a client driving an YaST installation in any language. With help of "curl" and "jq" it would be possible even in shell... (but I'd avoid it)
The YaST UI actions which can be done by user are quite limited (e.g. you can press a button, select a checkbox, write something to an input field... and that's basically it). So made a simple Cucumber wrapper which defines such steps. Unfortunately, this was initial reason why we still on current state of
On 10/3/18 11:00 AM, Ladislav Slezak wrote: things. Even with simplest yast module (e.g. hostnames), we need interaction with tables, selecting rows, checking values in the cells, etc. So this is a must to proceed. That's also the reason why we can consider other tools which can already operate on qt.
Of course, if you do not like Cucumber you can use any other framework, you could extend the current openQA or start something new from scratch. That's a big advantage of the generic REST API, if the testing frontend does not fit any more you can easily change it without affecting YaST itself. We could even use a different frontend in our CI (Travis/Jenkins) and different one in openQA built on top of the same REST backend.
See more details about the REST API in my hackweek blog posts [1], [2].
If you want to give it a try I have prepared RPM packages for SLE15/Leap15 in [3], how to install the packages is described in the Cucumber wrapper at GitHub [4].
(Note: I built the packages several months ago, if something does not work now just ping me.)
This project is far from complete, there are still some missing parts like support for some specific widgets, security improvements (authentication and encryption, isolating to a separate plugin so the REST API can be installed only when needed, not included in the default installation, etc...).
But the prototype is already able to test the default openSUSE/SLE installation well, (see the screencasts in the blog) so I think it would make sense to at least think about this possibility as an alternative to the UI macro player.
If you have any questions about this project just ask me.
I've actually seen those two projects and they look really promising, so I'm with Lukas here, seems like a good solution. However, I started this discussion to confirm preferable way, so we could collaborate.
Note: Because this REST API is available for any libyui application and is not YaST specific we could possibly cooperate with the Mageia (formerly Mandriva, formerly Mandrake) developers, their management tools also use the libyui [5] and having a testing framework would probably help them as well...
Ladislav
[1] https://blog.ladslezak.cz/2018/07/13/hackweek-17/ [2] https://blog.ladslezak.cz/2017/12/06/hackweek-16-yast-ui-rest-api/ [3] https://build.opensuse.org/project/show/home:lslezak:libyui-rest-api [4] https://github.com/lslezak/cucumber-yast [5] https://madb.mageia.org/package/show/name/manatools
--
Best Regards
Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/
--
Kind regards,
Rodion Iafarov
On Thu, 4 Oct 2018 09:02:46 +0200
Rodion Iafarov
-------- Forwarded Message -------- Subject: YaST GUI testing framework Date: Mon, 24 Sep 2018 18:37:18 +0200 From: Rodion Iafarov
To: yast-devel@opensuse.org Dear YaSTees,
I would like to kindly ask for your help. In order to make next game changing step for YaST GUI testing, we need to improve macro player functionality. My first question would be: is the macro player good enough? If we want to invest some time in this area, maybe we could use something else. Yes, as for the recorder it could speed up test development a bit, but in general we just need player. In case we cannot get enough support and development is too heavy, we may use squish or ldtp, which have certain limitations, but will server the purpose. I have spent two hackweeks developing an automatic YaST UI testing
Dne 02. 10. 18 v 11:18 Lukas Ocilka napsal(a): framework. It's based on a REST API (HTTP with JSON format), it can not only send the user input actions but can also test the content of the current dialog.
The advantage is that the REST API is not bound to any programming language, you can write a client driving an YaST installation in any language. With help of "curl" and "jq" it would be possible even in shell... (but I'd avoid it)
The YaST UI actions which can be done by user are quite limited (e.g. you can press a button, select a checkbox, write something to an input field... and that's basically it). So made a simple Cucumber wrapper which defines such steps. Unfortunately, this was initial reason why we still on current state of
On 10/3/18 11:00 AM, Ladislav Slezak wrote: things. Even with simplest yast module (e.g. hostnames), we need interaction with tables, selecting rows, checking values in the cells, etc. So this is a must to proceed. That's also the reason why we can consider other tools which can already operate on qt.
Problem of tools operating only on qt is that we really have to test also ncurses as many corporate customers uses ncurses ( info we get from support and sale guys ), so it is also very vital to test it ( like that everything fits into screen, all is accessible, no trash in output and so on ).
Of course, if you do not like Cucumber you can use any other framework, you could extend the current openQA or start something new from scratch. That's a big advantage of the generic REST API, if the testing frontend does not fit any more you can easily change it without affecting YaST itself. We could even use a different frontend in our CI (Travis/Jenkins) and different one in openQA built on top of the same REST backend.
See more details about the REST API in my hackweek blog posts [1], [2].
If you want to give it a try I have prepared RPM packages for SLE15/Leap15 in [3], how to install the packages is described in the Cucumber wrapper at GitHub [4].
(Note: I built the packages several months ago, if something does not work now just ping me.)
This project is far from complete, there are still some missing parts like support for some specific widgets, security improvements (authentication and encryption, isolating to a separate plugin so the REST API can be installed only when needed, not included in the default installation, etc...).
But the prototype is already able to test the default openSUSE/SLE installation well, (see the screencasts in the blog) so I think it would make sense to at least think about this possibility as an alternative to the UI macro player.
If you have any questions about this project just ask me.
I've actually seen those two projects and they look really promising, so I'm with Lukas here, seems like a good solution.
However, I started this discussion to confirm preferable way, so we could collaborate.
I think first step can be to write down your use cases and also your requirements, so hopefully we can somehow prioritize work on it. Josef
Note: Because this REST API is available for any libyui application and is not YaST specific we could possibly cooperate with the Mageia (formerly Mandriva, formerly Mandrake) developers, their management tools also use the libyui [5] and having a testing framework would probably help them as well...
Ladislav
[1] https://blog.ladslezak.cz/2018/07/13/hackweek-17/ [2] https://blog.ladslezak.cz/2017/12/06/hackweek-16-yast-ui-rest-api/ [3] https://build.opensuse.org/project/show/home:lslezak:libyui-rest-api [4] https://github.com/lslezak/cucumber-yast [5] https://madb.mageia.org/package/show/name/manatools
--
Best Regards
Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Thanks for the reply, I decided to reply to this e-mail and not to previous one to keep answers in one place. On 10/4/18 9:15 AM, Josef Reidinger wrote:
On Thu, 4 Oct 2018 09:02:46 +0200 Rodion Iafarov
wrote: -------- Forwarded Message -------- Subject: YaST GUI testing framework Date: Mon, 24 Sep 2018 18:37:18 +0200 From: Rodion Iafarov
To: yast-devel@opensuse.org Dear YaSTees,
I would like to kindly ask for your help. In order to make next game changing step for YaST GUI testing, we need to improve macro player functionality. My first question would be: is the macro player good enough? If we want to invest some time in this area, maybe we could use something else. Yes, as for the recorder it could speed up test development a bit, but in general we just need player. In case we cannot get enough support and development is too heavy, we may use squish or ldtp, which have certain limitations, but will server the purpose. I have spent two hackweeks developing an automatic YaST UI testing
Dne 02. 10. 18 v 11:18 Lukas Ocilka napsal(a): framework. It's based on a REST API (HTTP with JSON format), it can not only send the user input actions but can also test the content of the current dialog.
The advantage is that the REST API is not bound to any programming language, you can write a client driving an YaST installation in any language. With help of "curl" and "jq" it would be possible even in shell... (but I'd avoid it)
The YaST UI actions which can be done by user are quite limited (e.g. you can press a button, select a checkbox, write something to an input field... and that's basically it). So made a simple Cucumber wrapper which defines such steps. Unfortunately, this was initial reason why we still on current state of
On 10/3/18 11:00 AM, Ladislav Slezak wrote: things. Even with simplest yast module (e.g. hostnames), we need interaction with tables, selecting rows, checking values in the cells, etc. So this is a must to proceed. That's also the reason why we can consider other tools which can already operate on qt. Problem of tools operating only on qt is that we really have to test also ncurses as many corporate customers uses ncurses ( info we get from support and sale guys ), so it is also very vital to test it ( like that everything fits into screen, all is accessible, no trash in output and so on ). That's true, but we need to start somewhere. If we cover qt for functional, we can at least limit coverage with ncurses, as performing same action will trigger same code underneath.
Of course, if you do not like Cucumber you can use any other framework, you could extend the current openQA or start something new from scratch. That's a big advantage of the generic REST API, if the testing frontend does not fit any more you can easily change it without affecting YaST itself. We could even use a different frontend in our CI (Travis/Jenkins) and different one in openQA built on top of the same REST backend.
See more details about the REST API in my hackweek blog posts [1], [2].
If you want to give it a try I have prepared RPM packages for SLE15/Leap15 in [3], how to install the packages is described in the Cucumber wrapper at GitHub [4].
(Note: I built the packages several months ago, if something does not work now just ping me.)
This project is far from complete, there are still some missing parts like support for some specific widgets, security improvements (authentication and encryption, isolating to a separate plugin so the REST API can be installed only when needed, not included in the default installation, etc...).
But the prototype is already able to test the default openSUSE/SLE installation well, (see the screencasts in the blog) so I think it would make sense to at least think about this possibility as an alternative to the UI macro player.
If you have any questions about this project just ask me. I've actually seen those two projects and they look really promising, so I'm with Lukas here, seems like a good solution.
However, I started this discussion to confirm preferable way, so we could collaborate. I think first step can be to write down your use cases and also your requirements, so hopefully we can somehow prioritize work on it.
These is combined list for the needs for your previous email: 1. Operating on all existing widgets, including tables, trees, comboboxes, tabs. Being able to perform same set of actions like user can do normally. 2. Possibility to assess widgets properties e.g. values in the table cells, labels text, UI properties (if control is visible/disabled) 3. Some way of integration with openQA (that's important for QA team, as we have to use it, but it should not be an issue as we can simply create file with results and parse it) 4. There is not to complex way to launch tests when running installation (e.g. booting from installer DVD and triggering test) 5. There is a way to run tests with local changes without configuring any kind of complex setup 6. Solution can be used to run CI per branch 7. Can be used for ncurses and qt GUIs 8. Running tests minimizes influence on SUT (we should keep difference of real installation vs integration one as little as possible not installing hundreds of the dependencies). As we cannot get it all, my proposal would be to start with something more simple, e.g. yast module instead of installer. But even for simplest ones e.g. timezones and hostnames, we lack functionality at the moment. Also, I would like to mention, we as QA department are really interested in the development and considered driving it with the support of the YaST team, as I fully understand lack of resources for it. But we need your support to decide on a solution and helping to create tasks so we could pick them up.
Josef
Note: Because this REST API is available for any libyui application and is not YaST specific we could possibly cooperate with the Mageia (formerly Mandriva, formerly Mandrake) developers, their management tools also use the libyui [5] and having a testing framework would probably help them as well...
Ladislav
[1] https://blog.ladslezak.cz/2018/07/13/hackweek-17/ [2] https://blog.ladslezak.cz/2017/12/06/hackweek-16-yast-ui-rest-api/ [3] https://build.opensuse.org/project/show/home:lslezak:libyui-rest-api [4] https://github.com/lslezak/cucumber-yast [5] https://madb.mageia.org/package/show/name/manatools
--
Best Regards
Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/
--
Kind regards,
Rodion Iafarov
Hello! Dne 04. 10. 18 v 9:48 Rodion Iafarov napsal(a): I'll comment the requirements from the libyui REST API prototype POV so we could see if that would fit and we could compare it with the other possibilities:
These is combined list for the needs for your previous email:
1. Operating on all existing widgets, including tables, trees, comboboxes, tabs.
The prototype does not support the complete set of widgets yet (ENOTIME) but adding support for the missing ones is quite easy. The most complex will be the package manager widget as from the YaST POV the whole dialog is a single widget like a single push button. And they are implemented in separate packages (libyui-*-pkg).
Being able to perform same set of actions like user can do normally.
It turned out that the prototype was actually more powerful than the user at the beginning, e.g. I could change the state of a disabled check box, but user cannot click it in reality. Later I added a check and disallowed changing the disabled widgets. But I think this should not be a problem, we just need to think about it, carefully compare the real UI and REST API possibilities and make it as close as possible. Josef already mentioned that it's important to check that all widgets are properly displayed etc... The REST API currently does not provide any information about the rendered widgets on the screen. But it should be possible to get some data from the layout engine or the UI itself. Then we could return additional flags like "out-of-screen" or "partially-displayed" for each widget and test in the frontend that no widget has these flags set.
2. Possibility to assess widgets properties e.g. values in the table cells, labels text, UI properties (if control is visible/disabled)
The REST API works directly on the libyui level, we can read (almost) everything. We can get much more details than it is possible from the YaST code.
3. Some way of integration with openQA (that's important for QA team, as we have to use it, but it should not be an issue as we can simply create file with results and parse it)
The integration should be simple, you just need to add some REST API client which can be implemented in any language and process the JSON response. The problem would be when testing a network-less scenario (e.g. network not connected or even no network card present) because an HTTP REST API naturally requires a network. But in that case you could connect to the REST API locally via serial console, run "curl http://localhost/..." there and pass the output further. That looks a bit annoying but should be solvable quite easy with some wrapper.
4. There is not to complex way to launch tests when running installation (e.g. booting from installer DVD and triggering test)
The prototype is already able to test the installation, you just need to set some environment variables via the boot command line to enable the REST API server (disabled by default) and that's it. Then the installation can be driven from outside via the REST API [1].
5. There is a way to run tests with local changes without configuring any kind of complex setup
It mainly depends on how you design the frontend part of the testing framework, the backend part is trivial and once we integrate it into YaST it will run out of box (or you just install few packages before using it). For example the Cucumber prototype is very easy to use, you only need Ruby and Bundler installed. Install the gems [2], prepare the environment (start a VM if needed) and you can run the tests [3].
6. Solution can be used to run CI per branch
I'm not sure what does "branch" mean in this case. Product branch like GA, SP1, SP2? Could you clarify?
7. Can be used for ncurses and qt GUIs
The prototype already supports both.
8. Running tests minimizes influence on SUT (we should keep difference of real installation vs integration one as little as possible not installing hundreds of the dependencies).
So far it just needs two additional packages (libjsoncpp and libmicrohttpd), in total it's ~170KB (packed RPMs) so the footprint is really minimal. That's even more important for the installer as the inst-sys space is very limited. In installed system you might additionally need to open the HTTP port at firewall (or you can use the same trick as with the network-less installation). Summary of the most important features: - The REST API can be used from any programming language, it's language independent - It makes a clean boundary between the frontend and the backend - The internal JSON format is human readable, we could store the responses for later debugging if needed - The frontend may use any testing framework (Cucumber, RSpec, ... or a custom one), it can be easily updated/changed - Small dependencies for the backend - Works with both Qt and ncurses [1] https://github.com/lslezak/cucumber-yast#testing-the-yast-installer [2] https://github.com/lslezak/cucumber-yast#installing-cucumber-ruby-gem [3] https://github.com/lslezak/cucumber-yast#running-the-tests -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Hi Lada! Thanks for the detailed answers, it looks very promising to me. I will try to do some practice exercise with it to get better understanding of how it works. On 10/5/18 12:43 PM, Ladislav Slezak wrote:
Hello!
Dne 04. 10. 18 v 9:48 Rodion Iafarov napsal(a):
I'll comment the requirements from the libyui REST API prototype POV so we could see if that would fit and we could compare it with the other possibilities:
These is combined list for the needs for your previous email:
1. Operating on all existing widgets, including tables, trees, comboboxes, tabs. The prototype does not support the complete set of widgets yet (ENOTIME) but adding support for the missing ones is quite easy. The most complex will be the package manager widget as from the YaST POV the whole dialog is a single widget like a single push button. And they are implemented in separate packages (libyui-*-pkg).
Being able to perform same set of actions like user can do normally. It turned out that the prototype was actually more powerful than the user at the beginning, e.g. I could change the state of a disabled check box, but user cannot click it in reality. Later I added a check and disallowed changing the disabled widgets.
But I think this should not be a problem, we just need to think about it, carefully compare the real UI and REST API possibilities and make it as close as possible.
Josef already mentioned that it's important to check that all widgets are properly displayed etc... The REST API currently does not provide any information about the rendered widgets on the screen.
But it should be possible to get some data from the layout engine or the UI itself. Then we could return additional flags like "out-of-screen" or "partially-displayed" for each widget and test in the frontend that no widget has these flags set.
2. Possibility to assess widgets properties e.g. values in the table cells, labels text, UI properties (if control is visible/disabled) The REST API works directly on the libyui level, we can read (almost) everything. We can get much more details than it is possible from the YaST code.
3. Some way of integration with openQA (that's important for QA team, as we have to use it, but it should not be an issue as we can simply create file with results and parse it) The integration should be simple, you just need to add some REST API client which can be implemented in any language and process the JSON response.
The problem would be when testing a network-less scenario (e.g. network not connected or even no network card present) because an HTTP REST API naturally requires a network.
But in that case you could connect to the REST API locally via serial console, run "curl http://localhost/..." there and pass the output further. That looks a bit annoying but should be solvable quite easy with some wrapper.
4. There is not to complex way to launch tests when running installation (e.g. booting from installer DVD and triggering test) The prototype is already able to test the installation, you just need to set some environment variables via the boot command line to enable the REST API server (disabled by default) and that's it. Then the installation can be driven from outside via the REST API [1].
5. There is a way to run tests with local changes without configuring any kind of complex setup It mainly depends on how you design the frontend part of the testing framework, the backend part is trivial and once we integrate it into YaST it will run out of box (or you just install few packages before using it).
For example the Cucumber prototype is very easy to use, you only need Ruby and Bundler installed. Install the gems [2], prepare the environment (start a VM if needed) and you can run the tests [3]. Right, but not to double the efforts for the test development, we need some common solution which is then executed on different stages of the development process. Therefore, it will be beneficial to agree on the front-end design to use same solution and avoid development of the same tests.
6. Solution can be used to run CI per branch I'm not sure what does "branch" mean in this case. Product branch like GA, SP1, SP2? Could you clarify?
As per one of our discussions we had during YaST workshop, we agreed that we want to test earlier. Ideally, we could apply such CI to run against feature branches, so developers can test their changes even before creating pull request. Otherwise, we still could implement CI to run against master + product branches, or at least just against master. So it's related to point 5 a bit, but is still different, as we still could use parts of openQA to spawn VMs, etc.
7. Can be used for ncurses and qt GUIs The prototype already supports both.
8. Running tests minimizes influence on SUT (we should keep difference of real installation vs integration one as little as possible not installing hundreds of the dependencies). So far it just needs two additional packages (libjsoncpp and libmicrohttpd), in total it's ~170KB (packed RPMs) so the footprint is really minimal. That's even more important for the installer as the inst-sys space is very limited.
In installed system you might additionally need to open the HTTP port at firewall (or you can use the same trick as with the network-less installation).
Summary of the most important features:
- The REST API can be used from any programming language, it's language independent - It makes a clean boundary between the frontend and the backend - The internal JSON format is human readable, we could store the responses for later debugging if needed - The frontend may use any testing framework (Cucumber, RSpec, ... or a custom one), it can be easily updated/changed - Small dependencies for the backend - Works with both Qt and ncurses
[1] https://github.com/lslezak/cucumber-yast#testing-the-yast-installer [2] https://github.com/lslezak/cucumber-yast#installing-cucumber-ruby-gem [3] https://github.com/lslezak/cucumber-yast#running-the-tests
--
Kind regards,
Rodion Iafarov
participants (5)
-
Ancor Gonzalez Sosa
-
Josef Reidinger
-
Ladislav Slezak
-
Lukas Ocilka
-
Rodion Iafarov