[yast-devel] openQA development restarted
Looks like some individuals and teams inside SUSE have coincidentally decided to give a hand to openQA development in the following weeks. It seems like a nice opportunity to put the focus into the things we miss for Yast testing. According to a meeting this morning, next goal will be to have openQA running on different architectures. We expect Dinar Valeev to make it run in POWER next week (shouldn't be a big deal). There are also chances of Bernhard Wiedemann putting some effort into bringing the real hardware (kvm2usb) support back. Once we have tests for POWER running succesfully, the next big thing would be aggregate the results of different architectures in just one openQA webUI (right now, workers and webUI must share the filesystem, which means a separate host with its webUI per architecture). The cheap way would be some mechanism for exporting/importing results, so we simply integrate the results of the POWER openQA into the main one. The really good way would be having distributed workers with just one central point for coordination and webUI. The former openSUSE team never had enough time and manpower to do that and always considered this to be a nice use case for SUSE's cloud technologies. Looks like now the Cloud Team, Dinar and some people from QA are eager to help in that regard. The goal of this mail is not only reporting what can we expect from the nearest future, but to collect more ideas. According to our recent thread about integration tests, we also want: - VNC (already implemented, actually). - Resurrect the OCR support. - Use cloud for parallelism and snapshots (already considered, as explained above). - Tree of requirements/snapshots, with information how to get to every state. - Improved debugging capabilities. I'd say that most of them (if not all) are already in the openQA project tracker. But there are way too many things there, so we need to rise the relevant things up. https://progress.opensuse.org/projects/openqav3/issues?per_page=100 So, time for wish-listing :-) Cheers. PS.- Quite some time ago, I did for openQA the same we are doing now for Yast, so if you are looking for a central piece of information about openQA, visit the documentation section of the landing page at http://os-autoinst.github.io/openQA/ -- Ancor González Sosa openSUSE Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Thu, Oct 02, 2014 at 01:30:04PM +0200, Ancor Gonzalez Sosa wrote:
The goal of this mail is not only reporting what can we expect from the nearest future, but to collect more ideas. According to our recent thread about integration tests, we also want:
- Easy way to install additional packages in the test. Since currently the repositories are not registered it is difficult for a test to do so. The test does not know whether it's running on openSUSE, staging:X or SLES12:Update:Test. - Easy way to run a bunch of scripts, that is without having to deal with needles. E.g. the scripts should simply report success/failure via the exit status. - Run tests not only for openSUSE Factory but also for SLE and all updates. - openQA itself is not so easy to work with, e.g. finding out why the tests do not start. Also several workarounds were required last time I tried it, e.g. "ip -6 r a to unreachable 2600:806:310::100/128". - Possibility to run tests with selfmade RPMs. E.g. when I get a pull request from the community I want to see that all tests still work even before merging the pull request on git and making a submit request in the buildservice. ciao Arvin -- Arvin Schnell, <aschnell@suse.de> Senior Software Engineer, Research & Development SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Thu, 02 Oct 2014 13:30:04 +0200 Ancor Gonzalez Sosa <ancor@suse.de> wrote:
Looks like some individuals and teams inside SUSE have coincidentally decided to give a hand to openQA development in the following weeks. It seems like a nice opportunity to put the focus into the things we miss for Yast testing.
According to a meeting this morning, next goal will be to have openQA running on different architectures. We expect Dinar Valeev to make it run in POWER next week (shouldn't be a big deal). There are also chances of Bernhard Wiedemann putting some effort into bringing the real hardware (kvm2usb) support back.
Hi Ancor, thanks for bringing it attention. At first I am not sure if kvm2usb is good way. I think nicer way is to use special hardware similar like customers so serial console + vnc or ssh with forwarded X
Once we have tests for POWER running succesfully, the next big thing would be aggregate the results of different architectures in just one openQA webUI (right now, workers and webUI must share the filesystem, which means a separate host with its webUI per architecture). The cheap way would be some mechanism for exporting/importing results, so we simply integrate the results of the POWER openQA into the main one. The really good way would be having distributed workers with just one central point for coordination and webUI. The former openSUSE team never had enough time and manpower to do that and always considered this to be a nice use case for SUSE's cloud technologies. Looks like now the Cloud Team, Dinar and some people from QA are eager to help in that regard.
yes, this will be very cool. Similar architecture we use for jenkins where main server only check if new run is needed and collect results, but tasks are done on runners.
The goal of this mail is not only reporting what can we expect from the nearest future, but to collect more ideas. According to our recent thread about integration tests, we also want:
- VNC (already implemented, actually).
Really use VNC for all tasks. I hope that vnc can bring easy real hardware testing.
- Resurrect the OCR support.
Maybe research if it really makes tests more robust, so we can have tasks like click OK button, write "abc" to text entry with label label1 and so on.
- Use cloud for parallelism and snapshots (already considered, as explained above). - Tree of requirements/snapshots, with information how to get to every state. - Improved debugging capabilities.
I will add thinks like - hooks to run it when OBS build finished ( like if we create our own testing iso ) - callbacks ( so we can configure it to send mail to mailing list if yast specific test failed or to notify jenkins that new build passed, so we can delay submit request till openQA confirm we do not break anything ). - easier needle modification ( or how they call screenshots :) I think on side like send pull request, someone confirm that it really should look like it and merge it. - user management
I'd say that most of them (if not all) are already in the openQA project tracker. But there are way too many things there, so we need to rise the relevant things up. https://progress.opensuse.org/projects/openqav3/issues?per_page=100
So, time for wish-listing :-)
Cheers.
PS.- Quite some time ago, I did for openQA the same we are doing now for Yast, so if you are looking for a central piece of information about openQA, visit the documentation section of the landing page at http://os-autoinst.github.io/openQA/
nice, thanks for sharing Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 10/02/2014 02:01 PM, Josef Reidinger wrote: > On Thu, 02 Oct 2014 13:30:04 +0200 > Ancor Gonzalez Sosa <ancor@suse.de> wrote: > >> [...] >> According to a meeting this morning, next goal will be to have openQA >> running on different architectures. We expect Dinar Valeev to make it >> run in POWER next week (shouldn't be a big deal). There are also >> chances of Bernhard Wiedemann putting some effort into bringing the >> real hardware (kvm2usb) support back. >> > Hi Ancor, > thanks for bringing it attention. > At first I am not sure if kvm2usb is good way. I think nicer way is to > use special hardware similar like customers so serial console + vnc or > ssh with forwarded X That's all out of my area of knowledge, but make sure you have a talk with Bernhard, I think you both have a very clear view of what is needed and the available options. >> - Use cloud for parallelism and snapshots (already considered, as >> explained above). >> - Tree of requirements/snapshots, with information how to get to >> every state. >> - Improved debugging capabilities. > I will add thinks like > - hooks to run it when OBS build finished ( like if we create our own > testing iso ) > - callbacks ( so we can configure it to send mail to mailing list if > yast specific test failed or to notify jenkins that new build passed, > so we can delay submit request till openQA confirm we do not break > anything ). > - easier needle modification ( or how they call screenshots :) I think > on side like send pull request, someone confirm that it really should > look like it and merge it. > - user management We have both user management and a visual editor of needles. As user you can be an admin or an operator. Operators have access to the needle editor in which you can modify needles in the browser (just on a failed test, right now). When clicking on "save" in the editor, a new needle will be created in the local instance of openQA and a pull request will be sent to the corresponding repository so the new needle is added to Github. Then, you can click on the restart button to restart the test using the new needles. If somebody want to try, just log into https://openqa.opensuse.org and ping me (I'm admin, so I can promote you to operators). Cheers. -- Ancor González Sosa openSUSE Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 10/02/2014 02:17 PM, Ancor Gonzalez Sosa wrote: > On 10/02/2014 02:01 PM, Josef Reidinger wrote: >> On Thu, 02 Oct 2014 13:30:04 +0200 >> Ancor Gonzalez Sosa <ancor@suse.de> wrote: >> >>> [...] >>> According to a meeting this morning, next goal will be to have openQA >>> running on different architectures. We expect Dinar Valeev to make it >>> run in POWER next week (shouldn't be a big deal). There are also >>> chances of Bernhard Wiedemann putting some effort into bringing the >>> real hardware (kvm2usb) support back. >>> >> Hi Ancor, >> thanks for bringing it attention. >> At first I am not sure if kvm2usb is good way. I think nicer way is to >> use special hardware similar like customers so serial console + vnc or >> ssh with forwarded X > That's all out of my area of knowledge, but make sure you have a talk > with Bernhard, I think you both have a very clear view of what is needed > and the available options. > >>> - Use cloud for parallelism and snapshots (already considered, as >>> explained above). >>> - Tree of requirements/snapshots, with information how to get to >>> every state. >>> - Improved debugging capabilities. >> I will add thinks like >> - hooks to run it when OBS build finished ( like if we create our own >> testing iso ) >> - callbacks ( so we can configure it to send mail to mailing list if >> yast specific test failed or to notify jenkins that new build passed, >> so we can delay submit request till openQA confirm we do not break >> anything ). >> - easier needle modification ( or how they call screenshots :) I think >> on side like send pull request, someone confirm that it really should >> look like it and merge it. >> - user management > We have both user management and a visual editor of needles. As user you > can be an admin or an operator. Operators have access to the needle > editor in which you can modify needles in the browser (just on a failed > test, right now). Well, to be precise, we also have the interactive mode, in which openQA stops the test and open the needle editor in every matching attempt. > When clicking on "save" in the editor, a new needle > will be created in the local instance of openQA and a pull request will > be sent to the corresponding repository so the new needle is added to > Github. Then, you can click on the restart button to restart the test > using the new needles. If somebody want to try, just log into > https://openqa.opensuse.org and ping me (I'm admin, so I can promote you > to operators). > > Cheers. > -- Ancor González Sosa openSUSE Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
participants (3)
-
Ancor Gonzalez Sosa -
Arvin Schnell -
Josef Reidinger