[yast-devel] YaST and openQA
Cross-posting. This should be interesting to both sides. At the end of the mail there is a small glossary, since I assume some readers will not be familiar with some of the used terms. Since we (the YaST team) plan to implement the writing openQA tests into our regular development workflow, Imo and me have spent the last couples of days playing with openQA and our current development tools. We are not interested in the iso-oriented approach used at openqa.suse.de and openqa.opensuse.org. We want to run single jobs for the code we are developing right now. There are 2 kind of jobs we want to run: testing the installation and testing a YaST module (like any other application). 1) Testing the installation If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process. [1] https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/460 https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/461 https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/4 [2] https://dhcp78.suse.cz/tests/12 https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/462 2) Testing the applications The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an uploaded DUD, as the installation already does. We could share the needles repositories of openSUSE/SLE and have all the application tests in a specific directory, so the mechanism for sharing code and needles will be very similar to the one already used for SLE and openSUSE. Most of the rake tasks would be completely reused. Generating and refreshing the installed systems (an asset) used as starting point for these jobs can be easily automated thanks to PUBLISH_HDD, as you can see here[3]. The test proving that the new distri works were lost in a recent hard disk accident, but they will be available today again [3] https://dhcp78.suse.cz/tests/1 So we already have all the pieces and our first installation test (test module to be precise) for a functionality still on development. Let's open the discussion! Cheers. And now the glossary: DUD - ftp://ftp.suse.com/pub/people/hvogel/Update-Media-HOWTO/Update-Media-HOWTO.html The most convenient way to deliver updates to the system (including the installer) in a file or cd image. asset - The elements used to feed the openQA jobs. Can be an ISO, a hard disk image or a repository. They can be registered manually (like our rake tasks do) or generated by another job (like in [3] that generates the installed system to be used in other tests). rake - https://github.com/ruby/rake The tool the YaST team uses to automate the development work. distri - A set of tests in openQA. It defines the steps (a.k.a. test modules), in which order the steps are run and how to react to some environment variables. We have a separate distri for openSUSE and SLE, although they share the definition of the steps (modules). -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Am 27.05.2015 um 09:16 schrieb Ancor Gonzalez Sosa:
We are not interested in the iso-oriented approach used at openqa.suse.de and openqa.opensuse.org. We want to run single jobs for the code we are developing right now. There are 2 kind of jobs we want to run: testing the installation and testing a YaST module (like any other application).
What about linking e.g. the Core DVD into YaST:Head, sync the built iso to openQA and use the normal job templates? No need for DUD then. That would also already reveal issues in case installation-images needs to be adjusted for YaST changes.
2) Testing the applications
The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an
Something like that is also required by maintenance QA. We need to think which mechanisms to support that scenario can be made generic and built into openQA. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Wed, 27 May 2015 09:43:26 +0200 Ludwig Nussel <ludwig.nussel@suse.de> wrote:
Am 27.05.2015 um 09:16 schrieb Ancor Gonzalez Sosa:
We are not interested in the iso-oriented approach used at openqa.suse.de and openqa.opensuse.org. We want to run single jobs for the code we are developing right now. There are 2 kind of jobs we want to run: testing the installation and testing a YaST module (like any other application).
What about linking e.g. the Core DVD into YaST:Head, sync the built iso to openQA and use the normal job templates? No need for DUD then. That would also already reveal issues in case installation-images needs to be adjusted for YaST changes.
I am not sure how situation looks now, but we try similar approach when we convert YaST from ycp to ruby to ensure installer works in ruby and problem is quite often, that changes in factory and yast itself cause that DVD almost never built and sometimes it is less then once per day. Maybe now with staging when factory gets packages in groups, rebuild is not so often.
2) Testing the applications
The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an
Something like that is also required by maintenance QA. We need to think which mechanisms to support that scenario can be made generic and built into openQA.
I think simple devel repo and running `zypper dup -r <repo>` should do all we needed, as we want to also test pre and post install scripts. Josef
cu Ludwig
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 05/27/2015 09:48 AM, Josef Reidinger wrote:
On Wed, 27 May 2015 09:43:26 +0200 Ludwig Nussel <ludwig.nussel@suse.de> wrote:
Am 27.05.2015 um 09:16 schrieb Ancor Gonzalez Sosa:
We are not interested in the iso-oriented approach used at openqa.suse.de and openqa.opensuse.org. We want to run single jobs for the code we are developing right now. There are 2 kind of jobs we want to run: testing the installation and testing a YaST module (like any other application).
What about linking e.g. the Core DVD into YaST:Head, sync the built iso to openQA and use the normal job templates? No need for DUD then. That would also already reveal issues in case installation-images needs to be adjusted for YaST changes.
I am not sure how situation looks now, but we try similar approach when we convert YaST from ycp to ruby to ensure installer works in ruby and problem is quite often, that changes in factory and yast itself cause that DVD almost never built and sometimes it is less then once per day. Maybe now with staging when factory gets packages in groups, rebuild is not so often.
Generating and uploading the DUD is a piece of cake and takes just a few seconds (verified). Quite convenient during development. In addition, it can be done using a local openQA installation, so the process is totally local and cheap.
2) Testing the applications
The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an
Something like that is also required by maintenance QA. We need to think which mechanisms to support that scenario can be made generic and built into openQA.
I think simple devel repo and running `zypper dup -r <repo>` should do all we needed, as we want to also test pre and post install scripts.
The advantage of using DUD in this scenario is that is more self-contained. You upload the DUD to openQA and you use it during the openQA job execution. No need to upload anything to a real repo (that needs to be hosted somewhere), can be used with locally generated (and totally experimental) packages, it's consistent over time if you restart the job as long as you don't change the asset containing the DUD... -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
1) Testing the installation
If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.
but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...) Michal Filka -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Wed, May 27, 2015 at 10:03:13AM +0200, Michal Filka wrote:
1) Testing the installation
If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.
but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...)
And if you modify snapper we will also have to update the snapper extension. ciao Arvin -- Arvin Schnell, <aschnell@suse.de> Senior Software Engineer, Research & Development SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
El Miércoles, 27 de mayo de 2015 10:31:15 Arvin Schnell escribió:
On Wed, May 27, 2015 at 10:03:13AM +0200, Michal Filka wrote:
1) Testing the installation
If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.
but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...)
And if you modify snapper we will also have to update the snapper extension.
Ok and I guess that there's no way of doing that through a DUD. So we must add support for submitting the whole DVD image.
ciao Arvin
Thanks! -- Imobach González Sosa YaST Team at SUSE LINUX GmbH
El Miércoles, 27 de mayo de 2015 10:03:13 Michal Filka escribió:
1) Testing the installation
If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.
but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...)
Hi, At this time, for AutoYaST integration tests, we use mksusecd to include the DUD into de DVD image (Schubi could correct me if I'm wrong). Does it make any difference to just loading the DUD? Is that approach enough? Anyway, I guess that DUD approach could be valid in a lot of situations. But support submitting a custom DVD image should be kinda trivial. Thanks a lot for your feedback! -- Imobach González Sosa YaST Team at SUSE LINUX GmbH
Am 27.05.2015 um 10:47 schrieb Imobach González Sosa:
El Miércoles, 27 de mayo de 2015 10:03:13 Michal Filka escribió:
1) Testing the installation
If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process. but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...) Hi,
At this time, for AutoYaST integration tests, we use mksusecd to include the DUD into de DVD image (Schubi could correct me if I'm wrong). Does it make any difference to just loading the DUD? Is that approach enough? As long we have a second stage in AutoYaST we need a DVD image.
Anyway, I guess that DUD approach could be valid in a lot of situations. But support submitting a custom DVD image should be kinda trivial.
Thanks a lot for your feedback!
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
1) Testing the installation
If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.
but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...)
Hi,
At this time, for AutoYaST integration tests, we use mksusecd to include the DUD into de DVD image (Schubi could correct me if I'm wrong). Does it make any difference to just loading the DUD? Is that approach enough?
If you use mksusecd for replacing rpm from iso by the new one then it should work IMO Michal Filka -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Hi, Internal IT guys just gave us another (fixed) address for our openQA instance: yast-openqa.suse.cz instead of dhcp78.suse.cz On 27.5.2015 09:16, Ancor Gonzalez Sosa wrote:
Please use https://yast-openqa.suse.cz/tests/12
Please use https://yast-openqa.suse.cz/tests/1 Bye Lukas -- Lukas Ocilka, Systems Management (Yast) Team Leader SLE Department, SUSE Linux -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 05/27/2015 09:16 AM, Ancor Gonzalez Sosa wrote:
2) Testing the applications
The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an uploaded DUD, as the installation already does. We could share the needles repositories of openSUSE/SLE and have all the application tests in a specific directory, so the mechanism for sharing code and needles will be very similar to the one already used for SLE and openSUSE. Most of the rake tasks would be completely reused. Generating and refreshing the installed systems (an asset) used as starting point for these jobs can be easily automated thanks to PUBLISH_HDD, as you can see here[3]. The test proving that the new distri works were lost in a recent hard disk accident, but they will be available today again
Here is the new distri (containing just one example test at the moment) https://github.com/yast/os-autoinst-distri-yast-modules And here you can see it in action. With the YAST_HEAD parameter set for the job (yes, I know it failed, but you get the idea :-) ) https://yast-openqa.suse.cz/tests/105/ Without that parameter (this worked in the first attempt) https://yast-openqa.suse.cz/tests/104 As promised, the needles are shared with openSUSE (and potentially SLE12) and the tests that can be relevant for openqa.o.o or openqa.suse.de are in a separate yast_modules directory. Cheers. -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
participants (8)
-
Ancor Gonzalez Sosa -
Arvin Schnell -
Imobach González Sosa -
Josef Reidinger -
Ludwig Nussel -
Lukas Ocilka -
Michal Filka -
Stefan Schubert