[yast-devel] YaST and openQA

older
[yast-devel] Pending Pull Requests

Ancor Gonzalez Sosa

27 May 2015 27 May '15

07:16

Cross-posting. This should be interesting to both sides. At the end of the mail there is a small glossary, since I assume some readers will not be familiar with some of the used terms. Since we (the YaST team) plan to implement the writing openQA tests into our regular development workflow, Imo and me have spent the last couples of days playing with openQA and our current development tools. We are not interested in the iso-oriented approach used at openqa.suse.de and openqa.opensuse.org. We want to run single jobs for the code we are developing right now. There are 2 kind of jobs we want to run: testing the installation and testing a YaST module (like any other application). 1) Testing the installation If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process. [1] https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/460 https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/461 https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/4 [2] https://dhcp78.suse.cz/tests/12 https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/462 2) Testing the applications The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an uploaded DUD, as the installation already does. We could share the needles repositories of openSUSE/SLE and have all the application tests in a specific directory, so the mechanism for sharing code and needles will be very similar to the one already used for SLE and openSUSE. Most of the rake tasks would be completely reused. Generating and refreshing the installed systems (an asset) used as starting point for these jobs can be easily automated thanks to PUBLISH_HDD, as you can see here[3]. The test proving that the new distri works were lost in a recent hard disk accident, but they will be available today again [3] https://dhcp78.suse.cz/tests/1 So we already have all the pieces and our first installation test (test module to be precise) for a functionality still on development. Let's open the discussion! Cheers. And now the glossary: DUD - ftp://ftp.suse.com/pub/people/hvogel/Update-Media-HOWTO/Update-Media-HOWTO.html The most convenient way to deliver updates to the system (including the installer) in a file or cd image. asset - The elements used to feed the openQA jobs. Can be an ISO, a hard disk image or a repository. They can be registered manually (like our rake tasks do) or generated by another job (like in [3] that generates the installed system to be used in other tests). rake - https://github.com/ruby/rake The tool the YaST team uses to automate the development work. distri - A set of tests in openQA. It defines the steps (a.k.a. test modules), in which order the steps are run and how to react to some environment variables. We have a separate distri for openSUSE and SLE, although they share the definition of the steps (modules). -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Show replies by date

Ludwig Nussel

27 May 27 May

07:43

New subject: [yast-devel] Re: [openqa] YaST and openQA

Am 27.05.2015 um 09:16 schrieb Ancor Gonzalez Sosa:

...

We are not interested in the iso-oriented approach used at openqa.suse.de and openqa.opensuse.org. We want to run single jobs for the code we are developing right now. There are 2 kind of jobs we want to run: testing the installation and testing a YaST module (like any other application).

What about linking e.g. the Core DVD into YaST:Head, sync the built iso to openQA and use the normal job templates? No need for DUD then. That would also already reveal issues in case installation-images needs to be adjusted for YaST changes.

...

2) Testing the applications

The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an

Something like that is also required by maintenance QA. We need to think which mechanisms to support that scenario can be made generic and built into openQA. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Josef Reidinger

07:48

New subject: [yast-devel] Re: [openqa] YaST and openQA

On Wed, 27 May 2015 09:43:26 +0200 Ludwig Nussel wrote:

...

Am 27.05.2015 um 09:16 schrieb Ancor Gonzalez Sosa:

...
We are not interested in the iso-oriented approach used at openqa.suse.de and openqa.opensuse.org. We want to run single jobs for the code we are developing right now. There are 2 kind of jobs we want to run: testing the installation and testing a YaST module (like any other application).

What about linking e.g. the Core DVD into YaST:Head, sync the built iso to openQA and use the normal job templates? No need for DUD then. That would also already reveal issues in case installation-images needs to be adjusted for YaST changes.

I am not sure how situation looks now, but we try similar approach when we convert YaST from ycp to ruby to ensure installer works in ruby and problem is quite often, that changes in factory and yast itself cause that DVD almost never built and sometimes it is less then once per day. Maybe now with staging when factory gets packages in groups, rebuild is not so often.

...

...
2) Testing the applications

The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an

Something like that is also required by maintenance QA. We need to think which mechanisms to support that scenario can be made generic and built into openQA.

I think simple devel repo and running `zypper dup -r <repo>` should do all we needed, as we want to also test pre and post install scripts. Josef

...

cu Ludwig

-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Ancor Gonzalez Sosa

07:56

New subject: [yast-devel] Re: [openqa] YaST and openQA

On 05/27/2015 09:48 AM, Josef Reidinger wrote:

...

On Wed, 27 May 2015 09:43:26 +0200 Ludwig Nussel wrote:

...
Am 27.05.2015 um 09:16 schrieb Ancor Gonzalez Sosa:

...
We are not interested in the iso-oriented approach used at openqa.suse.de and openqa.opensuse.org. We want to run single jobs for the code we are developing right now. There are 2 kind of jobs we want to run: testing the installation and testing a YaST module (like any other application).

What about linking e.g. the Core DVD into YaST:Head, sync the built iso to openQA and use the normal job templates? No need for DUD then. That would also already reveal issues in case installation-images needs to be adjusted for YaST changes.

I am not sure how situation looks now, but we try similar approach when we convert YaST from ycp to ruby to ensure installer works in ruby and problem is quite often, that changes in factory and yast itself cause that DVD almost never built and sometimes it is less then once per day. Maybe now with staging when factory gets packages in groups, rebuild is not so often.

Generating and uploading the DUD is a piece of cake and takes just a few seconds (verified). Quite convenient during development. In addition, it can be done using a local openQA installation, so the process is totally local and cheap.

...

...
...
2) Testing the applications

The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an

Something like that is also required by maintenance QA. We need to think which mechanisms to support that scenario can be made generic and built into openQA.

I think simple devel repo and running `zypper dup -r <repo>` should do all we needed, as we want to also test pre and post install scripts.

The advantage of using DUD in this scenario is that is more self-contained. You upload the DUD to openQA and you use it during the openQA job execution. No need to upload anything to a real repo (that needs to be hosted somewhere), can be used with locally generated (and totally experimental) packages, it's consistent over time if you restart the job as long as you don't change the asset containing the DUD... -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Michal Filka

08:03

...

1) Testing the installation

If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.

but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...) Michal Filka -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Arvin Schnell

08:31

On Wed, May 27, 2015 at 10:03:13AM +0200, Michal Filka wrote:

...

...
1) Testing the installation

If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.

but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...)

And if you modify snapper we will also have to update the snapper extension. ciao Arvin -- Arvin Schnell, Senior Software Engineer, Research & Development SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Imobach González Sosa

08:48

El Miércoles, 27 de mayo de 2015 10:31:15 Arvin Schnell escribió:

...

On Wed, May 27, 2015 at 10:03:13AM +0200, Michal Filka wrote:

...
...
1) Testing the installation

If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.

but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...)

And if you modify snapper we will also have to update the snapper extension.

Ok and I guess that there's no way of doing that through a DUD. So we must add support for submitting the whole DVD image.

...

ciao Arvin

Thanks! -- Imobach González Sosa YaST Team at SUSE LINUX GmbH

Imobach González Sosa

08:47

El Miércoles, 27 de mayo de 2015 10:03:13 Michal Filka escribió:

...

...
1) Testing the installation

If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.

but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...)

Hi, At this time, for AutoYaST integration tests, we use mksusecd to include the DUD into de DVD image (Schubi could correct me if I'm wrong). Does it make any difference to just loading the DUD? Is that approach enough? Anyway, I guess that DUD approach could be valid in a lot of situations. But support submitting a custom DVD image should be kinda trivial. Thanks a lot for your feedback! -- Imobach González Sosa YaST Team at SUSE LINUX GmbH

Stefan Schubert

10:33

Am 27.05.2015 um 10:47 schrieb Imobach González Sosa:

...

El Miércoles, 27 de mayo de 2015 10:03:13 Michal Filka escribió:

...
...
1) Testing the installation

If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process. but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...) Hi,

At this time, for AutoYaST integration tests, we use mksusecd to include the DUD into de DVD image (Schubi could correct me if I'm wrong). Does it make any difference to just loading the DUD? Is that approach enough? As long we have a second stage in AutoYaST we need a DVD image.

Anyway, I guess that DUD approach could be valid in a lot of situations. But support submitting a custom DVD image should be kinda trivial.

Thanks a lot for your feedback!

-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Michal Filka

10:48

...

...
...
1) Testing the installation

If we introduce these changes (or very similar ones) in openQA[1], we will be able to generate a DUD, submit it to the openQA host and execute a single job to run the installation including our changes. We already applied the process successfully[2] and we already have rake tasks that automate the whole process.

but dud will not cover all cases needed for testing installation. It updates first stage only. If you need to test second stage (in case of SLE12 it is valid for autoyast only), you need to update installation source too (remaster iso, use addon, ...)

Hi,

At this time, for AutoYaST integration tests, we use mksusecd to include the DUD into de DVD image (Schubi could correct me if I'm wrong). Does it make any difference to just loading the DUD? Is that approach enough?

If you use mksusecd for replacing rpm from iso by the new one then it should work IMO Michal Filka -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Lukas Ocilka

08:25

Hi, Internal IT guys just gave us another (fixed) address for our openQA instance: yast-openqa.suse.cz instead of dhcp78.suse.cz On 27.5.2015 09:16, Ancor Gonzalez Sosa wrote:

...

[2] https://dhcp78.suse.cz/tests/12

Please use https://yast-openqa.suse.cz/tests/12

...

[3] https://dhcp78.suse.cz/tests/1

Please use https://yast-openqa.suse.cz/tests/1 Bye Lukas -- Lukas Ocilka, Systems Management (Yast) Team Leader SLE Department, SUSE Linux -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

Ancor Gonzalez Sosa

15:47

On 05/27/2015 09:16 AM, Ancor Gonzalez Sosa wrote:

...

2) Testing the applications

The DUD approach proved to be very useful for installation and we think can be applied to applications as well. I already have an special distri that takes an already installed system, boots it, applies all the updates from the YaST:Head repo and then run a list of application tests. My plan is to modify it to support installing packages from an uploaded DUD, as the installation already does. We could share the needles repositories of openSUSE/SLE and have all the application tests in a specific directory, so the mechanism for sharing code and needles will be very similar to the one already used for SLE and openSUSE. Most of the rake tasks would be completely reused. Generating and refreshing the installed systems (an asset) used as starting point for these jobs can be easily automated thanks to PUBLISH_HDD, as you can see here[3]. The test proving that the new distri works were lost in a recent hard disk accident, but they will be available today again

Here is the new distri (containing just one example test at the moment) https://github.com/yast/os-autoinst-distri-yast-modules And here you can see it in action. With the YAST_HEAD parameter set for the job (yes, I know it failed, but you get the idea :-) ) https://yast-openqa.suse.cz/tests/105/ Without that parameter (this worked in the first attempt) https://yast-openqa.suse.cz/tests/104 As promised, the needles are shared with openSUSE (and potentially SLE12) and the tests that can be relevant for openqa.o.o or openqa.suse.de are in a separate yast_modules directory. Cheers. -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org

3265

Age (days ago)

3265

Last active (days ago)

List overview

Download

11 comments

8 participants

participants (8)

Ancor Gonzalez Sosa
Arvin Schnell
Imobach González Sosa
Josef Reidinger
Ludwig Nussel
Lukas Ocilka
Michal Filka
Stefan Schubert