Am Mittwoch, 11. Dezember 2013, 00:25:08 schrieb Adrian Schröter:

Am Dienstag, 10. Dezember 2013, 21:47:27 schrieb Michal Marek:

...

Dne 6.12.2013 09:50, Adrian Schröter napsal(a): ...

...

The reason why I want to have this spec file as part of the kernel package is that it should be also branched, when someone is branching any kernel-* package. OBS does create branches for all local linked packages, so the new and modified kernel will be used also directly in this project branching the kernel.

So if I upload a kernel that panics during boot to my home project, the whole project will stop building forever?

yes

...

For me and I guess most other developers on this list, it is actually an advantage that the kernel does not need itself to build.

it won't be different as like updating to a broken glibc. We would notice before it got merged into mainline. Otherwise our users would be the first who notice that the kernel is not booting.

And there would be a way out of this, kernel developers (mainline and people who branched it) would just able to call "osc wipebinaries". If we have it only in mainline you would rely on OBS admins to fix the situation, no one else would be able to do it.

So, I would really like to ask to re-think this position. it is just one small text file in addition. And I am happy to take over the responsibility in case there are bug reports for it.

re-think sounds harsh, sorry it was late. It was a friendly request to re-consider :) Also, there would be also a way out without OBS admins, but not so obvious. However, I still think that testing the kernel before submission is still the way to go. It fits to the recently discussion about improving factory quality. A kernel which gets submited should have been at least tested once in some proven way. Also we would break some builds in future by branching a kernel into a project, because we will rely on the running kernel matching the installed kernel modules. And people would need to know the hidden magic behind kernel-obs-build package to branch it manually in addition. But with the package as part of the kernel-source, you can just fix the sources and enable builds again via wipebinaries (since it would use the kernel from the project below, eg. openSUSE:Factory). But the breakage would have been noticed. good morning adrian -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

Am Mittwoch, 11. Dezember 2013, 09:27:29 schrieb Michal Marek:

Dne 11.12.2013 00:25, Adrian Schröter napsal(a):

...

Am Dienstag, 10. Dezember 2013, 21:47:27 schrieb Michal Marek:

...

For me and I guess most other developers on this list, it is actually an advantage that the kernel does not need itself to build.

it won't be different as like updating to a broken glibc.

Yes, and that precisely is the problem - we do not want to deal with the bootstrap issues that the toolchain guys have to. For you, it's probably not such a big difference if one more package becomes part of the bootstrap set, but for us, being or not being in the bootstrap set matters.

well, since we are forced to use the distro kernel for package builds now you have also little choice. This is just going to happen to be able to load kernel modules, which are required meanwhile by some tools. The discussion where to place the OBS vm kernel spec file will not change that. It will just become even harder for you and everybody else get out of the kernel-not-booting situation again :(

...

We would notice before it got merged into mainline. Otherwise our users would be the first who notice that the kernel is not booting.

I actually plan to boot test each kernel pushed to the git, before actual users see it. But failure to boot will not affect subsequent attempts to build a fixed package. Hardware for that is ordered, it will not cover ppc64(le|be) though, only x86_64 (unless there is already usable qemu emulation for ppc64le?)

and how will eg a broken tool chain be detected or a broken mkinitrd? And how would this become visible to the other packagers and release managers? IMHO the test driven approach should really test the entire kernel eco system in a transparent. Having additional tests outside of OBS is fine, but it is unlikely that further people except you can use it when deciding to take kernel relevant changes or to work on own versions. -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

Am Mittwoch, 11. Dezember 2013, 10:41:17 schrieb Michal Marek:

Dne 11.12.2013 09:47, Adrian Schröter napsal(a):

...

Am Mittwoch, 11. Dezember 2013, 09:27:29 schrieb Michal Marek:

...

Dne 11.12.2013 00:25, Adrian Schröter napsal(a):

...

Am Dienstag, 10. Dezember 2013, 21:47:27 schrieb Michal Marek:

...

For me and I guess most other developers on this list, it is actually an advantage that the kernel does not need itself to build.

it won't be different as like updating to a broken glibc.

Yes, and that precisely is the problem - we do not want to deal with the bootstrap issues that the toolchain guys have to. For you, it's probably not such a big difference if one more package becomes part of the bootstrap set, but for us, being or not being in the bootstrap set matters.

well, since we are forced to use the distro kernel for package builds now you have also little choice. This is just going to happen to be able to load kernel modules, which are required meanwhile by some tools.

That's fine. I only object to forcing the bootstrap on everyone who wants to build their kernel.

We can declare the kernel-obs-build package as "Support" package. Means, it does not trigger any builds actively. So there would not be a bootstrap cycle. Would that help?

...

The discussion where to place the OBS vm kernel spec file will not change that. It will just become even harder for you and everybody else get out of the kernel-not-booting situation again :(

Well, what would most likely happen is that the automated kernel builds in the Kernel:$branch projects would exclude the kernel-obs spec file to avoid this situation, so the QA gain would be minimal.

Erm, actually I would object against this. Even when the spec file is not part of kernel-sources, we should have at least the kernel-obs-build package in these projects or the first breakage would happen in openSUSE:Factory. (but see below first maybe :)

Only individual developers who build their kernel in the OBS/IBS would be tricked into the bootstrap. But we have scripts even for that, so...

And they would need to know the additional manual branch call which could be avoided when all spec files stick together. Also it would avoid some "if $package =~ kernel; else ..." statements in some external check scripts to handle just the kernel package differently.

...

...

I actually plan to boot test each kernel pushed to the git, before actual users see it. But failure to boot will not affect subsequent attempts to build a fixed package. Hardware for that is ordered, it will not cover ppc64(le|be) though, only x86_64 (unless there is already usable qemu emulation for ppc64le?)

and how will eg a broken tool chain be detected or a broken mkinitrd?

It would not. We have enough fun with broken kernels already ;-).

I assume that we agree that such breakages should not appear first time in Factory, right? So you say basically we need another QA project somewhere where all kernel & tool chain shall be tested before submission to factory, right? So Kernel:$branch projects stay as kernel-only projects, but the devel project for factory will become some other project? bye adrian -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

Michal Marek writes:

If I upload a new kernel to my project, the previous, possibly broken, kernel-obs-build will be used, won't it?

You can always wipe it. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

Am Mittwoch, 11. Dezember 2013, 11:39:21 schrieb Michal Marek:

Dne 11.12.2013 10:54, Adrian Schröter napsal(a):

...

Am Mittwoch, 11. Dezember 2013, 10:41:17 schrieb Michal Marek:

...

Dne 11.12.2013 09:47, Adrian Schröter napsal(a):

...

Am Mittwoch, 11. Dezember 2013, 09:27:29 schrieb Michal Marek:

...

Dne 11.12.2013 00:25, Adrian Schröter napsal(a):

...

Am Dienstag, 10. Dezember 2013, 21:47:27 schrieb Michal Marek: > For me and I guess most other > developers on this list, it is actually an advantage that the kernel > does not need itself to build.

it won't be different as like updating to a broken glibc.

Yes, and that precisely is the problem - we do not want to deal with the bootstrap issues that the toolchain guys have to. For you, it's probably not such a big difference if one more package becomes part of the bootstrap set, but for us, being or not being in the bootstrap set matters.

well, since we are forced to use the distro kernel for package builds now you have also little choice. This is just going to happen to be able to load kernel modules, which are required meanwhile by some tools.

That's fine. I only object to forcing the bootstrap on everyone who wants to build their kernel.

We can declare the kernel-obs-build package as "Support" package. Means, it does not trigger any builds actively. So there would not be a bootstrap cycle. Would that help?

If I upload a new kernel to my project, the previous, possibly broken, kernel-obs-build will be used, won't it? In that case, "Support" won't help.

Right, and that is actually the goal here that the kernel behaves exactly as all the other packages. You could disable that though via the "useforbuild" disable flag (and quite soon this flag will have also an effect afterwards).

...

...

...

...

I actually plan to boot test each kernel pushed to the git, before actual users see it. But failure to boot will not affect subsequent attempts to build a fixed package. Hardware for that is ordered, it will not cover ppc64(le|be) though, only x86_64 (unless there is already usable qemu emulation for ppc64le?)

and how will eg a broken tool chain be detected or a broken mkinitrd?

It would not. We have enough fun with broken kernels already ;-).

I assume that we agree that such breakages should not appear first time in Factory, right?

Yes. I only do not like the bootstrap (or whatever you call it) in all projects that build the kernel.

hm, this is how we work with all other packages, except the kernel. We want to use the build result to be able to verify it. And given our test driven development model, we want to see the effect before we merge submissions. We could even add some kind of additional QA package beside and, for example, check the kernel interface (eg. POSIX compliance) if we have such tests.

...

So you say basically we need another QA project somewhere where all kernel & tool chain shall be tested before submission to factory, right? So Kernel:$branch projects stay as kernel-only projects, but the devel project for factory will become some other project?

That can certainly be done. I didn't think much about Factory in my previous email, but more about the released products and SLE betas. The factory kernels are nowadays submitted by a daily cronjob, so a _separate_ staging project which does the bootstrap makes sense here.

So, in effect this staging project would replace Kernel:HEAD as devel project for openSUSE:Factory? -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

Am Mittwoch, 11. Dezember 2013, 14:52:16 schrieb Michal Marek:

Dne 11.12.2013 12:07, Adrian Schröter napsal(a):

...

Am Mittwoch, 11. Dezember 2013, 11:39:21 schrieb Michal Marek:

...

Dne 11.12.2013 10:54, Adrian Schröter napsal(a):

...

I assume that we agree that such breakages should not appear first time in Factory, right?

Yes. I only do not like the bootstrap (or whatever you call it) in all projects that build the kernel.

hm, this is how we work with all other packages, except the kernel.

Please do not exaggerate. Most other packages are not needed to build themselves.

okay, most of the _bootstrap_ packages are needed to build themselves. (and some more)

...

We want to use the build result to be able to verify it. And given our test driven development model, we want to see the effect before we merge submissions.

We could even add some kind of additional QA package beside and, for example, check the kernel interface (eg. POSIX compliance) if we have such tests.

I can add a kernel-qa.spec to the kernel git if you like. But only as long as this is a leaf package.

Hm, makes only sense when the kernel-build-obs package would be also part of kernel git and Kernel projects, otherwise this package would have no effect. Question is if we have test cases which would make sense to run automatically inside of VM's, but I am told by QA that there are some which can be applied. We do check already on submit requests, if all spec files in that package does successfull build. So we would verify automatically on a submit request to Factory, if checks in kernel-qa were successful. But again, this does currently rely that all spec files are comming from one package container.

...

...

...

So you say basically we need another QA project somewhere where all kernel & tool chain shall be tested before submission to factory, right? So Kernel:$branch projects stay as kernel-only projects, but the devel project for factory will become some other project?

That can certainly be done. I didn't think much about Factory in my previous email, but more about the released products and SLE betas. The factory kernels are nowadays submitted by a daily cronjob, so a _separate_ staging project which does the bootstrap makes sense here.

So, in effect this staging project would replace Kernel:HEAD as devel project for openSUSE:Factory?

Yes, that sounds logical.

Hm, so the workflow would be commiting kernel-source to Kernel:HEAD, submiting to Kernel:QA and from there to openSUSE:Factory? -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

Am Mittwoch, 11. Dezember 2013, 16:29:16 schrieb Michal Marek:

Dne 11.12.2013 15:27, Adrian Schröter napsal(a):

...

Am Mittwoch, 11. Dezember 2013, 14:52:16 schrieb Michal Marek:

...

Dne 11.12.2013 12:07, Adrian Schröter napsal(a):

...

We want to use the build result to be able to verify it. And given our test driven development model, we want to see the effect before we merge submissions.

We could even add some kind of additional QA package beside and, for example, check the kernel interface (eg. POSIX compliance) if we have such tests.

I can add a kernel-qa.spec to the kernel git if you like. But only as long as this is a leaf package.

Hm, makes only sense when the kernel-build-obs package would be also part of kernel git and Kernel projects, otherwise this package would have no effect.

Hm, right. Could we have kernel-qa-build.spec and kernel-qa.spec? The former would package the /.build.{kernel,initrd} files and the latter would require it and run some tests?

Yes

So we would use the freshly built kernel to build another package, but the normal kernel packages would be built using some kernel from the underlying distribution.

hm, currently a bit hard to configure. But can maybe support something like #!VMInstallIgnore: kernel-obs-build similar for #!BuildIgnore in the spec file for kernel-default.spec . I will check if we can add that... One problem will remain, it looks like we will have workers to support where no kernel and no initrd is configured for KVM builds. So we would need also a solution for that situation ...

...

We do check already on submit requests, if all spec files in that package does successfull build. So we would verify automatically on a submit request to Factory, if checks in kernel-qa were successful. But again, this does currently rely that all spec files are comming from one package container.

Yes. But it's not critical that _all_ the packages use the new kernel for building. kernel-qa would be enough, right?

in this case yes, but it is a bit hard to find out which other packages do rely on the kernel. So I would like to go with an opt-out instead of an opt-in. bye adrian -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On 13.12.2013 10:33, Adrian Schröter wrote:

After discussion with Michal during a beer and thinking again about all this, I like to propose the following:

Let's stay with one project per kernel code stream, eg, Kernel:HEAD, but let's handle the problematic of having a broken kernel breaking also the builds of the fixed kernel with seperate repositories.

This means, we have two repositories in Kernel:HEAD, for example:

standard: This builds all kernels as we have them today. The prjconf will explicit skip the installation of kernel-obs-build, so the worker fallback kernel will be used. => This avoids changes in the kernel spec files, which would make it impossible to build the kernel also using kernel-obs-build. Alternative solution would be to disable "useforbuild" flag for kernel-obs-build for this repo, so the kernel inside of openSUSE:Factory is used, which should always work.

qa: This is a second repository which just builds kernel-obs-build and some kernel-qa* package which is used to validate the kernel. We may also use the mkinitrd (or successor) from the devel project here to include this the testing for that one as well here.

I think this should work. Kernel developers will have to be taught to always use the scripts/osc_wrapper script that will exclude the kernel-obs-build package as well (TBD). Kernel:* projects are handled by scripts exclusively, so that is not a problem. Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

Dne 18.12.2013 14:34, Adrian Schröter napsal(a):

On Dienstag, 17. Dezember 2013, 13:56:45 wrote Michal Marek: ...

...

I think this should work. Kernel developers will have to be taught to always use the scripts/osc_wrapper script that will exclude the kernel-obs-build package as well (TBD). Kernel:* projects are handled by scripts exclusively, so that is not a problem.

Okay, I have prepared both kernel-obs-build and kernel-obs-qa spec files here as git commit (as requested):

https://github.com/adrianschroeter/kernel-source.git

Please tell if you need any help with the project setup later on.

Hi Adrian, sorry for the delay. I applied this to master and SLE12. Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org