Quoting Josef Reidinger :

Hi Coolo, good article and interesting idea. I think I also commented original mail about rings. I am not sure if it is right direction how it should go, because there is still too much human interaction. I think that in optimal case all work should do machines and humans only solve exceptions or special cases. Of course it depends if we have enough build power to keep all work to machines.

My idea is much simplier:

1) for every package submission to factory create own COW copy of factory where try to build package and all of its dependencies ( so quick for leaf package and slow for core package )

2f) if this fail reject submission, then create from copy staging project and fix all problems related to failure

2o) if everything is fine, then create ISO and test it in openQA and follow to 3

Very similar to what we are proposing so far! But this stage can be avoided IMHO. We do not need ISOs to test GCC, kernel or systemd. The ISO is needed to test the installation process: YaST, KIWI or whatever tool is used in the medium. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 28.11.2013 16:53, Josef Reidinger wrote:

...

But this stage can be avoided IMHO. We do not need ISOs to test GCC, kernel or systemd. The ISO is needed to test the installation process: YaST, KIWI or whatever tool is used in the medium.

Sorry, but I completelly disagree. Maybe it is because I am in Yast team, but you can be surprised how many Yast "bugs" is caused by problems in underlaying layers. Yast start bunch of various scripts, try to manage services in systemd and a lot of similar things. So if systemd is broken or even if they incompatible change behavior Yast start failing without any changes in Yast code. And in fact it is often source of broken Yast code, that noone told us that something change in system and we do not adapt Yast code to this change.

Yeah, I agree Alberto's examples are wrong. But there are updates that are indeed hard to break yast with, e.g. chromium updates. We don't need ISOs for *that* :) Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Josef Reidinger - 17:17 28.11.13 wrote:

On Thu, 28 Nov 2013 16:56:24 +0100 Stephan Kulow wrote:

...

On 28.11.2013 16:53, Josef Reidinger wrote:

...

...

But this stage can be avoided IMHO. We do not need ISOs to test GCC, kernel or systemd. The ISO is needed to test the installation process: YaST, KIWI or whatever tool is used in the medium.

Sorry, but I completelly disagree. Maybe it is because I am in Yast team, but you can be surprised how many Yast "bugs" is caused by problems in underlaying layers. Yast start bunch of various scripts, try to manage services in systemd and a lot of similar things. So if systemd is broken or even if they incompatible change behavior Yast start failing without any changes in Yast code. And in fact it is often source of broken Yast code, that noone told us that something change in system and we do not adapt Yast code to this change.

Yeah, I agree Alberto's examples are wrong. But there are updates that are indeed hard to break yast with, e.g. chromium updates. We don't need ISOs for *that* :)

Greetings, Stephan

Yes, iso is not needed. But it would be nice if we have also automatic test for it ( of course I expect that we start with important packages to cover ). At least simple install and run it if some window appear and try to show opensuse.org page can be enough.

Well, we have automatic test for this - openQA ;-)

I agree that iso for stuff that do not affect installation is not needed. Still I think it should be covered. In past I play with something similar and having cloud for this task and midnight snapshot of latest factory is make task much easier. You simply say that you want machine with the latest snapshot, run some tests there ( like update of package, start it etc. ) and then kill machine. For non-trivial testing including browser, synchronization of repositories and creating rpm repository metadata it took around 1 hour, which is still for me acceptable time.

We hope to use openQA for something like this as well in the future... -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 28.11.2013 16:01, Tomáš Chvátal wrote:

One of the options on the table was "lets force staging projects on everything" but sadly that is too much on the OBS and we would just be boiling it with all the rebuilds. In the end we go with above way which is technically possible and we can learn and improve it on the way wrt detecting culprits.

Basically Review team will have update SR reviewing tools with possibility to group, review, and mark for qa/staging on any submission with some possibility to edit the severity/f**kupablity of the package so they can do the right decision.

But even tho I am now interested for sure to talk about it, lets wait till we

As I said: we have to talk to the OBS team about possible solutions for the "too much on the OBS" problem. Perhaps our assumption is wrong and we can do indeed what Josef suggests: create a staging project for every submission. Then the whole discussion on who reviews and decides what will be wasted time. But for now we assume certain limits on the resources we have and that means more humans required :) Greetings, Stephan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iD8DBQFSl15QwFSBhlBjoJYRAo1yAJ0UyR/1P6y806Y9j52Dhn9vNrhPswCffifv Yn8It6fPxPYl1ZhmyNfTY90= =D5nG -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am Donnerstag, 28. November 2013, 16:16:32 schrieb Stephan Kulow:

On 28.11.2013 16:01, Tomáš Chvátal wrote:

...

One of the options on the table was "lets force staging projects on everything" but sadly that is too much on the OBS and we would just be boiling it with all the rebuilds. In the end we go with above way which is technically possible and we can learn and improve it on the way wrt detecting culprits.

Basically Review team will have update SR reviewing tools with possibility to group, review, and mark for qa/staging on any submission with some possibility to edit the severity/f**kupablity of the package so they can do the right decision.

But even tho I am now interested for sure to talk about it, lets wait till we

As I said: we have to talk to the OBS team about possible solutions for the "too much on the OBS" problem. Perhaps our assumption is wrong and we can do indeed what Josef suggests: create a staging project for every submission. Then the whole discussion on who reviews and decides what will be wasted time.

we can try at least. At least as long we do not publish it and our priority handling ensures that they do not block all other people from using OBS. -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am Donnerstag, 28. November 2013, 16:31:38 schrieb Josef Reidinger:

On Thu, 28 Nov 2013 16:22:41 +0100 Adrian Schröter wrote:

...

Am Donnerstag, 28. November 2013, 16:16:32 schrieb Stephan Kulow:

...

On 28.11.2013 16:01, Tomáš Chvátal wrote:

...

One of the options on the table was "lets force staging projects on everything" but sadly that is too much on the OBS and we would just be boiling it with all the rebuilds. In the end we go with above way which is technically possible and we can learn and improve it on the way wrt detecting culprits.

Basically Review team will have update SR reviewing tools with possibility to group, review, and mark for qa/staging on any submission with some possibility to edit the severity/f**kupablity of the package so they can do the right decision.

But even tho I am now interested for sure to talk about it, lets wait till we

As I said: we have to talk to the OBS team about possible solutions for the "too much on the OBS" problem. Perhaps our assumption is wrong and we can do indeed what Josef suggests: create a staging project for every submission. Then the whole discussion on who reviews and decides what will be wasted time.

we can try at least. At least as long we do not publish it and our priority handling ensures that they do not block all other people from using OBS.

I think that only tricky part of my suggestion is COW, because now if I create new repo with copy of another one everything is rebuilded, but I want just copy what is there and modify only one package, that can possible trigger another build, but not whole 6k packages rebuild if not needed.

we have the trigger="localdep" setting. Means OBS will only rebuild packages which depend on that one. However, on the long run, this only tells if something compiles. It does not say if it works actually. So, with splitting it up, we do also need a way better automated QA since no one will be able to test all the staging projects anymore. -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hello, Am Donnerstag, 28. November 2013 schrieb Josef Reidinger:

I hope that in future we can more focus on automatic testing of results, then on manual review of source code changes. <corporate>Because results matter</corporate> :)

I have to disagree ;-) I'm fine with adding automatic testing, but the manual code review should stay in place. A good example would be code like test "$USER" == 'jreidinger' && rm -fr ~jreidinger If something like this is added, a human reviewer will easily notice it. However, the automatic test (which is running as a different user and therefore skips the rm command) will tell you: Test succeeded - everything is fine! Results matter, right? ;-) Another example where reading the code is the better choice are changes in rarely used code paths - it happened to me more than once that I found interesting[tm] (and buggy) code when reading the sources of various programs. Sometimes those code sections were "hidden" from most users, for example in an error handling section or in code that only 1% of the users use. It might be possible to write tests for the "only used by 1% of the users" code, but testing all error handling (and reproducing the errors to trigger this code path) is probably harder than proofreading the source code ;-) Regards, Christian Boltz -- I've already burnt my fingers with upstream patches ... now let's see whats happen and let's wait that my fingers will cool down. [Werner Fink in https://bugzilla.novell.com/show_bug.cgi?id=752422] -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 29.11.2013 10:24, Josef Reidinger wrote:

Tricky part is that for manual code review you need time and with increased number of packages it is not possible unless manpower is increased. I agree with you that code review is really important and for example in Yast we see really good impact on code quality, but experience is that for product quality automatic testing is more important.

We thought about that too and it's indeed a limiting factor, but IMO having reviews is so valuable that I wouldn't want to throw it away just because it's hard. Right now reviews have no priority at all, so the more packages we have the easier it is to get lost. And there is no support for sharing work in reviews the webui interface to do reviews is suboptimal too (it's dominated by the diff and the discussion is very much hidden). On the other hand we do have enough experienced packagers who can do reviews, but at the moment being a reviewer or not is black & white. So if you volunteer to review because you know how to package perl packages well, you end up being part of the same team as the one reviewing kernel submissions. So possibly we have do have the reviews done by more specific groups instead of one big group? That would leave us close to the Signed-Off tags kernel patches bear, but the OBS has no support for marking patches as reviewed by $Josef. Yet another idea how to improve things. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Le jeudi 28 novembre 2013, à 16:01 +0100, Tomáš Chvátal a écrit :

Dne Čt 28. listopadu 2013 14:32:54, Dominique Leuenberger a.k.a. Dimstar napsal(a):

...

(just to show, this is not black/white magic.. and describing this in a way that it's clear and not 'subject to the mood of Stephan' is difficult, but should be a goal to strive for).

Yes I wholeheartly agree that currently it is just a magic we do there.

For the record, I don't think we need to have a perfect solution at the very beginning. It's just a matter of having the review team learn by experience what might break and have that take the long way instead of the short way. Once we learn about the breakage patterns (and I'm pretty sure we know quite a few of them ;-)), and we use the proposal for them, we'll see less breakage and we can identify the next batch of breakage patterns. (+ it's always possible to revert in Factory if a breakage is found) Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 28 Nov 2013, Michal Hrusecky wrote:

Richard Biener - 15:45 28.11.13 wrote:

...

On Thu, 28 Nov 2013, Stephan Kulow wrote:

...

Hi

It's Thanksgiving today and we had a great harvest - "Bottle" rocks. But today is also a good time to plant new seeds for even better openSUSE releases. Factory has a healthy growth, but we have to make sure it's growing in the right direction and so the openSUSE team at SUSE had an on-and-off discussion basically since 12.3 on how to improve things.

But first let me give you some background, that you might not be aware of: 10.3 had 3334 packages, 11.1 3746, 3605 for 11.2, 3807 for 11.3, 4784 in 12.1, 5710 in 12.2, 6246 in 12.3, 6678 in 13.1, 6800 right now in Factory. If you need a picture, look at http://s.kulow.org/packages

Integrating these to make a good distribution is real work. And one of my favourite songs (in that context) goes:

No one said it would be easy But no one said it'd be this hard No one said it would be easy No one thought we'd come this far

In that song Sheryl Crow sings "It's just a question of eliminating obstacles", so what did we in the openSUSE Team do to help? We focused on getting a grip on testing by improving openqa (http://s.kulow.org/openqa-blog), but we soon found out that it was not good enough to test Factory ISOs. Factory is broken often enough not to produce ISOs at all, ISOs can't be installed and once these problems are sorted out, we found in openqa very basic things to be broken, but it was too late to protect factory users to run into them.

One thing I tried was to setup "rings" to help easing the very painful staging projects (with 6800 packages, every staging project as we use them is a monster). That experiment has shown rings to be worthy way to check, but they won't work as I thought with the OBS as it is. We need to think bigger. So we tried to come up with an idea on how to improve the factory development process that includes a more clever way to utilize staging projects and openQA.

As this development process is a bit hard to explain in email, Alberto and Ancor prepared an interactive diagram:

https://progress.opensuse.org/workflow/factory-proposal.html

We basically want to put the pressure on the submitting packager not the user. Using factory should be safe, for this we want to revive a thing that has been lost on the way: Bernhard's factory-tested project.

And we want to open another submission path: from staging projects. Consider a situation where recent updates to GNOME and automake 'clash', causing problems when they're installed together. Right now we throw both into Factory, breaking both for our Factory users until we solve the issues. Instead, we think working on these issues in a separate 'staging project' could be the solution. None of us knows how *exactly* it will look alike because we need to get a conversation with the OBS team going. But the basic idea is:

- GNOME:Factory stays the devel project of things that relate to GNOME updates - devel:tools stays the devel project of things that relate to automake updates - on automake updates, we open up a new devel project that stages GNOME packages for the new automake. In there, GNOME devs and automake experts work together to fix them and updates in there are submitted either back to GNOME:Factory and are integrated right into Factory. Or if that's not possible, the automake update is "grouped" with various GNOME package updates and these updates end in Factory together.

There are several problems with the current "everything through devel project" approach we need to solve. Our ideas are just ideas, but I had several discussions in various places and nobody offered a better idea. So we really would like to start with it and I would like to hear your concerns so they can be part of the final solution.

We have more ideas, but we can only achieve that if we get help, so let me finish with another favourite of mine

What would you think if I sang out of tune? Would you stand up and walk out on me? Lend me your ears and I'll sing you a song And I'll try not to sing out of key Oh, I get by with a little help from my friends

Greetings, Stephan

Back in the old times some people inside SUSE thought of following the Debian testing way. The road block then was the inability to track bugs filed against a package. To recap how Debian testing works - a package update gets pushed to "unstable", then, if no bugs against it appear for $X time, it gets automatically pushed to "testing" (if all dependencies it has at this point can be satisfied there - mind that Debian has a lot more versioned dependencies). We still don't have a "package" field in bugzilla, so copying this scheme 1:1 doesn't work.

But we don't have to re-invent the wheel, no?

The closest thing to Debian testing we have is openSUSE Tumbleweed though it's a very manual process there.

But isn't the proposal to have a Tumbleweed for Factory?

Don't really get the last question, but let me share my opinions. Tumbleweed is great, but it is a workaround for not stable enough Factory. If Factory would have been stable enough for everyday use (with only few minor hiccups), we wouldn't have needed Tumbleweed IMHO.

IMHO we would have "auto"-Factory which simply aggregates from all devel projects for example and "Factory" which is a Tumbleweed of that. But wait ... it already works that way. I don't think there will be a good solution without "splitting" Factory. What you then call Factory is up to you. With the "Tumbleweed for Factory" I was simply proposing that in additional to Factory as we have it today we have the "stable" Factory that is operated in "Tumbleweed" mode - pick up changes from Factory in a more controlled manner (aka when they work). Richard. -- Richard Biener SUSE / SUSE Labs SUSE LINUX Products GmbH - Nuernberg - AG Nuernberg - HRB 16746 GF: Jeff Hawn, Jennifer Guild, Felix Imend"orffer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Robert Schweikert - 7:22 1.12.13 wrote:

...

If Factory is stable enough for everyday use one could argue we should be able to release anytime. Thus the discussion about changing the release cycle would become moot.

Well, even if we stabilize Factory well enough for geek to use it everyday, it doesn't necessarily mean, that Joes grandma will be happy with it. There is this installer thing and first time setup. There is this where did my menu go, it was right here yesterday, my icon changed, everything is broken. And stable enough for everyday use for geeks might not be stable enough for grandma - aka this configuration changed, you need to edit this and that, we renamed your interfaces, software X no longer provides feature Y, ... -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Stephan Kulow schrieb:

On 28.11.2013 16:04, Robert Kaiser wrote:

...

Whenever discussions like this come up, I wonder if pieces from the Mozilla process that I'm so immersed in might apply.

I'm having a bit of a problem following your thoughts, do you have some URLs describing that process?

There's a lot of documentation about all kinds of process at Mozilla on developer.mozilla.org but I couldn't find a document that describes very well what I wanted to point to here. That's why I tried to describe things in my message here. Which parts are hard for you to understand from my explanation?

The merging of things is at the moment exactly our problem. A gcc update is not a patch you can take in and out, so you have to be *really* careful and try somewhere else. But from what I understand, our goal is not that far away.

I think that what would fit my description best would be to run compilation and tests for any dependencies of any package (or set of packages) that is to be pulled into Factory and only do make this "merge" effective when those succeed. I guess for many packages that would hopefully be a small set that needs to be run, on things like gcc it could be huge, of course. The question is then if the infrastructure is able to deal with that load - otherwise, I guess some compromise is needed. Robert Kaiser -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am Donnerstag, 28. November 2013, 16:18:57 schrieb Dominique Leuenberger a.k.a. Dimstar:

Quoting Adrian Schröter :

...

My hope is that we can developer some kind of objective QA measurements in future for this. Eg. X server is not only compiling, but even starting :) Or bluez update does not break the tools for one of the desktops like it is even the case for openSUSE:13.1 released version.

Please, stop using this example! This was CONSCIOUSLY decided, HERE on the Factory mailing lists.

Well, I do not want to spoil this discussion with that, but it comes to the point where we need to define our quality criterias before and follow them. And such criterias must be objective and should not be driven by current technical problems if we want to have the user acceptance. bye adrian PS: seriously, can you imagine that an Android ships breaking sound for the users? No, it would just not ship, because it does not meet the QA criteria. That is why I really recommend everbody of my friends to use android instead of classic GNU/Linux these days. And together with the new PA breakages I have to use two command line every few minutes, just because it looses connections while it worked before. Even worse, I do not care about to create bugreports anymore (well I did for bluez), because it is fighting against wind mills. But with a working QA system, I would actually provided test cases in such situations, because I would believe that my work is not void tomorrow. Yes, long way to go... -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thursday 28 November 2013 16:33:19 Dominique Leuenberger a.k.a. Dimstar wrote:

Quoting Adrian Schröter :

...

Am Donnerstag, 28. November 2013, 16:18:57 schrieb Dominique

Leuenberger a.k.a. Dimstar:

...

Quoting Adrian Schröter :

...

My hope is that we can developer some kind of objective QA measurements in future for this. Eg. X server is not only compiling, but even

starting :)

...

Or bluez update does not break the tools for one of the desktops like it is even the case for openSUSE:13.1 released version.

Please, stop using this example! This was CONSCIOUSLY decided, HERE on the Factory mailing lists.

Well, I do not want to spoil this discussion with that, but it comes to the point where we need to define our quality criterias before and follow them.

And such criterias must be objective and should not be driven by current technical problems if we want to have the user acceptance.

I didn't say it was necessary the *right* decision.. I only stated it was a conscious one. And as such defeated by all those automatics.

Yeap, and while it might have been painful, it is woefully off-topic here, so let's move on indeed ;-) These threads are already growing big and hairy as-is... /J

Dominique

Am 28.11.2013 19:32, schrieb Susanne Oberhauser-Hirschoff:

However how will this help dependent multi-target projects (like gnome, or kde or databases or d:lang:*) to likewise be stable at all^wmost times?

It looks to me like the flow that is proposed here continues to break projects that build for both factory and other released distributions.

Hi Susanne, Do you know how I solve a rubik's cube? Layer by layer! Not because it's faster, not because it's easier, but because I know how to solve it that way. Does that solution work for chess puzzles too? No, it doesn't. That's why I don't solve chess puzzles! Multi-target projects are the chess puzzles in above - in case that wasn't clear yet ;) Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Susanne Oberhauser-Hirschoff - 18:32 28.11.13 wrote:

...

There is one thing that I'm missing, but that probably would need other changes on obs than using it differently and adding submit request groups.

The build server is what it says: a *build* server.

We use it as an integration server, quite successfully, and it comes close, but it is not explicitely targeted at integration.

So it is missing a few features to better support integration:

* Tools like git support 'merge' tracking of changes in branches back to mainline and from progress in mainline back to the branch. This then also allows to bisect regressions to the integration issue.

Well, this would be feature request for OBS team. I think it was discussed in the past many times already - that if during initial obs design instead of reinventing the wheel git would be used to do version control system, things would be simpler now... But hard to revert that although some tried over the years...

* Integration means testing, and testing may be a gate/decision point whether further builds make sense at all (think rings). This tracking of test status is not in the tool. And tests should gate further work based on test status. And tests, automatic or manual, have a smart and a stupid order doing them.

So you would like to see better integration between openQA and OBS?

...

However how will this help dependent multi-target projects (like gnome, or kde or databases or d:lang:*) to likewise be stable at all^wmost times?

It looks to me like the flow that is proposed here continues to break projects that build for both factory and other released distributions.

Few comments: * It will not break stable versions * Changes done in devel project should be send to Factory anyway ** If changes are in Factory, base is not going to change to break everything without fix provided Sooo, the solution is simple, if you want your stuff not getting broken as nobody know what you have, submit your changes to Factory and don't keep them to yourself :-) Devel projects are packages on the way to Factory anyway ;-) -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Susanne Oberhauser-Hirschoff - 21:06 28.11.13 wrote:

Michal Hrusecky writes:

...

Susanne Oberhauser-Hirschoff - 18:32 28.11.13 wrote:

...

...

* Integration means testing, and testing may be a gate/decision point whether further builds make sense at all (think rings). This tracking of test status is not in the tool. And tests should gate further work based on test status. And tests, automatic or manual, have a smart and a stupid order doing them.

So you would like to see better integration between openQA and OBS?

openQA imnsho is just another flavour of build (as in rpm, kiwi, deb, openQA)

Not now, but probably could be integrated that way.

I'm talking about

1. support _manual_ steps in the flow: hardware tests in the community are manual, not automatic. bluez audio comes to mind.

Hmmm, looking at the chart, we really have pretty much automatic everywhere. Maybe we should weaken it so it wouldn't look like manual testing is not possible. But manual testing is slow and if we want to get packages through fast enough and with limited resources, everything should be as automatized as much as possible. And most of the manual testing happens once it gets at least to Factory integration part.

2. clearly distinguish test from build: a failing test may still mean an ok package, just only 'gold', not 'platinum'

Currently it is. It's independent service. That's also why there is a QA team in workflow (teams can overlap in real world). To interpret/fix results.

...

...

However how will this help dependent multi-target projects (like gnome, or kde or databases or d:lang:*) to likewise be stable at all^wmost times?

It looks to me like the flow that is proposed here continues to break projects that build for both factory and other released distributions.

Few comments:

* It will not break stable versions * Changes done in devel project should be send to Factory anyway ** If changes are in Factory, base is not going to change to break everything without fix provided

Sooo, the solution is simple, if you want your stuff not getting broken as nobody know what you have, submit your changes to Factory and don't keep them to yourself :-) Devel projects are packages on the way to Factory anyway ;-)

Like an upstream project at random times would push their stuff into my working branch and break it "because I'll merge it upstream later anyhow" ;-)

All changes to Factory would still have to go through devel project. So nobody can break your stuff. There might be updates of some (build) required packages, but in that case, it shouldn't brake your previous version and when is the better time to know about new incompatibility than when you are updating package already anyway. -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Nov 28, 2013 at 07:55:20PM +0100, Michal Hrusecky wrote:

Well, this would be feature request for OBS team. I think it was discussed in the past many times already - that if during initial obs design instead of reinventing the wheel git would be used to do version control system, things would be simpler now...

During initial obs design git did not exist. Both git and OBS were started in 2005. And we didn't reinvent the wheel but simply reused the old autobuild srcrep code. M. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

"Susanne Oberhauser-Hirschoff" writes:

Michael Schroeder 11/29/13 9:45 AM

...

the old autobuild srcrep code.

source repository that is. no insults here.

He didn't write srcrap :-) Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 28 Nov 2013 18:32:56 +0000 Susanne Oberhauser-Hirschoff wrote:

Stephan Kulow writes:

The build server is what it says: a *build* server.

We use it as an integration server, quite successfully, and it comes close, but it is not explicitely targeted at integration.

So it is missing a few features to better support integration:

* Tools like git support 'merge' tracking of changes in branches back to mainline and from progress in mainline back to the branch. This then also allows to bisect regressions to the integration issue.

+1, I really miss some features from git and easy merging is one of it.

* Integration means testing, and testing may be a gate/decision point whether further builds make sense at all (think rings). This tracking of test status is not in the tool. And tests should gate further work based on test status. And tests, automatic or manual, have a smart and a stupid order doing them.

What would be really nice here, is to have hooks in BS like github. If new pull request is created, then there is hook that can told it to CI like https://travis-ci.org/ or code quality metter like https://codeclimate.com/ which in response set status for such request so you immediatelly see if request passes tests or if quality of code goes up or down ( I think it will be really useful for e.g. rpmlint warnings, now I don't see if submit request increase or decrease number of warnings ). Now it is partially done in BS itself with their own check if rpm build. And if it is generic, then we can easily add different services in future like attach to project security scanner and check if in new version is new security warning. I prefer component system before monolitic application.

S.

Josef -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 29.11.2013 11:26, Adrian Schröter wrote:

...

...

* Integration means testing, and testing may be a gate/decision point whether further builds make sense at all (think rings). This tracking of test status is not in the tool. And tests should gate further work based on test status. And tests, automatic or manual, have a smart and a stupid order doing them.

What would be really nice here, is to have hooks in BS like github. If new pull request is created, then there is hook that can told it to CI like https://travis-ci.org/

http://openbuildservice.org/2013/11/22/Source-Update-Via_Token/

?

I knew you would jump on it but hooks for github are something else than hooks like github ;) Our solution for that are at the moment automatic reviewers that poll for new reviews to do and then queue something. So if you do a new SR factory-auto, legal-auto, repo-checker run on it and do their CI. We can extend that heavily - but nothing of that will be "Source update" ;) Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, 29 Nov 2013 11:49:24 +0100 Adrian Schröter wrote:

Am Freitag, 29. November 2013, 11:43:54 schrieb Stephan Kulow:

...

On 29.11.2013 11:26, Adrian Schröter wrote:

...

...

...

* Integration means testing, and testing may be a gate/decision point whether further builds make sense at all (think rings). This tracking of test status is not in the tool. And tests should gate further work based on test status. And tests, automatic or manual, have a smart and a stupid order doing them.

What would be really nice here, is to have hooks in BS like github. If new pull request is created, then there is hook that can told it to CI like https://travis-ci.org/

http://openbuildservice.org/2013/11/22/Source-Update-Via_Token/

?

I knew you would jump on it but hooks for github are something else than hooks like github ;)

When offering also this for reviews, these review hooks can run everywhere.

We can also discuss to run them within OBS, but I think other possibilities, like rpmlint or source services are the better approach then.

I mean it other way - submit requestion on BS -> call hoooks -> get back result so something like factory_auto and others. I don't know how easy or hard is to add such reviewers, if I want to use it outside of factory for my own project. From reviews at http://en.opensuse.org/openSUSE:Build_Service_Concept_Review it looks like you need to create fake user for it.

...

Our solution for that are at the moment automatic reviewers that poll for new reviews to do and then queue something. So if you do a new SR factory-auto, legal-auto, repo-checker run on it and do their CI. We can extend that heavily - but nothing of that will be "Source update" ;)

right, but the /trigger route could also be used to do other defined tasks. For example like changing the review state. That was what I proposed here.

So if I get right how reviews work, then if we have e.g. security scanner, then if new thing appear he must open review for someone to manual check new detected issue, right? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am Freitag, 29. November 2013, 12:34:59 schrieb Josef Reidinger:

On Fri, 29 Nov 2013 11:49:24 +0100 Adrian Schröter wrote:

...

Am Freitag, 29. November 2013, 11:43:54 schrieb Stephan Kulow:

...

On 29.11.2013 11:26, Adrian Schröter wrote:

...

...

...

* Integration means testing, and testing may be a gate/decision point whether further builds make sense at all (think rings). This tracking of test status is not in the tool. And tests should gate further work based on test status. And tests, automatic or manual, have a smart and a stupid order doing them.

What would be really nice here, is to have hooks in BS like github. If new pull request is created, then there is hook that can told it to CI like https://travis-ci.org/

http://openbuildservice.org/2013/11/22/Source-Update-Via_Token/

?

I knew you would jump on it but hooks for github are something else than hooks like github ;)

When offering also this for reviews, these review hooks can run everywhere.

We can also discuss to run them within OBS, but I think other possibilities, like rpmlint or source services are the better approach then.

I mean it other way - submit requestion on BS -> call hoooks -> get back result so something like factory_auto and others. I don't know how easy or hard is to add such reviewers, if I want to use it outside of factory for my own project.

You can define default reviewer in any project or package. Means, a new request for this project/package gets this reviewer by default. When your external

From reviews at http://en.opensuse.org/openSUSE:Build_Service_Concept_Review it looks like you need to create fake user for it.

or a group. this group would define which users can act on behalf of the group. We could also add the token/trigger system there.

...

...

Our solution for that are at the moment automatic reviewers that poll for new reviews to do and then queue something. So if you do a new SR factory-auto, legal-auto, repo-checker run on it and do their CI. We can extend that heavily - but nothing of that will be "Source update" ;)

right, but the /trigger route could also be used to do other defined tasks. For example like changing the review state. That was what I proposed here.

So if I get right how reviews work, then if we have e.g. security scanner, then if new thing appear he must open review for someone to manual check new detected issue, right?

That is one way to do it. -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Sun, 1 Dec 2013, Christian Boltz wrote:

Hello,

Am Sonntag, 1. Dezember 2013 schrieb Robert Schweikert:

...

1.) Putting pressure on the submitter

...

Putting pressure on the submitter is a good concept to avoid "dump and run scenarios", i.e. put you code in and everyone has to fix the fall out. However, stated as such the submitter can very easily feel overwhelmed and left alone, thus the submission may never take place. What I think we need is a process/environment that holds the submitter sufficiently responsible to avoid "dump and run" while at the same time providing enough support such that the submitter does not feel left alone and overwhelmed.

While "the submitter fixes everything he breaks" would be ideal, I'd define the goal as:

The submitter has to coordinate fixing everything he breaks.

This can mean: - the submitter fixes it - the submitter works with the maintainers of the broken packages to get them fixed - the submitter asks for help on the factory ML

We should also have a rule saying (assuming it didn't happen in a staging project):

If the fixes don't happen in a reasonable time, revert the commit that caused the breakage.

I know "reasonable time" is vague, but we'll probably need it that way because it depends on the number of broken packages, time until the release etc.

...

In a staging model I have no idea how to get there.

That's easy - the package can move from staging to factory after the fallout is fixed ;-)

...

2.) The staging approach

...

staging trees upon staging trees. But this only solves the problem superficially as the target tree will move ahead and thus the staging tree by definition is always out of date. Unless the target tree is frozen until a particular staging tree is merged.

The staging tree should link all packages from factory (except the changed package), so it can't be outdated. It just needs to be rebuild.

We'll see if we have enough build power ;-) (For speedup, copying the factory binaries to the staging project and only rebuilding the updated package and its dependencies might save some time.)

This seems to concentrate fully on build-time fallout. While that's certainly part of the quality of Factory a failed build cannot break my running system because I cannot install a failed-to-build package. Yes, with enough build power we can re-build the world for every tiny change. But what does it really mean if the tiny change causes something to no longer build? It means that our dependencies are too weak (foo requires bar-devel instead of bar-devel = 10.2) or incomplete? Or that the now failing packages are simply broken? That said, how does ensuring everything builds enhance the user experience when you have Factory installed? With the Debian 'testing' distribution approach you scale QA by making people using 'unstable' (aka Factory) do testing and file bugs which blocks packages from migrating from 'unstable' to 'testing' unless they are fixed. So to throw in another name (than the appearantly misleading Tumbleweed), 'testing' is a rolling release for 'unstable'. Do we want a rolling-released-Factory? Richard. -- Richard Biener SUSE / SUSE Labs SUSE LINUX Products GmbH - Nuernberg - AG Nuernberg - HRB 16746 GF: Jeff Hawn, Jennifer Guild, Felix Imend"orffer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Mon, 2013-12-02 at 11:50 +0100, Stephan Kulow wrote:

Our testing are devel projects. I bet some use devel:gcc to test gcc and report bugs before you send it to Factory. But not enough.

That is exactly how I prefer to operate when testing/maintaining GNOME Using GNOME:Factory (the gnome devel repo) with Factory gives me a way of testing all of the 'incoming' GNOME changes. If something in our GNOME devel repo fouls up, I'm a simple 'zypper removerepo' and 'zypper dup' away from getting my machine back to 'usable' so I can get working on figuring out what broke in GNOME The only reason I don't use this approach 24/7, 365 days a year is that with our current approach, Factory isn't 'stable' enough. Or to put it another way, there's about as much chance (if not more) every time I 'zypper up' my machine that recent updates to Factory are going to break my system, as the recent updates to our GNOME devel project.. I think the suggestions so far take us huge strides towards making Factory usable in exactly the kind of way *I* need it.. which is why I'm so excited by them. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Mon, 2013-12-02 at 11:53 +0000, Dominique Leuenberger a.k.a. Dimstar wrote:

Quoting Richard Brown :

...

On Mon, 2013-12-02 at 11:50 +0100, Stephan Kulow wrote:

...

Our testing are devel projects. I bet some use devel:gcc to test gcc and report bugs before you send it to Factory. But not enough.

That is exactly how I prefer to operate when testing/maintaining GNOME

Using GNOME:Factory (the gnome devel repo) with Factory gives me a way of testing all of the 'incoming' GNOME changes.

If something in our GNOME devel repo fouls up, I'm a simple 'zypper removerepo' and 'zypper dup' away from getting my machine back to 'usable' so I can get working on figuring out what broke in GNOME

Just some food for thought here: * if the goal is an always 'stable' Factory, I'm not sure at what time we are now best to submit GNOME:Factory to openSUSE:Factory.

There are two conflicting approaches: * Release Often / Release early (approach used so far) * Release stable (new ?)

Stable indicates no GNOME 3.11.x, as those are 'expected' to have bugs and introduce changes mid-way. So in order to satisfy the '2nd' entry, I'd argue that no submits should be done until at least RC1 of GNOME (in this case). This is different to what was done so far, where GNOME Team made sure it 'works', but was a bit more 'lose' on accepting to forward with minor issues here and there (knowing that Factory was dev / integration project and some minor instabilities were acceptable).

Dominique

A good point.. for the purposes of "New Factory" I would suggest we define stable/acceptable in Factory as either A) 'this has been tested (ideally by both human and automated means) and is confirmed by those tests to be functioning' or B) 'this has been tested (ideally by both human and automated means), confirmed to be working, and is felt to be 'stable' by the maintainers of that project' I personally prefer B) as it encourages maintainers to make a judgement call based on their knowledge of the upstream project/packages, like the one you describe. I can also think of other recent cases (ATI drivers) where one of our maintainers has made a call that an upstream 'beta' package was good enough for our users. With either A or B, we'd still be able to strive for "Release Often / Release early" while also making sure we only "Release working" -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Quoting Robert Kaiser :

Richard Brown schrieb:

...

A) 'this has been tested (ideally by both human and automated means) and is confirmed by those tests to be functioning'

or

B) 'this has been tested (ideally by both human and automated means), confirmed to be working, and is felt to be 'stable' by the maintainers of that project'

I disagree on B because Factory ought to be the development distro, not a "fully stable" distro. There can be bugs, that's OK, as long as it's in a state where it's well enough usable for day-to-day operation and any significant bugs are worked on and fixed fast.

So, I'd replace "stable" with "well-usable in daily operation" in your definition of B and then I feel it matches what Factory should be. What I would add in though is that is should not apply to "this" in terms of the particular package only, but "the result of this change", which should include what it does to all dependencies that are in Factory (in most cases, this will not make a huge difference, but in cases like gcc it can be huge).

Valid definition... and something that can be agreed. But it will still mean, I will not submit GNOME 3.11.2 to Factory; there are surely bugs (that's why upstream does not release it stable branch) and I don't know how fast any of the given bugs can be fixed... and I'm not willing to chase down 250 git commits until the next snapshot release comes (probably in 4 - 6 weeks). So, as it stands, I will keep on waiting for 3.12.0 to become available, at which point I probably can't submit it due to none of the integration work being done / prepared (the upgrade will be too large for all other packages to absorb it... and the GNOME Team will not have time to run after everything, as we'd want 3.12.1 in there soon too). so far, Factory was a 'integration' project, with the aim to be usable.. new it shall be a usable project, with the aim to integrate new stuff. A small, but subtle difference. And the ultimate target to have it stabilized was the release. Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 12/02/2013 09:15 AM, Stephan Kulow wrote:

On 02.12.2013 15:08, Dominique Leuenberger a.k.a. Dimstar wrote:

...

Valid definition... and something that can be agreed.

But it will still mean, I will not submit GNOME 3.11.2 to Factory; there are surely bugs (that's why upstream does not release it stable branch) and I don't know how fast any of the given bugs can be fixed... and I'm not willing to chase down 250 git commits until the next snapshot release comes (probably in 4 - 6 weeks).

So, as it stands, I will keep on waiting for 3.12.0 to become available, at which point I probably can't submit it due to none of the integration work being done / prepared (the upgrade will be too large for all other packages to absorb it... and the GNOME Team will not have time to run after everything, as we'd want 3.12.1 in there soon too).

so far, Factory was a 'integration' project, with the aim to be usable.. new it shall be a usable project, with the aim to integrate new stuff. A small, but subtle difference. And the ultimate target to have it stabilized was the release.

Yeah, there are several ideas floating around to get you back that always-broken project.

I don't think people are looking for "always broken", but more for "mostly functioning a good chunk of the time". The problem will be, if we have such a project we will need to find (a) new shepherd(s) for that project. Now every one depends mostly on coolo and that and will no longer be able to do in the future. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 12/02/2013 09:08 AM, Dominique Leuenberger a.k.a. Dimstar wrote:

Quoting Robert Kaiser :

...

Richard Brown schrieb:

...

A) 'this has been tested (ideally by both human and automated means) and is confirmed by those tests to be functioning'

or

B) 'this has been tested (ideally by both human and automated means), confirmed to be working, and is felt to be 'stable' by the maintainers of that project'

I disagree on B because Factory ought to be the development distro, not a "fully stable" distro. There can be bugs, that's OK, as long as it's in a state where it's well enough usable for day-to-day operation and any significant bugs are worked on and fixed fast.

So, I'd replace "stable" with "well-usable in daily operation" in your definition of B and then I feel it matches what Factory should be. What I would add in though is that is should not apply to "this" in terms of the particular package only, but "the result of this change", which should include what it does to all dependencies that are in Factory (in most cases, this will not make a huge difference, but in cases like gcc it can be huge).

Valid definition... and something that can be agreed.

But it will still mean, I will not submit GNOME 3.11.2 to Factory; there are surely bugs (that's why upstream does not release it stable branch) and I don't know how fast any of the given bugs can be fixed... and I'm not willing to chase down 250 git commits until the next snapshot release comes (probably in 4 - 6 weeks).

So, as it stands, I will keep on waiting for 3.12.0 to become available, at which point I probably can't submit it due to none of the integration work being done / prepared (the upgrade will be too large for all other packages to absorb it... and the GNOME Team will not have time to run after everything, as we'd want 3.12.1 in there soon too).

so far, Factory was a 'integration' project, with the aim to be usable.. new it shall be a usable project, with the aim to integrate new stuff. A small, but subtle difference. And the ultimate target to have it stabilized was the release.

I think GNOME is a good example but the same problem applies to many other devel projects. I also think that neither A or B as proposed by Richard are solutions to the problem described by Dominique. The changes in wording from "stable" to "usable" would definitely be necessary, but this is also a very slippery slope. A bug that breaks functionality in a given way may not effect one person and thus the system is "usable" while it will leave another person completely stuck, thus the system is "not usable". Therefore, even the "usable" definition appears to open up a can of worms. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 12/02/2013 06:25 AM, Richard Brown wrote:

On Mon, 2013-12-02 at 11:50 +0100, Stephan Kulow wrote:

...

Our testing are devel projects. I bet some use devel:gcc to test gcc and report bugs before you send it to Factory. But not enough.

That is exactly how I prefer to operate when testing/maintaining GNOME

Using GNOME:Factory (the gnome devel repo) with Factory gives me a way of testing all of the 'incoming' GNOME changes.

If something in our GNOME devel repo fouls up, I'm a simple 'zypper removerepo' and 'zypper dup' away from getting my machine back to 'usable' so I can get working on figuring out what broke in GNOME

The only reason I don't use this approach 24/7, 365 days a year is that with our current approach, Factory isn't 'stable' enough. Or to put it another way, there's about as much chance (if not more) every time I 'zypper up' my machine that recent updates to Factory are going to break my system, as the recent updates to our GNOME devel project..

I think the suggestions so far take us huge strides towards making Factory usable in exactly the kind of way *I* need it.. which is why I'm so excited by them.

Well I HATE rebooting my machine, sitting on top of Factory would imply kernel updates on a more regular basis and that doesn't work for me. Rebooting 2 or 3 times during the 8 month release cycle is more than enough for me. If we do not chase the kernel in the new Factory model, but only update the Factory kernel every 3 months or so when there is a new upstream release then I am OK with it, but if we have RC releases in Factory and I would end up having to reboot every two weeks as the kernel release cycle is approaching a new release that is just not going to work for me. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 02.12.2013 15:41, Robert Schweikert wrote:

Well I HATE rebooting my machine, sitting on top of Factory would imply kernel updates on a more regular basis and that doesn't work for me. Rebooting 2 or 3 times during the 8 month release cycle is more than enough for me.

If we do not chase the kernel in the new Factory model, but only update the Factory kernel every 3 months or so when there is a new upstream release then I am OK with it, but if we have RC releases in Factory and I would end up having to reboot every two weeks as the kernel release cycle is approaching a new release that is just not going to work for me.

The fun part about rolling release you can decide yourself when to jump on it. There is no reason to update daily. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Monday 02 December 2013 16:17:26 Stephan Kulow wrote:

On 02.12.2013 15:41, Robert Schweikert wrote:

...

Well I HATE rebooting my machine, sitting on top of Factory would imply kernel updates on a more regular basis and that doesn't work for me. Rebooting 2 or 3 times during the 8 month release cycle is more than enough for me.

If we do not chase the kernel in the new Factory model, but only update the Factory kernel every 3 months or so when there is a new upstream release then I am OK with it, but if we have RC releases in Factory and I would end up having to reboot every two weeks as the kernel release cycle is approaching a new release that is just not going to work for me.

The fun part about rolling release you can decide yourself when to jump on it. There is no reason to update daily.

Quite frankly, I consider this still the best option to have a stable Factory. It's brain-dead simple and may just require a little social interaction: 0) If you are not subscribed to opensuse-factory@ (and probably packaging / buildservice), you should not run Factory. 1) So check opensuse-factory@ for issues regularly. If there was a mail two weeks ago about udev (random choice of mine) being heavily broken and no fix was pushed meanwhile, it may be a good hint to not update today :-) 2) Asking your colleagues or community friends for their recent experience often helps to discover such things. When I ask people, I either get "bah, X is broken again" or "nope, things have been running smooth lately". Of course there's a certain chance that things will go unnoticed, but the ugly stuff will. 3) As coolo said, there's no need to update every morning, once every two weeks is more than enough. 4) Lastly, if zypper tells you some dependencies can't be satisfied, think twice if you want to update. Of course that depends on what is broken and how important it is to you. What we can do though is to make this kind of information more accessible. If dunno if it the amount of bugs filed against Factory per week would be a good indicator, but it could be tried. Add the count of failing packages and some more and maybe we can come up with some kind of "Factory Health Report". This could be published somewhere, maybe even sent to the factory ML every Monday. Just an idea... ----- So I am running Factory as of yesterday. I don't experience any issues besides plasma-networkmanager being unable to load. NetworkManager works, so all green. -- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)

On Tuesday 03 December 2013 10:20:13 Michal Hrusecky wrote:

Sascha Peilicke - 10:02 3.12.13 wrote:

...

...

Quite frankly, I consider this still the best option to have a stable Factory. It's brain-dead simple and may just require a little social interaction:

0) If you are not subscribed to opensuse-factory@ (and probably packaging / buildservice), you should not run Factory.

1) So check opensuse-factory@ for issues regularly. If there was a mail two weeks ago about udev (random choice of mine) being heavily broken and no fix was pushed meanwhile, it may be a good hint to not update today :-)

2) Asking your colleagues or community friends for their recent experience often helps to discover such things. When I ask people, I either get "bah, X is broken again" or "nope, things have been running smooth lately". Of course there's a certain chance that things will go unnoticed, but the ugly stuff will.

Sounds like a lot of work just to make sure that it is safe to update,

That's where the Factory Health Report could help.

which overall means that Factory is not stable enough to be trusted, which I think should be changed (not in the perception part but in reality)

It depends on the definition of "stable enough" . But I assume we'll have a shared one once we draw conclusions out of this big thread.

...

3) As coolo said, there's no need to update every morning, once every two weeks is more than enough.

When I used to run Gentoo I used to update every other day, but sometimes I skipped longer periods of times because of vacation or something. Both should be possible.

...

4) Lastly, if zypper tells you some dependencies can't be satisfied, think twice if you want to update. Of course that depends on what is broken and how important it is to you.

This should be something that we can fix automatically.

...

What we can do though is to make this kind of information more accessible. If dunno if it the amount of bugs filed against Factory per week would be a good indicator, but it could be tried. Add the count of failing packages and some more and maybe we can come up with some kind of "Factory Health Report". This could be published somewhere, maybe even sent to the factory ML every Monday. Just an idea...

Well, we should strive for no failing packages :-) But having some automatic metrics on how calm factory is right know sound like a good idea :-)

-- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)

On Thu, Dec 05, 2013 at 03:03:04PM +0100, Ludwig Nussel wrote:

Coming back from vacation and sick leave I just experience again how much it sucks to find relevant information in old email threads :-) Maybe package meta data could be made to carry some kind of good/bad score. If a certain build of a package is rated down by people, zypper could display a warning. Maybe the solver could even try to come up with a solution that does not upgrade to the faulty build.

Sorry, I'm very much against the idea of the solver choosing packages by some number the user does not see. In other words: I can implement that, but then I'll assign all bug reports about strange solver results to you. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 5 Dec 2013, Richard Biener wrote:

On Thu, 5 Dec 2013, Ludwig Nussel wrote:

...

Sascha Peilicke wrote:

...

On Monday 02 December 2013 16:17:26 Stephan Kulow wrote:

...

The fun part about rolling release you can decide yourself when to jump on it. There is no reason to update daily.

Quite frankly, I consider this still the best option to have a stable Factory. It's brain-dead simple and may just require a little social interaction:

0) If you are not subscribed to opensuse-factory@ (and probably packaging / buildservice), you should not run Factory.

1) So check opensuse-factory@ for issues regularly. If there was a mail two weeks ago about udev (random choice of mine) being heavily broken and no fix was pushed meanwhile, it may be a good hint to not update today :-)

Coming back from vacation and sick leave I just experience again how much it sucks to find relevant information in old email threads :-) Maybe package meta data could be made to carry some kind of good/bad score. If a certain build of a package is rated down by people, zypper could display a warning. Maybe the solver could even try to come up with a solution that does not upgrade to the faulty build.

If we have package metadata amendable by users can we at least record (open) bugreports there? OTOH bugreports should be eventually visible to the maintainer so he can do the association as well.

Then you could implement

...

zypper show-bugs-I-get-when-dup

New bugs, of course. And I'd rather restrict metadata to contain (important) regressions. You can then pin the packages you don't want to regress and the solver should be able to update the rest as far as possible. Richard. -- Richard Biener SUSE / SUSE Labs SUSE LINUX Products GmbH - Nuernberg - AG Nuernberg - HRB 16746 GF: Jeff Hawn, Jennifer Guild, Felix Imend"orffer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 5 Dec 2013, Adrian Schr?ter wrote:

Am Donnerstag, 5. Dezember 2013, 16:01:23 schrieb Richard Biener:

...

On Thu, 5 Dec 2013, Ludwig Nussel wrote:

...

Sascha Peilicke wrote:

...

On Monday 02 December 2013 16:17:26 Stephan Kulow wrote:

...

The fun part about rolling release you can decide yourself when to jump on it. There is no reason to update daily.

Quite frankly, I consider this still the best option to have a stable Factory. It's brain-dead simple and may just require a little social interaction:

0) If you are not subscribed to opensuse-factory@ (and probably packaging / buildservice), you should not run Factory.

1) So check opensuse-factory@ for issues regularly. If there was a mail two weeks ago about udev (random choice of mine) being heavily broken and no fix was pushed meanwhile, it may be a good hint to not update today :-)

Coming back from vacation and sick leave I just experience again how much it sucks to find relevant information in old email threads :-) Maybe package meta data could be made to carry some kind of good/bad score. If a certain build of a package is rated down by people, zypper could display a warning. Maybe the solver could even try to come up with a solution that does not upgrade to the faulty build.

If we have package metadata amendable by users can we at least record (open) bugreports there? OTOH bugreports should be eventually visible to the maintainer so he can do the association as well.

Then you could implement

...

zypper show-bugs-I-get-when-dup

Richard.

Argh, that means we need to re-generate the meta data on each change in bugzilla?

You infrastructure murder!

Well, as it's probably not possible to automate this via bugzilla I'd rather add a osc addbug prj pkg but yes, you probably need to re-generate the meta data. Richard. -- Richard Biener SUSE / SUSE Labs SUSE LINUX Products GmbH - Nuernberg - AG Nuernberg - HRB 16746 GF: Jeff Hawn, Jennifer Guild, Felix Imend"orffer -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am Donnerstag, 5. Dezember 2013, 15:20:02 schrieb Dominique Leuenberger a.k.a. Dimstar:

Quoting Richard Biener :

...

Well, as it's probably not possible to automate this via bugzilla I'd rather add a

osc addbug prj pkg

There must of course be a way to mark them solved..

no, we do track already bugs in package sources. We do parse the .changes files, eg check osc api /source/openSUSE:Tools/build?view=issues These are of course only bugs which are already handled to some degree, because they are mentioned in the sources.

In fact we'd want it removed on checkin.. so osc rq accept should have the logic that bugs, that are mentioned in the .changes, should be removed again...

We do track bugs, which we want to solve in _patchinfo files. They exist usually in maintenance projects to define what needs to be fixed. But it does not depend on concrete packages, because it is often enough not clear in which package the fix must be applied. You know that usually only after you have fixed it :) -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 12/02/2013 08:32 AM, Richard Biener wrote:

On Mon, 2 Dec 2013, Stephan Kulow wrote:

...

On 02.12.2013 11:39, Richard Biener wrote:

...

With the Debian 'testing' distribution approach you scale QA by making people using 'unstable' (aka Factory) do testing and file bugs which blocks packages from migrating from 'unstable' to 'testing' unless they are fixed. So to throw in another name (than the appearantly misleading Tumbleweed), 'testing' is a rolling release for 'unstable'. Do we want a rolling-released-Factory?

Our testing are devel projects. I bet some use devel:gcc to test gcc and report bugs before you send it to Factory. But not enough.

Probably. I thought of devel projects as 'experimental', because the devel projects are where development happens, not where candidates for Factory reside, no?

Well, yes an no. As Michal, I think, pointed out yesterday and I am paraphrasing "devel projects are for packages on their way to factory." But, while this may have been the original intend, in practice devel projects are being used for packages on their way to factory as well as packages people would like to have visible in a devel project but do not want to maintain in the "distribution proper." THus devel projects today serve at least 2 purposes.

Which immediately makes me think that we can automate this by immediately accepting submitrequests to Factory into a single Factory:staging project (without "review").

Every problem in computer science can be solved by a level of indirection ;) Thus factory:staging turns into what factory is today. Or one can go the other way as has also been discussed and say factory stays as it is and we use factory:tested as the "rolling factory release". The latter may be more amenable to the picture people already have in there heads. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 02.12.2013 13:14, Adrian Schröter wrote:

I do not see the need for this actually. It is perfectly fine to run the staging approach, but replace more then one package in it.

We may should have a staging project for collecting such small changes. But we should release them altogether to Factory after validating it with one rebuild (plus possible QA checks if we have some in future).

That sounds like a very sensible thing to do. I wonder how to manage it though. Any ideas? Let's assume we have a limited set of staging projects, we'd need to put a change into one of them "where it fits best". Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am Montag, 2. Dezember 2013, 13:38:52 schrieb Stephan Kulow:

On 02.12.2013 13:14, Adrian Schröter wrote:

...

I do not see the need for this actually. It is perfectly fine to run the staging approach, but replace more then one package in it.

We may should have a staging project for collecting such small changes. But we should release them altogether to Factory after validating it with one rebuild (plus possible QA checks if we have some in future).

That sounds like a very sensible thing to do. I wonder how to manage it though. Any ideas?

we could start with creating a staging project for large submissions like entire KDE update. And then have some collection projects, where we collect everything, which did not fit somewhere else for, let's say 2 weeks. After 2 weeks we do open the next staging project and continue to fix the former one until it settles. What do you think? -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Dne St 4. prosince 2013 19:58:00, Stephan Kulow napsal(a):

Am 04.12.2013 18:34, schrieb Adrian Schröter:

So as me and Michal played with this for a week or so before we got another crazy stuff here is what we prepped.

...

we could start with creating a staging project for large submissions like entire KDE update.

That actually is not needed as kde submission is quite self-contained. But on the other hand the Gnome stuff often need multiple passes in repo-checker to get in so it might not hurt to start there. :)

...

And then have some collection projects, where we collect everything, which did not fit somewhere else for, let's say 2 weeks. After 2 weeks we do open the next staging project and continue to fix the former one until it settles.

What do you think?

I think that there is a lot to fix and enhance before we can handle it and discuss. E.g. I have no way to see if openSUSE:Factory:Staging:NoFam already contains changes or if its pure

We worked on this in the factory-auto repo [1]. The verification is actually half done, we wrote staging plugin for osc and this is mostly the only part that is done and working - checking whether staging project contains only links without any local modifications. What needs to be added is checking if the changes are all part of the GR# or actually already merged in factory if it is the fact. As now the checker just validates that all the changes are back in devel projects. The script now can create staging (might need adjustment to be the no-publish repo and maybe other tweaks to reduce the load) and check if it is clean (no changes directly in there) and of course delete the staging, but without verification for now so it can in fact wipe almost anything even if you don't want it to. There are also completely empty functions to push to factory, eg mark the grouprequest as passed over staging if it is the fact. And function to submit the stuff back to devel projects from the changes within the staging (that one is easy to do but we didn't work there yet as it was lower on the testing list).

IMO we basically need to find out what problems are common to all models and start hacking. Once we're there, we can test different scenario. The models will only be as good as the tools to support them.

The problems we faced there were more on the tooling side which we can overcome by writting the api where needed (which I personaly can't do even if I try best, I wrote like 20 lines top in Ruby). So the issues we faced: If you create loads of stagings the wait-on-result was quite longish TM. MIght be solved by not publishing it or creating them as "frozen" set that can be refreshed when needed. No way how to update the staging for now. Our plan was to use the groups in a way where you could call update, and it would check the grouping request that is in the staging and refresh it based on new submissions in there if needed (could be also hook on group request that each time the GR is updated the staging is refreshed, would be probably much better than command on the client side). Addition is not written, you have to do the linkpac yourself in the repo now. Might be better to allow it in two ways, adding directly to group request (with osc group add) or via staging where you can do (osc staging add SR#) which would do the same; add SR# to the staging groupID and update the project. If you look on the repo there is .dot file which has "workflow" as thought us and it is heavily dependant on the grouping requests at least how we pictured it. So we need to adjust the groupings to be able to accept them, get properly their state and status of all subrequests within the group. Cheers Tom + Michal [1] https://github.com/coolo/factory-auto/blob/master/osc-staging.py

Robert Schweikert - 8:32 1.12.13 wrote:

...

What do I have to offer other than concerns?

There is the component model that I had proposed a while back. The component model may merge well with the idea of rings, that's something that could be explored.

I agree with Christian that component model will just mean more mess than it will do good if we really would split stuff. See how messy devel projects are. -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 12/02/2013 07:16 AM, Adrian Schröter wrote:

Am Sonntag, 1. Dezember 2013, 08:32:47 schrieb Robert Schweikert:

...

On 11/28/2013 08:49 AM, Stephan Kulow wrote: ... The staging model will not catch adverse interactions reliably. The reason is that by definition the staging tree is always out of date, unless the target tree is frozen and after one staging tree is accepted all other staging trees get rebuilt. This is not conducive to parallel development.

It depends how you run it. If you have large enough Staging projects, I think we can build them entirely and merge in factory. Afterwards we to wait for the other staging projects that right.

But it is a question how many and therefore how large staging projects we have.

Yes, but larger staging projects imply unrelated things having to wait for each other. Lets use a simple example. Lets say we have a staging tree that has Perl and Python in it. Developer A works on Perl stuff and developer B works on Python stuff. Developer A upgrades Perl to a new version and things break, developer B upgrades Python and everything works smoothly. Or the other way around, I am not being biased. Having both of these in the same staging tree implies that developer B cannot get the stuff into factory because something unrelated is busted in the staging tree. Thus, developer B will be annoyed. Yes ideally everything would work before pushing things into the staging tree, but lets just assume we do not live in an ideal world. The solution to this problem is to split the staging tree, thus we end up with 2, one for Perl changes and one for Python changes. In this case developer B does no longer care if stuff in Perl is broken because the Python staging tree moves forward and gets merged into factory as expected. This is the root cause for the "ever growing" staging tree problem. Even if we set out with lets say one staging tree per devel project one will have the A has to wait on B problem. Look at devel:filesystems. A good chunk of the stuff there is somewhat unrelated and if you clump it all into one staging project you end up with the problem outlined in the "Perl-Python" example above. In general, this is not a technical problem. With enough build power one can theoretically have a staging project for everything and push build times toward an acceptable small time limit. The problem we face is a people problem. Number of people available to chaperone the staging projects, number of people continuing to remain interested when their stuff gets delayed because of some unrelated problem on a relative frequent basis. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 12/02/2013 02:19 PM, Michal Hrusecky wrote:

Robert Schweikert - 11:42 2.12.13 wrote:

...

On 12/02/2013 07:16 AM, Adrian Schröter wrote:

...

Am Sonntag, 1. Dezember 2013, 08:32:47 schrieb Robert Schweikert:

...

On 11/28/2013 08:49 AM, Stephan Kulow wrote: ... The staging model will not catch adverse interactions reliably. The reason is that by definition the staging tree is always out of date, unless the target tree is frozen and after one staging tree is accepted all other staging trees get rebuilt. This is not conducive to parallel development.

It depends how you run it. If you have large enough Staging projects, I think we can build them entirely and merge in factory. Afterwards we to wait for the other staging projects that right.

But it is a question how many and therefore how large staging projects we have.

Yes, but larger staging projects imply unrelated things having to wait for each other.

Well, unrelated things shouldn't go into same staging project...

...

Lets use a simple example. Lets say we have a staging tree that has Perl and Python in it. Developer A works on Perl stuff and developer B works on Python stuff.

Shouldn't happen unless Python for some strange reason depend on perl and vice versa :-D

So you are saying Perl and Python would not be in the same staging project, fair enough. Use a different example and think about the basic problem I was trying to illustrate. As soon as more than one package is in a staging project you have the problem I was trying to describe.

...

...

This is the root cause for the "ever growing" staging tree problem. Even if we set out with lets say one staging tree per devel project one will have the A has to wait on B problem. Look at devel:filesystems. A good chunk of the stuff there is somewhat unrelated and if you clump it all into one staging project you end up with the problem outlined in the "Perl-Python" example above.

We should not set up Staging project per devel project (devel project is kind of limited staging project already) but we should setup staging project for every problematic package.

Every package is potentially problematic to some extend. If we rule out that we are going to have 6k+ staging projects you end up with the basic problem outlined in the example: Developer A ends up in a staging project with developer B and developer A's changes cannot progress because developer B's stuff breaks something. There is no way around this unless every package/developer gets their own staging project. That is not realistic and creates a completely different nightmare of coordination. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am 02.12.2013 22:44, schrieb Robert Schweikert:

So you are saying Perl and Python would not be in the same staging project, fair enough. Use a different example and think about the basic problem I was trying to illustrate. As soon as more than one package is in a staging project you have the problem I was trying to describe.

You're creating the same problem we're having in factory for a very long time. It's not a nice problem, but surely not a complete no-go situation. We're quite experienced with it ;) Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 03.12.2013 16:33, Robert Schweikert wrote:

On 12/03/2013 01:32 AM, Stephan Kulow wrote:

...

Am 02.12.2013 22:44, schrieb Robert Schweikert:

...

So you are saying Perl and Python would not be in the same staging project, fair enough. Use a different example and think about the basic problem I was trying to illustrate. As soon as more than one package is in a staging project you have the problem I was trying to describe.

You're creating the same problem we're having in factory for a very long time. It's not a nice problem, but surely not a complete no-go situation. We're quite experienced with it ;)

I am not creating a problem. I am trying to understand how the proposed I didn't mean "Robert" when I said "you" - just as you didn't mean "Stephan" when you said "you have the problem", did you? :)

staging model resolves the problem. Apparently by pointing out problems I have experienced with the staging model in the past it leads us back to where we are.

Yes, to some degree we move the "current factory" problem into staging projects. Factory is no longer the integration project, the staging projects will be. It's still an open question how many of those we can bear, but what we learned: one integration project is too little. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Robert Schweikert writes:

2.) The staging approach I can only speak from experience and thus this might sound a little lame, sorry. I have seen two implementations of the staging model in action at companies that produce large software suites. In both cases I consider the approach as a failure.

The problem in both cases is that the number of staging trees/branches/projects has an ever increasing slope, thus consuming ever more manpower to manage the ever increasing number of staging projects.

While the original problem of "how do we deal with unknown adverse interactions between updates" remains unresolved. The "solution" to this problem taken in one case was to have intermediate staging trees where "known risky updates" were tested together. Yes, staging trees upon staging trees. But this only solves the problem superficially as the target tree will move ahead and thus the staging tree by definition is always out of date. Unless the target tree is frozen until a particular staging tree is merged. Anyway it is a maze that requires potentially a lot of people.

The other problem with the staging model is that the "potentially risky interactions" knowledge is an implicit set of interactions that the staging tree managers happen to know. This is not expressed anywhere and thus makes it difficult for other people to learn. We have this problem today and from my point of view this will not be resolved with more staging trees.

The staging model will not catch adverse interactions reliably. The reason is that by definition the staging tree is always out of date, unless the target tree is frozen and after one staging tree is accepted all other staging trees get rebuilt. This is not conducive to parallel development.

What I like in your little umm rant ;) is the notion of *interactions* that need to be tested. What I'd like to see in this whole discussion in addition is 'cadenced flow' and 'integration decision'. When you add these, the resulting staged flow will actually get both bug fixes and package or subsystem upgrades available fast and in stable quality, continously (bold claim) here's how: Let's assume there is some code stable base, call it Factory. The goal is to get updates in there, reliably, regularly, to get it to the next level of being a stable code base. For "leaf packages" that is simple: build the package, test it's functionality, then release it. For for lack of a better word "intermediate packages" (say libpng), you start like above, testing the functionality, until from that perspective it's good to release. But then you also need to get it integrated with "the rest of the world", (maybe 50-100 packages for libpng, 44 on my system). Then there is these "multi-scope packages", like NetworkManager or the bluetooth stack, which affect several whole integration scopes, desktops in this case. they have interactions in two stages (first the desktop, then all other desktops). There is the "transversal packages", affecting almost everything, like the toolchain. And then there is packages affecting few other packages that nonetheless have a lot of interactions that need to be tested (kernel, xorg, ...) Now supposed there is a cascade of staging projects, which potentially 'release', say, every week or every other week (that's the "cadence"). They build a tree structure, something like this: a t \u o \l \l \t o \i \e \o l \b \a gcc \- s \s \f --------------------------------------------------------> Factory / / /\l n/ / \x r/ K/\G \d e/ D/ \N \e k E/ \O \M \E The number of nodes from the root (Factory) to a branch corresponds to the interactions that need to be tested for what goes into that branch. This gives growing rings of scope for interaction testing and integration succes. Successfull build and automatic tests are necessary, sometimes even sufficient for interaction test and integration success. They propagate automatically, to give a 'tentative' next build. That, however, does not affect the 'last known good' build --- that last known good state remains available, too. Thus at each branch, every week a decision can be made: is the combination of the 'new' stuff good enough already to *pull* (!!) it in together? or do we --- for the combination! --- have to stick with what we had so far, the last known good version? btw, 'remote' breaks of the last known good source version, because of some, say, toolchain upgrade, clearly indicate said root cause, the toolchain upgrade, needs some love, too. Integration Manager, set priorities... what do you want in this week's Factory? maybe the new toolchain will not be there yet... A GNOME release that needs a new NM or bluetooth may need a number of cycles to get to a shape where it can be merged with KDE. This will also lead to races. For example a new gcc might race a desktop integration, i.e. the new gcc works with the last known good version of the desktops. Then the new desktop integration will need to do their homework and pull in the new gcc, and until then, the new gcc will just build the last known good desktop. Also a new GNOME will have at least to ensure integration issues are resolved with the current stable KDE (last known good), and hopefully the KDE guys are good citicens and are willing to spend the time to make sure this works. They will. They need the GNOME guys to do the same a few weeks later... Sometimes for several weeks the integration master needs to pull some old versions (the last known good combination), just because the new combination is not ready yet. Now how can small leaf package updates skip such a major barrier? e.g. a new gimp? At each junktion it is clear what needs to be tested. If the new leaf gtk+ application, gimp, can also be integrated with the 'last known good version', the one that is still in Factory, then it can be integrated into that and then thus moves on, ahead of the rest of GNOME, at the next cycle. So if we tilt the above tree and look at it sideways, it almost looks like git integration: new stuff merge success ---------------------------------------------*-------- / \ / \pull new gimp / \ last known good --------------------*-------------------------------R.I.P. Such a system of 'staging' or integration projects gives a clear flow of both updates and upgrades into a well integrated and tested Factory, which is 'released' at a weekly cadence. The cadence also scopes the size of the integration projects: they should be small enough to allow something like a weekly or at most bi-weekly lock-step. Beta users for some integration point or branch can use this a few days ahead of the release, (zypper rr + zypper dup gets them sane again) A new kernel can be available for pull for a longer while, until it has enough love and testing to actually be pulled in as The Kernel, how to manage kernel beta test is an own excercise when we get a stable Factory. Branch projects can also be used by those happy with a partial integration (new KDE even if it breaks GNOME or vice versa), or experimenting early with completely new feature sets (systemd, ...). I believe this model overcomes some of the issues Robert has brought up about the traditional 'staging' model. It gives clear responsibilities and a clear cadence. No model can solve the problem *who* is going to do the integration work, but this staging model at least clearly scopes and cadences what needs to be done and when to keep the flow going. my .02 EUR S. -- Susanne Oberhauser SUSE LINUX Products GmbH +49-911-74053-574 Maxfeldstraße 5 Processes and Infrastructure 90409 Nürnberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Robert Schweikert - 19:39 2.12.13 wrote:

...

Well, I do not think it is that simple. One could argue that Perl is a leaf package.

No one cannot. Perl is really nice example what is NOT a leaf package. Too many things depends on Perl. Leaf package is mc, linphone, maybe pidgin (although it has plugins, I believe these are built from the same source).

A similar argument can be made for KIWI, which depends on a lot of leaf packages but, KIWI is very important to create our ISO images. Thus, the line for leaf packages is blurry at best.

KIWI is a leaf package as no other package depends on it. KIWI maintainer should check before submitting it into Factory that it works. If some bug slips and we are not able to produce DVDs, fine, we are not going to do it for Factory every day anyway.

...

Yes, however, what is being neglected is that there is a fundamental problem with the cadence. The cadence itself is influenced by the process, through rebuild times and other snafus that are inevitable.

Yes, we will slow down, but we will make sure that stuff works and doesn't break anything. I can live with getting new gcc month later if my computer can work in the meantime.

For illustration purposes lets say the cadence for the autotools staging is every other Monday, and the desktops get their say every other Wednesday on the off weeks, i.e. auto tools goes weeks 1, 3, 5 and so on and the desktops merge in weeks 2, 4, 6, and so on. At the beginning of week 2 the autotools merge has to be completed in order to give the desktop staging tree sufficient time to rebuild to meet it's merge window on Wednesday. During this time (Monday of week 2 until end of Wednesday in week 2) nothing else is allowed to be merged into factory or the desktop staging tree would have to be rebuilt again. That's all fine but we have a time problem....

Why fixed schedule? We should do it *on demand*. Only when new change comes. And we can probably skip stagings (in case we have not enough power), for simple bugfix releases. How long does it take for new gcc to get stable? A year? We can do staging for it once a year :-) -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Robert, thanks for your feedback. There is one important thing I did not state clearly which is the interactions between the branches (like libs underlying and affecting desktop, or what Coolo mentioned, libs that are part of some desktop and that are used by packages outside the desktop). Robert Schweikert writes:

On 12/02/2013 12:16 PM, Susanne Oberhauser-Hirschoff wrote:

...

Let's assume there is some code stable base, call it Factory.

The goal is to get updates in there, reliably, regularly, to get it to the next level of being a stable code base.

For "leaf packages" that is simple: build the package, test it's functionality, then release it.

Well, I do not think it is that simple. One could argue that Perl is a leaf package. But we have perl-bootloader and thus and update to Perl could break perl-bootloader which in turn would be a bad thing with pretty far reaching effects. Father, perl-bootloader does not stand unto it's own, it uses Perl modules that should definitely be considered as leaf packages.

A similar argument can be made for KIWI, which depends on a lot of leaf packages but, KIWI is very important to create our ISO images. Thus, the line for leaf packages is blurry at best.

There is packages that without much harm can be integrated late because they have very little to no impact on other packages. That to me wrt integration is a leaf package. perl is certainly not amongst them. And this whole tree is arranged by a reasonable (heuristic) order of integration.

...

Now supposed there is a cascade of staging projects, which potentially 'release', say, every week or every other week (that's the "cadence").

They build a tree structure, something like this:

a t \u o \l \l \t o \i \e \o l \b \a gcc \- s \s \f --------------------------------------------------------> Factory / / /\l n/ / \x r/ K/\G \d e/ D/ \N \e k E/ \O \M \E

The number of nodes from the root (Factory) to a branch corresponds to the interactions that need to be tested for what goes into that branch.

This gives growing rings of scope for interaction testing and integration succes. Successfull build and automatic tests are necessary, sometimes even sufficient for interaction test and integration success. They propagate automatically, to give a 'tentative' next build. That, however, does not affect the 'last known good' build --- that last known good state remains available, too.

Yes, however, what is being neglected is that there is a fundamental problem with the cadence. The cadence itself is influenced by the process, through rebuild times and other snafus that are inevitable.

[waterfall, one common build target in all branches at any time, hundreds of staging projects, thesis on how (toolchain) changes propagate]

Ah. I definitively did not communicate clearly. Every package needs two ways to get into the next stable base, the next Factory: very quickly flow through the system, no matter which branch you twig off, because the integration impact is low. Or in flow with the big wave that comes down the integration flow. The first path is what I call the 'fast track'. It is for minor updates, patches, fixes, things that don't impact much and can be released with little integration testing, or sufficiently reliable automatic tests. The second path is the one for things that really need integration work. That path is indeed taking longer but I don't get to match your math with my model. I don't see why a usefull tree would have more than a few dozen branches with a total depth of at most half a dozen stages. Then there is an asumption that there is exactly one build and integration target of new stuff. That indeed would cause the dealays and friction you describe. However packages added at a branch are built for both what is comming next from upstream and the 'last known good state' from upstream. First let's look from above: \ \ OTHER BRANCH \'devel' \'devel' \ \ \ 'integrate' 'devel' \ 'integrate' \_____ next cycle \_____ _____*--------------------------*------ / decision decision / / /'alpha' THE FORK UPSTREAM THIS BRANCH DOWNSTREAM pull add new stuff locally integrate integrate with other release (or don't) In a real world factory production line (steam, dust, oil) that is using kanban, when you move a part down the line, at well defined points you will send a 'signal kanban' to branches that are going to be merged soon. Like "make me a new engine, I'll need it in a short while". This is how rebuilds and automatic tests should be triggered for what you anticipate to provide soon. So here comes the part that I did not spell out clearly: There have to be *three* target builds for each branch, built from *one* source: - Build for the last accepted good state, Factory unmodified. This gives you clarity on whether your local changes work. - Build for the 'fast track' from upstream, changes that heuristically have no or only well known integration impact. - Build for the proposed next state bigger update from upstream, if any. The "chaperones" of each branch will make two major decisions during each cycle: At the beginning of each cycle they decide what to do in this cycle in this branch: small changes? big changes? and when will they start integration through 'the fork' with the 'other branch'? They may then work on three possible things, until they make the cycle end decision: * If there is a fast track from upstream, they will move that along, possibly adding their own fast track stuff, possibly negotiating with their upstream what has priority, upstream additions or their own. They might also release it mid-cycle to further accelerate downstream processing of this fast track. * If there is work on their branch they work on it, possibly getting it ready as next upgrade. * They switch to integration with the 'other branch' to have a next integrated version ready for their downstream. They may not do this at all during a cycle. That's saying: no upgrade this time from this branch. At the end of the cycle the "chaperones" decide whether they release their work for their downstream, so they decide what they offer to their downstream to start the next cycle with. They also cosnciously decide what they provide as fast track to their downstream: the fast track they got from upstream (only doing integration with the other branch), only their own fast track stuff, holding back the fast track from upstream, or both. The real novelty of this tree structure is clear focus and scope for integration: What do I need to ensure at this decision point? It is *not* less work. It just gives the work that needs to be done clear areas of responsibility: in each branch, the work is local. At each fork, the work is local for two branches, meaning collaboration. At the end of each cycle there is two outputs after each fork: - a fast track build - a major update/upgrade build So at the end of each cycle, there is a sane starting state for the next branch downstream to pull from and build upon. It is then their job to try with the most recent 'good to pull', the most recent 'next version'. The more I think about it, the tree resembles the merging part of a git merge graph. The novelty here is, that this proposal structures the branches around things that integrate easily locally, in the branches, and where the real integration then needs to happen, at the bifurcation/branch point/fork. The tree structure gives a heuristically proven order to usually best do this. In the past there was concern about an explosion of builds in such a model. This doesn't happen, though: you only build for - the last accepted good state. It gives you clarity on what you have locally changed. this will be your fallback fast track output for your downstream. - the proposed minor updates to said last known good state. this is accepting things on the fast track upstream, for integration with your fast track upgrades. If you manage to use it successfully, *this* will be your fast track merge with the other branch and your joint offer to your downstream. - the proposed next state from upstream. This is where you also contribute your big changes. So there is no build explosion. You decide which of your results is good enough to be provoded to your downstream in two buckets (fast track and big update/upgrade), plus the underlying factory. If you scope rebuilds to not percolate ahead of your integration review and test, it may even do fewer builds.

...

Thus at each branch, every week a decision can be made: is the combination of the 'new' stuff good enough already to *pull* (!!) it in together?or do we --- for the combination! --- have to stick with what we had so far, the last known good version?

There is no way to know if you can pull things in together because staging branches do not get cross built against each other. In the figure above everything is nicely spaced, but that probably does not reflect the real world. If libs and the desktops are ready at he same time one can still not merge them into factory at the same time because they have not built against each other, they have built against "current" factory.

They won't be ready at the same time in this way. At each junction point, these options exist: - just one of the branches is good enough - no branch is good enough - both branches are good to merge Only if both are good to merge, integration builds need to be created and tested and then, based on that result, be provided one level further downstream or not. So downstream will always see three options of base builds: - last known good state (no changes, aka "Factory") - fast track changes - serious new integration changes And downstream will make a conscious decision what they can digest at this point in time.

But the technical requirement creates a people problem ;) . People hate the waterfall and hurry up and wait stuff.

That's what the 'fast track' option is for. Which should really only be used for things that should create little to no pain, and that should be an easily overseeable amount of changes, where the chaperones at each branch can tell what to look at.

Therefore, what tends to happen is that multiple staging branches get merged into the reference branch based on heuristic historical data of "no adverse interaction when merging a given set of branches in the past". This data has a number of problems:

- past behavior does not guarantee future performance if perl-bootloader or kiwi depend on a new leaf package the heuristic data of those staging trees is useless as a new set of interactions is created

- the heuristic knowledge is intrinsic to the chaperone of the reference branch (granted, this is not necessarily much different than it is today, but we are looking for improvement and not "the same")

- the bus factor remains 1, i.e. the chaperone of the reference branch

To truly solve that you'd need to solve the halting problem... What alternative do you see to structure integration work based on heuristics or even measured past performance?

...

At each junction it is clear what needs to be tested. If the new leaf gtk+ application, gimp, can also be integrated with the 'last known good version', the one that is still in Factory, then it can be integrated into that and then thus moves on, ahead of the rest of GNOME, at the next cycle.

But this implies that gimp has it's own staging branch, thus one is feeding the "ever expanding number of staging branches" monster. One cannot pull a part of a staging branch without placing the pulled pieces into a staging tree of its own and building and testing that staging tree against a "frozen" reference branch.

No monsters under this bed: gimp is on one of the branches. It is built to 'fast track', 'known good' and 'next integration'. That's it.

...

So if we tilt the above tree and look at it sideways, it almost looks like git integration:

new stuff merge success ---------------------------------------------*-------- / \ / \pull new gimp / \ last known good --------------------*-------------------------------R.I.P.

True, but one still has to build and test the cherry picked stuff, i.e. that's where the need for yet another staging project is created. This rests on the basic assumption that only stuff built and tested against the reference branch can be merged.

Again: No additional staging project. Just the three build targets. And the *build* is what is handed down or rejected. The 'last known good' is the reference tree you mention. The 'fast track' is updates with (heuristically) low impact. The 'integration' is upgrades with (heuristically) high impact. Things on the fast track propagate quickly to become the next 'known good' state. As there is no skip lining at integration decision points, they will trigger rebuilds of all dependent packages in all branches. if one of them turns the update down (by not using it as the base for their fast track), communication and negotiation and fixes are needed, corresponding to what would normally happen where the two 'normally' meet. Once the 'fast track' is connected all the way from all tips to the root, it is the next 'known good'. Likewise 'integration'. You also mentioned base libs getting ready at the same time as some app. Well, one of the two will be further up in the tree, and will arrive at the end earlyer. They at the same time get into their respectvie next build, which then can be handed down. If we assume a tree of 6 steps and a weekly cadence that's a maximum of 6 weeks, unless the change is on a fast track. (Lightbulb) now I get something I didn't see before :) In a situation like this, say you worked something all the way from B to ABCDE, whcih was painfull and too several cycles. D\ /E \B \C * \F \ \ \DE \ ------*-----*-------*------*------- A AB ABC ABCDE ABCDEF Now A is already working on the next big thing, and they could do it faster, but not as in fast track they might want to catch up. They don't, B is going in the next ABCDEF. So what I realized is that this is rather layers of upgrades, generations, integration versions, releases, cadence ids, whatever you may call it. In this model, you create new builds in your branch, based on a latest common release, and these are handed downstream and because this is cadenced, each cycle will produce an integrated result at the end. So it's not two layers (fast track and next generation) but no harm is done, because the number remains reasonably low: The number of layers is the number of cycles a branch is releasing a new base for their downstream, be it fast track (update) or full integration (upgrade). And that number should be limited. If a branch is piling up releases that are never picked up and never make it all the way to the user, something is going wrong. Btw, if the builds are tagged by 'quality', a power beta tester user can request beta level for some branch. An end user can request release quality for some branch. Developers could also pick alpha level code for testing, before they tag it beta. S. -- Susanne Oberhauser SUSE LINUX Products GmbH +49-911-74053-574 Maxfeldstraße 5 Processes and Infrastructure 90409 Nürnberg GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Robert Schweikert - 14:18 12.12.13 wrote:

... @ ALL TECHNICAL CONTRIBUTORS/DEVELOPERS (sorry for shouting) - If we collectively manage to "calm" factory down to the point where it will always boot and you do not have to fiddle with the very low level, kernel, bootloader, glibc , X11, things after doing "zypper dup" would you be

- inclined - very likely - unlikely - very unlikely

to switch to factory on your every day working machine?

Those that run factory already should probably not answer this question ;)

Well, I run factory on my less used desktop and I run it on my notebook since some beta or something when it should be calm enough. If we stabilize it, I'll run it on my notebook all the time and I'll migrate my server to Factory :-)

... One of my primary concerns that has not really received an answer, I think, is of the number of people that now get the "promotion" to staging tree manager.

- do we have people that work at the level of code that requires staging trees willing to take on the "new", additional work/responsibility?

Well, people have to do it nowadays anyway, they just break Factory and have to fix it in Factory while Factory is broken while now the breakage will move into staging tree and get merged once it's fixed.

- are we introducing a new set of arbitrary decision points?

Kinda. Currently reviewers have rpmlint and several bots to help them with review, in the new workflow, they will two additional optional "helpers" - staging projects and some basic QA.

- do we need to think of trying to develop/find a number of people that are interested in being primarily staging tree chaperones?

I do not believe that we should have special role/people for staging tree management. It should be managed by whoever is the maintainer of the package that this project was created for. This person have the most knowledge to see/decide whether he has a bug in his package, or other people depending on him. But there will be definitely some notification actions needed in OBS to help the manager...

...

So we want to have a distributed, balanced and robust development process ie share the workload among teams with different roles. To make sure we get motivated people for those teams the process needs to be set up in a way that promotes mentoring and recognition➄.

Well, yes. But I'd say of all things we probably struggle with the recognition part the most, from my perspective at least.

Well, there is this Karma thing to do the recognition...

... Will a more everyday usable factory automatically lead to a reduced effort of testing in the end game?

I believe yes. Not only testing, but also integration. Nowadays coolo has to hunt everybody down to fix their stuff so he can release a milestone. Getting milestone to compile and produce DVD is actually a lot of work. And if that would be done in Factory during development continuously... -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 12/12/2013 02:42 PM, Michal Hrusecky wrote:

Robert Schweikert - 14:18 12.12.13 wrote:

...

... @ ALL TECHNICAL CONTRIBUTORS/DEVELOPERS (sorry for shouting) - If we collectively manage to "calm" factory down to the point where it will always boot and you do not have to fiddle with the very low level, kernel, bootloader, glibc , X11, things after doing "zypper dup" would you be

- inclined - very likely - unlikely - very unlikely

to switch to factory on your every day working machine?

Those that run factory already should probably not answer this question ;)

Well, I run factory on my less used desktop and I run it on my notebook since some beta or something when it should be calm enough. If we stabilize it, I'll run it on my notebook all the time and I'll migrate my server to Factory :-)

...

... One of my primary concerns that has not really received an answer, I think, is of the number of people that now get the "promotion" to staging tree manager.

- do we have people that work at the level of code that requires staging trees willing to take on the "new", additional work/responsibility?

Well, people have to do it nowadays anyway, they just break Factory and have to fix it in Factory while Factory is broken while now the breakage will move into staging tree and get merged once it's fixed.

Sorry, for not expressing this correctly. Yes, people that break factory have to help fix factory. But there are more eyes on factory, more help and more "pressure" than we will get in staging tree XYZ. Thus the incentive for "everyone" (and I use everyone very loosely) to jump in and help is much greater as compared to a staging tree. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am 12.12.2013 21:40, schrieb Jan Engelhardt:

But for anything that was not already run during the build process — such as UIs — you need users, and you can only get them in Factory.

Well, parts of the plan is to integrate openQA in the build process and the results there don't need users. Of course you still need users to find the *real* problems, but you can remove most of the problems that these users struggle with. Uninstallable updates, obviously broken boots, ... Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thursday 2013-12-12 20:18, Robert Schweikert wrote:

@ ALL TECHNICAL CONTRIBUTORS/DEVELOPERS (sorry for shouting) - If we collectively manage to "calm" factory down to the point where it will always boot and you do not have to fiddle with the very low level, kernel, bootloader, glibc , X11, things after doing "zypper dup" would you be

- inclined - very likely - unlikely - very unlikely

to switch to factory on your every day working machine?

No (very unlikely). I favor reasonably current software, but not a continuous update stream that Factory or Tumbleweed has, as running zypper often would spend my time without bringing me an immediate benefit — the set of {packages with new versions} that I am interested in is bound (=limited). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 12/12/2013 04:02 PM, Stephan Kulow wrote:

Am 12.12.2013 20:18, schrieb Robert Schweikert:

...

I think we have long way to go to figure this out.

Hi Robert,

You appear very stressed. I wonder why - it's Christmas after all :)

Let's give this whole thing a more personal touch - instead of all that 'we's:

I want to improve our development process, because *I* feel the pain with it.

Ack, and we discussed this probably 2 years ago that something will have to change.

You don't and that's perfectly fine - but I spend every morning (this includes weekends) reviewing what hell broke lose this time. I implemented several measures and convinced my colleagues to implement even more. So our team was busy the last year or so:

- openqa improvements - factory-auto reviewer - groups of requests - repo checker - cycle detecter - Factory:Rebuild syncer - automatic setup of staging projects - webui improvements for reviews - legal-auto improvements - faster factory status page - working mail notifications for OBS ...

All great stuff no doubt about it and this makes things better for everyone.

But OBS lacks some features to get a good grip on staging with so many packages, so I experimented with fixed source rings (openSUSE:Factory:Build is the bootstrap cycle, openSUSE:Factory:Core is the minimal+X11 DVD, I'm not yet sure what the next ring will be, most likely KDE+GNOME live cd combined).

And it's *really* hard to do that as a side project - there is always something coming in the way, so I kind of hijacked Agustin's "2016" thingy to make the team aware about the basic problems of our current development process and we sat together to brain storm. Unfortunately we didn't even have the time to document the brain storming in a way fitting to a mailing list (I'm sure we still have the picture of the white board if you're interested in *that* form of getting informed :).

So the whole thing sat around a bit and when we discussed it again, we noticed some gaps and even more ideas. Agustin then made matters worse by insisting on having these ideas presented as part of a long term strategy - which it's not really. If you look closer, all we're proposing are 3 things:

- further improvements to openQA (that's a pretty small tool if you look closer, we can easily estimate the effort there - even though we can be wrong) - taking staging projects seriously including QA (that will require a lot of experiments and as previously discussed it might be easy or a total failure - no one knows) - talk about an improved trust system instead of treating all packages and all contributors the same. The discussion so far has shown that there is more to it than you believe when you sit and brain storm, but that's perfectly fine, isn't it?

Will this have an impact on the next release? No idea! Did I know for certain we would be able to release 13.1? No, I didn't. We still managed to.

As I said in another thread: I'll happily ditch 13.2 or the 8 months cycle if we find something better - and I do have my preferences there, but I don't think we even need to discuss it right now.

What I ask you (and this is Stephan not the openSUSE team): give us some credit.

You have all the credit in the world from my side. As I said, in this thread I think; All I can do is offer up my experience with a staging model. I have lived within it at the very lowest levels of application code development and tools development. I will not be able to help with any implementation in OBS, or other parts of the system. In the end those that do the work have to make the decision. The distribution and development certainly does not hinge on any packages I maintain.

Let us take some risks - and don't expect that we have all answers right now.

Sorry if it came across that way. I do not expect you personally or as the openSUSE Team, to have all the answers. But we should be able to work together at least on a list of things that may or may not highlight potential pitfalls to avoid storming off and implementing things to end up with an "oh shit" moment. There'll be plenty of those no matter how much we talk, that's a given. But there are also plenty of those that can be avoided.

I hear everyone talking about their expectations from the openSUSE team, but I don't see too many asking where they can help. So I guess, I have a question too: Why is that?

Let me hazard a few guesses ;) - everyone is extremely busy already - people are uncertain where they potentially can help - the intermixing of the rather lengthy discussions of tangentially related things has a negative effect on engagement. Anyway, if you or the team have thought about the concerns I have raised, and even if the answers are not complete, then it shouldn't be that hard to just acknowledge the risks and the potential pitfalls. If this has happened in one of the responses and I missed it I apologize. I do not expect perfection or having answers to all questions. All I can hope for is that my experience that I am trying to convey/share will not fall on deaf ears. As I said, I am so hopelessly overloaded that contributing any code to this effort is currently not even thinkable. Those that do decide. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am 12.12.2013 23:17, schrieb Patrick Shanahan:

* Stephan Kulow [12-12-13 16:03]: [...]

...

What I ask you (and this is Stephan not the openSUSE team): give us some credit. Let us take some risks - and don't expect that we have all answers right now. I hear everyone talking about their expectations from the openSUSE team, but I don't see too many asking where they can help. So I guess, I have a question too: Why is that?

I hazzard that the problem is fit. Where do I fit? I can script a little, I test what I use and make bug reports. I know a lot, but that lot is a little about *many* things, and am master of none. I run Tumbleweed and have *many* add on repos for things not "main-stream" Tumbleweed. I have enough experience to get into trouble, and usually find my way out but occasionally need help. The help is *usually* forth-coming and I usually learn a little more.

Provide me tasks "where I fit" and I will help as time permits and I have a lot of time when my grandson is between soccer seasons.

I still see "fit" as the problem, especially for "non-programmer/packager" types.

Thank-you all for your many and continuing contributions.

See Robert? We already have the first "staging champion" :) Thanks Patrick. I have nothing to give you right away, but it's good to know there are indeed helping hands. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi, engagement should be part of development. I wrote somewhere else that from my experience in managing development projects, mixing people that works full time with others working just a very few number of hours in the project is very challenging. Making this approach efficient requires quite some effort also from the core team. This effort cannot be an aside task of the full time developers. It need to be a full time task for at least one of them. I am determined to work on this area, so we can engage people like you in the development of the changes. It won't be easy but we will work on it. It can make a difference. On Friday 13 December 2013 20:39:41 Per Jessen wrote:

Stephan Kulow wrote:

...

Am 12.12.2013 23:17, schrieb Patrick Shanahan:

...

* Stephan Kulow [12-12-13 16:03]: [...]

...

What I ask you (and this is Stephan not the openSUSE team): give us some credit. Let us take some risks - and don't expect that we have all answers right now. I hear everyone talking about their expectations from the openSUSE team, but I don't see too many asking where they can help. So I guess, I have a question too: Why is that?

I hazzard that the problem is fit. Where do I fit? I can script a little, I test what I use and make bug reports. I know a lot, but that lot is a little about *many* things, and am master of none. I run Tumbleweed and have *many* add on repos for things not "main-stream" Tumbleweed. I have enough experience to get into trouble, and usually find my way out but occasionally need help. The help is *usually* forth-coming and I usually learn a little more.

Provide me tasks "where I fit" and I will help as time permits and I have a lot of time when my grandson is between soccer seasons.

I still see "fit" as the problem, especially for "non-programmer/packager" types.

Thank-you all for your many and continuing contributions.

See Robert? We already have the first "staging champion" :)

Thanks Patrick. I have nothing to give you right away, but it's good to know there are indeed helping hands.

Once we publish a list of jobs/tasks that need doing, I'm certain we will quickly find that there are plenty of helping hands available.

-- Agustin Benito Bethencourt openSUSE Team Lead at SUSE abebe@suse.com -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Robert Schweikert wrote:

On 12/12/2013 10:59 AM, Ludwig Nussel wrote:

...

Stephan Kulow wrote: At the same time however more users that actually use Factory are needed to find issues early and fix them quickly.

Yes, also a point that is difficult to argue. [...] Extremely difficult to answer of course, I know, thus maybe we can try a different approach to get closer to a picture that helps us.

@ ALL TECHNICAL CONTRIBUTORS/DEVELOPERS (sorry for shouting) - If we collectively manage to "calm" factory down to the point where it will always boot and you do not have to fiddle with the very low level, kernel, bootloader, glibc , X11, things after doing "zypper dup" would you be

- inclined - very likely - unlikely - very unlikely

to switch to factory on your every day working machine?

Hard to answer at this point too. Depends on whether we get the right balance between having the latest and greatest hot new features quickly and not breaking too badly.

I think there are many open questions about the staging, and maybe that's just me, and I am more than happy to just shut up if it is just me.

It's not just you. I also have my doubts but at the same time think it's worth a try. Esp since we have the chance now to not introduce this as stand alone feature but rather back it up with other other technical as well as social actions.

One of my primary concerns that has not really received an answer, I think, is of the number of people that now get the "promotion" to staging tree manager.

As Michal already wrote the current idea is to make the packager whose submission originally triggered the staging project responsible. If it turns out that some people are interesting in helping out in staging projects no matter what topic they are about that would be even better of course :-)

- are we introducing a new set of arbitrary decision points? For this question I'll briefly revisit some earlier points in the discussion. I asked whether Richard would be willing to take on the new additional work and Stephan pointed out that gcc has a higher level "priority/interest" over other changes that might require staging trees, Stephan mentioned the "usr merge" effort as another example for a staging tree project. So who decides what project/movement gets the "higher level of interest" stamp, presumably more help from everyone, over the you are stuck with it staging branch? I suppose this is where

I guess it's kind of natural. If one gets stuck with your staging branch because noone wants to help then maybe the change was not that great after all.

the potential change process comes into place to avoid arbitrary decisions. However, this feels a bit like changes in the factory model forcing other changes, such as a change process, upon everyone. It appears that we should be able to separate those a bit better.

I'm not sure I can follow you here.

...

All the work for that cannot be put on the shoulders of just one guy.

Yes, but that's not a new concern, I remember having a discussion about that at oSC11.

Which actions were taken back then to address the concern?

For 12.3 and 13.1 the openSUSE Team as a combined force working in unison has done a lot to get the release out the door. I think it is generally accepted that the quality has gone up over previous releases.

I take that as compliment :-)

Do we have a list of things we can drop, i.e. that were done but are more nice to have than absolutely necessary?

What is the absolutely necessary? progress.opensuse.org lists most not so technical tasks at least. http://en.opensuse.org/openSUSE:Public_release_action_plan_12.2 has lists of things we started with before we had progress if you want to look at something simpler. Some tasks were already dropped for 13.1. For example taking care of the manuals. On the technical side there's for sure also room for simplification. Do we need 14 iso images for example? How about dropping the Live and Promo images? i586? Rescue? Do we need 6 "Desktops"? Full hard disk encryption?

Can we do with less effort in testing at the end stages of the release cycle?

Will a more everyday usable factory automatically lead to a reduced effort of testing in the end game?

Depends. Not having to deal with really basic problems like crashing installer can mean less effort or more focus on the polishing with the same resources.

Is there a proposal coming that will question the 8 month release cycle? How would a longer release cycle benefit the elusive "end user" that cannot possibly be captured in one group? We have already heard from some people that run servers and say "longer is better" but I am certain we can find just as many "end users" that say 8 month is just about right.

The success of Tumbleweed and Evergreen at least suggests that openSUSE releases leave things to be desired on both ends of the spectrum. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am 12.12.2013 20:18, schrieb Robert Schweikert:

@ ALL TECHNICAL CONTRIBUTORS/DEVELOPERS (sorry for shouting) - If we collectively manage to "calm" factory down to the point where it will always boot and you do not have to fiddle with the very low level, kernel, bootloader, glibc , X11, things after doing "zypper dup" would you be

- inclined - very likely - unlikely - very unlikely

to switch to factory on your every day working machine?

I would be inclined to search something not "calmed down", as i'm very happy with factory on my main machine as it is today. Where will all the fun be if it always just works? :-)

Those that run factory already should probably not answer this question ;)

Oh, come on. We should really still provide something that resembles todays factory for the brave! Actually for me, running factory (today) is less painful than upgrading twice a year: I have small breakages every now and then, but seldom really big stuff. And I have to fix one thing per week. After updating my server from 12.3 to 13.1, I have to fix up lots of small things, all at once, to get it working as intended again. And I won't even talk about updating my kids' machines (they use GNOME -- nothing anyone ever would even consider when running Factory :-)), that is always a hell of a task to get running again. I'll probably also might switch them to Factory / XFCE. Best regards -- Stefan Seyfried "If your lighter runs out of fluid or flint and stops making fire, and you can't be bothered to figure out about lighter fluid or flint, that is not Zippo's fault." -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am 19.12.2013 22:10, schrieb Dimstar / Dominique Leuenberger:

On Thu, 2013-12-19 at 14:45 +0100, Stefan Seyfried wrote:

...

And I won't even talk about updating my kids' machines (they use GNOME -- nothing anyone ever would even consider when running Factory :-)),

Why? :) I've been running Factory/GNOME since 2010 without ever re-installing the system from scratch on this very machine.

GNOME updates have been painful for me: Stuff that worked before is no longer there in the next version. Extensions do no longer work and / or are not available for the current version. You can no longer suspend the system without somebody explaining how it works, there is no extension to bring the "suspend system" button back. My kids are really pissed about these constant changes, and they don't use Factory (yet :-) That's the "update to next release" problem. While I was running GNOME on Factory (for a pretty short time), it just broke too often (and even though I liked some of the Ideas, the complete package was not my cup of tea). What I like about XFCE is that there apparently are just enough developers to keep it working, and that apparently nobody is trying to implement his "vision" :-) But it is probably just a preference: I prefer to run kernel of the day and a few other more-recent-than-Factory packages, mostly from areas where I have some experience. If these things break, I'm prepared and can handle that. You prefer running latest unstable GNOME -- if it breaks, you can handle that. To each his own, I suppose :-) Best regards, seife -- Stefan Seyfried "If your lighter runs out of fluid or flint and stops making fire, and you can't be bothered to figure out about lighter fluid or flint, that is not Zippo's fault." -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

David C. Rankin - 15:59 12.12.13 wrote:

...

One distro that has done a very good job at making the rolling release work is Archlinux. In implementing their rolling release they employ two basic collections of packages. The use a "testing" repository where packages incorporate the latest stable source from all vendors and then ultimately move to "core" and become the Archlinux packages. Their "testing" is essentially what you envision as "staging" and "core" is what you have as factory.

Having staging projects goes one step further. There are two aspects that speaks for staging projects instead of yet another testing layer IMHO. a) Staging project has a limited scope, so when something break, it's much easier to figure out what caused it. But it waste some resources, I'll admit that. b) People will not test testing and will concentrate on Factory. c) And once you decide to move packages from testing to stable, you have the same issue as when you want to move packages from devel project to the Factory. Unless you move completely everything, there will be some dependencies that you forget in testing and will not work the same in stable... -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org