Togan Muftuoglu - 20:26 2.12.13 wrote:

...

...

...

...

...

"Michal" == Michal Hrusecky writes:

Michal> Hi, Michal> we are having a nice torrent of ideas in this mailing list since coolo Michal> presented this proposal last Thursday: Michal> https://progress.opensuse.org/workflow/factory-proposal.html

Michal> Some of them are related to staging projects, so I will try to Michal> expand further one of the options regarding what could be Michal> happening inside that dashed box in the diagram.

Thanks for the explanation, as the boxes in the proposal was not easy to follow with my eyes ;)

Michal> 2.2) The submitter coordinates with people who maintain packages Michal> that got broken. That means several people working on several Michal> packages (very often from different devel projects) all together Michal> to fix the new staging project. The fixes can be done in the devel Michal> projects or directly in the staging project.

I am not sure I understand how this one work. For example in one of the gcc staging phase, there was a list of broken packages sent to factory list. I am thinking we want to automate this, rather than sending a list of packages that is manually compiled and send to factory.

In case of gcc new staging project will get created and we will wait a reaaaaallly long time as everything depends on it :-D Than we will have staging project where there are quite some packages failing. We can list only failing packages, but we can simply point people to the web monitor of this staging project.

So in this new approach how is it planned to coordinate effort to fix the failing packages ?

It is just an idea, one of many possibilities. In gcc case, we should probably call for project wide help, but gcc maintainer will probably know the most frequent cases and probably could do some mass submission to fix them.

Michal> This is just a proposal on how staging projects could work. The Michal> main goal of this proposal is to partially move the responsibility Michal> of the integration process into the community of packagers so: - Michal> Staging project maintainer has to cooperate with package Michal> maintainer to get his SR in. - Package maintainers have motivation Michal> to fix their stuff, otherwise new versions don't get included.

What is the delegation of work if the package maintainer is not responding in a timely manner ?

Well, we should probably detect and handle inactive maintainers in general, but that would be out of scope of staging projects. We should somehow handle packages that nobody fixes, packages with security bugs nobody cares about and similar. Currently there is a group called confusingly 'factory-maintainers' that acts as backup for maintainers, but in general, I would leave this open and we can address inactive contributors later in discussion more generally ;-)

Michal> The main advantages if this approach are: Michal> - Everything has to go through package maintainer to make sure Michal> changes don't get lost, package maintainers knows about the problems Michal> and agrees with the solution Michal> - We move from one good state to another, accepting only changes Michal> together with fixes.

Michal> What do you think? What do you like/dislike? Opinions?

Overall sounds fine to me.

Glad you like it :-) -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Sascha Peilicke - 10:17 3.12.13 wrote:

On Monday 02 December 2013 19:52:47 Michal Hrusecky wrote:

...

Hi,

we are having a nice torrent of ideas in this mailing list since coolo presented this proposal last Thursday: https://progress.opensuse.org/workflow/factory-proposal.html

Some of them are related to staging projects, so I will try to expand further one of the options regarding what could be happening inside that dashed box in the diagram.

The main goal of staging projects is preventing Factory from breaking so often by creating a "temporary and alternative Factory" where dangerous and conflicting submissions have to learn to live in peace and harmony before entering the real Factory. The original mail from coolo contains a great example with new automake.

So let's look closer into all those boxes, arrows and decision boxes:

1) Developer sends SR#1 to Factory and review team decides it needs staging, so staging project gets created (let's assume the decision is made by the review team, even if this is currently being discussed in the mailing list). The staging project initially contains only the package that is being sent and rebuilds everything from Factory that depend on it.

The art here is how to make that decisions. If it would be based on past experience, we would always recommend to stage glibc :-)

And probably rightfully. Although I would say that bugfix releases not changing API/ABI probably don't need to go to such an extent.

On the other hand, there's udev, which doesn't break builds but peoples running systems. So staging udev wouldn't reveal anything until people actually test the staged udev.

Well, apart from staging projects, there is need QA part in the proposed workflow ;-)

Though this is a counter-example, I like the general idea.

Glad to hear so.

The Factory maintainers and the review team could come up with a list of packages that are likely to cause issues. For those we could just demand staging, no matter what. We would publish / version the list somewhere and just add a check to factory-auto.

Well, I would make it a little softer as part of the review - if you see that there is just typo in man fixed, no need for staging, if all headers got renamed, we need staging for sure. These to being just extremes, but what I'm saying is that I would like to make it more fuzzy...

Another category would be packages where the submitter demands staging. I expect this to happen far less frequently but we should emphasize that this option is available. Responsible submitters may want to use it if they bring major changes to the distro.

Sure, packages should be able to ask for it as they should know much better how drastic are the changes.

...

2) [snip]

...

This is just a proposal on how staging projects could work. The main goal of this proposal is to partially move the responsibility of the integration process into the community of packagers so:

...

- Staging project maintainer has to cooperate with package maintainer to get his SR in.

To whom does the SR belong here? Is it the staging prj maintainer that wants it or the packager? IMO its usually the latter wanting the SR to get in while the former wants to get rid of the staging project again.

Actually both. It was meant that the guy maintaining staging project has to work together with other people to get rid of his staging project and get stuff in, but it works also other way around. Packager need to cooperate with staging project maintainer to get his new and cool version in.

...

- Package maintainers have motivation to fix their stuff, otherwise new versions don't get included. I think this was discussed elsewhere already, but not-so-core community members are often asking why they should fix their package if breakage came from elsewhere (e.g. new glibc vs. random game pkg). But it's

Well, it works other way around as well, why should glibc maintainer fix something that uses API that was deprecated long time ago. The important part is that they should work it out together, otherwise neither of them can in.

...

The main advantages if this approach are: - Everything has to go through package maintainer to make sure changes don't get lost, package maintainers knows about the problems and agrees with the solution - We move from one good state to another, accepting only changes together with fixes.

I think you are correct that we have to bring the packagers into the fix-it boat. But let's not be overambitious, this won't solve everything :-)

Well, this should solve the failing packages in Factory and make Factory buildable/rebuildable all the time. To make it stable and working, we need QA, that was in another box :-D -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 12/03/2013 11:52 AM, Sascha Peilicke wrote:

On Tuesday 03 December 2013 10:41:59 Michal Hrusecky wrote:

...

Sascha Peilicke - 10:17 3.12.13 wrote:

...

On the other hand, there's udev, which doesn't break builds but peoples running systems. So staging udev wouldn't reveal anything until people actually test the staged udev. Well, apart from staging projects, there is need QA part in the proposed workflow ;-) Ah ok, I have to admit that this thread became TL;DR for me already. So who will do the QA part?

In the proposal, that would be the "QA Team". With heavy usage of a revamped openQA.

Before we propose anything there we should ask our stable release maintenance guys how pending updates are tested (if at all).

[...]

...

...

I think you are correct that we have to bring the packagers into the fix-it boat. But let's not be overambitious, this won't solve everything :-) Well, this should solve the failing packages in Factory and make Factory buildable/rebuildable all the time. To make it stable and working, we need QA, that was in another box :-D What we can improve though is the notification part of it. While we have coolo's reminder mails about new breakage, that's sometimes still overlooked by people (like me).

We are working in a heavy redesign of notifications in OBS. Expect good news about it in the short term. Cheers -- Ancor González Sosa openSUSE Team at Suse Linux GmbH -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Robert Schweikert - 12:59 3.12.13 wrote:

...

Correct. I agree that having basically one person doing this (coolo today) is not a good approach and we need to fix this problem. What I am not so confident about is that those touching the very core packages have the interest/knowledge/energy/time to chaperone a staging branch.

If AJ pulls a new glibc from upstream and tons of stuff breaks in the staging project we are basically asking AJ to be coolo and run after all the package maintainers that now have broken stuff. From my point of view that doesn't really resolve the basic problem, it just shifts the problem onto the shoulders of someone else. That someone else most likely has less time to chase all the broken stuff than coolo. I am not advocating to stay with what we have, I just fail to see how the staging of updates improves the situation overall. Certainly coolo will do less chasing and that's a good thing, but will the distribution as a whole improve or are we more likely to get stuck with older versions because the new chaperones of the staging branches do not have the time/energy etc. to chase everything that happens to break?

You missed the second part. No new version of the broken package will get in unless it fixes the staging project. So now both maintainers are stuck with old versions that work together. And it is in the interest of both of them to resolve the issues.

...

I think what we are actually doing is discouraging changes for packages that have a large impact such as gcc, glibc and others. As I mentioned in another thread, we do not want to encourage "dump and run" but this appears to go toward the other extreme. Basically if one is not willing to chaperone a staging project better not send any things that are potentially disruptive.

No, we are postponing the integration of changes till we are sure that integration will result in something that would work :-) -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Robert Schweikert - 14:23 3.12.13 wrote:

On 12/03/2013 01:21 PM, Michal Hrusecky wrote:

...

Robert Schweikert - 12:59 3.12.13 wrote:

...

...

Correct. I agree that having basically one person doing this (coolo today) is not a good approach and we need to fix this problem. What I am not so confident about is that those touching the very core packages have the interest/knowledge/energy/time to chaperone a staging branch.

If AJ pulls a new glibc from upstream and tons of stuff breaks in the staging project we are basically asking AJ to be coolo and run after all the package maintainers that now have broken stuff. From my point of view that doesn't really resolve the basic problem, it just shifts the problem onto the shoulders of someone else. That someone else most likely has less time to chase all the broken stuff than coolo. I am not advocating to stay with what we have, I just fail to see how the staging of updates improves the situation overall. Certainly coolo will do less chasing and that's a good thing, but will the distribution as a whole improve or are we more likely to get stuck with older versions because the new chaperones of the staging branches do not have the time/energy etc. to chase everything that happens to break?

You missed the second part. No new version of the broken package will get in unless it fixes the staging project.

Hmmm that would require that everyone that is a staging tree chaperone knows about all the broken packages in all staging projects. If the chaperone's do not poses this knowledge a new version of the broken package can enter factory through an unrelated staging branch.

No, staging project maintainer needs has a staging project and OBS shows him, which packages are broken. He can ask OBS who are the maintainers and try to ping them and help them to fix their stuff. He doesn't need to know anything about other packages, he just needs to know something about his and how to communicate.

...

So now both maintainers are stuck with old versions that work together. And it is in the interest of both of them to resolve the issues.

Nope, it is in the interest of the user that they resolve the issue, not necessarily in the interest of the packager.

Well, only if packager doesn't care if he has a package in Factory... What it doesn't solve is inactive - presume dead - maintainers. This is general issue, that should be addressed in different discussion. -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Greg Freemyer - 9:34 3.12.13 wrote:

=== - how SR batches are handled? For instance, if I have even just 2 SRs that I know depend on each other but I want them staged because I don't know what else may depend on them. How do I tell OBS to create a staging tree and include both of my SRs. Obviously for KDE or Gnome, these might be very large SR batches.

You write in SR of the lower package (the one that the other one depends on) please create Staging for this and in the second one that it depends on the first). AFAIK currently OBS doesn't handle dependencies between SRs.

=== - I like the idea of every SR (or small SR batch) going through a staging project. But I think we need 2 solutions: short term (hours) staging projects and long term (days/weeks) staging projects.

-- long term staging projects - this has been the focus of discussion, so I will ignore it here.

-- short term (hours) staging projects

I propose every SR (or SR batch) not tagged for long term staging be evaluated for short term staging viability. The criteria could be "causes 5 or less package builds and all of the package builds take less than 1 hour each".

Once a SR / SR batch is tagged for short term staging:

From here

--- create short term staging project - A BtrFS read/write snapshot of factory is created. This should only take a couple seconds

--- Update the affected packages based on the SR / SR batch - I assume a few more seconds

--- Run the build logic and build the updated packages and anything that depends on them - this is the time consumer, put hopefully this class of build can be given top priority.

to here we don't need special workflow as it would be handled by OBS

--- If no build errors, forward original SR / SR batch to factory reviewers and destroy BtrFS snapshot.

--- if build errors, hoover up the build logs from all the builds and attach them to the SR, reject the SR, and destroy the BtrFS snapshot.

If the above is totally automated, it seems the BtrFS snapshot may only live for a an hour or less typically.

With that process it seems the overhead of creating a staging projects is minimal and even single package SRs of leaf packages could be run through a quick staging compile / dependency compile. ===

Not a good idea to hack a new OBS from scratches living next to the official one. And I don't see what should be the difference between the short term and long term ones. Short ones are rejected right away without striving for solution in staging?

-- How many "leaf" SRs per day are there anyway?

No idea.

-- How many simultaneous BtrFS snapshots can exist and still get reasonable performance?

No idea and doesn't matter.

--- If one assumes 10 simultaneous BtrFS snapshots is reasonable to maintain, then a queue could be established to push leaf package staging projects through. If the average leaf staging project lives for 1 hour, then that's 240 SR / SR batches per day that could be treated this way..

Scheduling, building takes some time, there are other stuff, overall I think that we don't need staging for everything (although it might be nice). -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Dominique Leuenberger a.k.a. Dimstar - 15:39 3.12.13 wrote:

Quoting Michal Hrusecky :

...

Greg Freemyer - 9:34 3.12.13 wrote:

...

You write in SR of the lower package (the one that the other one depends on) please create Staging for this and in the second one that it depends on the first). AFAIK currently OBS doesn't handle dependencies between SRs.

AFAIK, OBS doesn't handle it 'natively' (well no webui or osc direct).

but there is a plugin: osc group

this is the 'magic' currently used on openSUSE:Factory by the release team.

Well, I know about it, but it requires some special rights and doesn't handle dependencies as much as allows grouping of requests, which is possible workaround around missing dependencies handling :-) -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org