[opensuse-factory] One of the options for staging projects
Hi, we are having a nice torrent of ideas in this mailing list since coolo presented this proposal last Thursday: https://progress.opensuse.org/workflow/factory-proposal.html Some of them are related to staging projects, so I will try to expand further one of the options regarding what could be happening inside that dashed box in the diagram. The main goal of staging projects is preventing Factory from breaking so often by creating a "temporary and alternative Factory" where dangerous and conflicting submissions have to learn to live in peace and harmony before entering the real Factory. The original mail from coolo contains a great example with new automake. So let's look closer into all those boxes, arrows and decision boxes: 1) Developer sends SR#1 to Factory and review team decides it needs staging, so staging project gets created (let's assume the decision is made by the review team, even if this is currently being discussed in the mailing list). The staging project initially contains only the package that is being sent and rebuilds everything from Factory that depend on it. 2) Stuff gets broken in staging, the submitter that sent SR#1 is now responsible for the project and in order to get SR#1 in Factory has to make sure staging project gets fixed. 2.1) The submitter fixes easy things that got broken. 2.2) The submitter coordinates with people who maintain packages that got broken. That means several people working on several packages (very often from different devel projects) all together to fix the new staging project. The fixes can be done in the devel projects or directly in the staging project. 2.2.a) Fixes in the devel projects gets into the staging project just by sending a SR to factory, more details below. 2.2.b) Every time a fix is done directly in the staging project it has to be sent back to the corresponding devel project. 3) When everything is fixed, stuff is merged in Factory and the staging project is deleted. See details below. Let's elaborate a little bit how 2.2.a works: - From one of the involved devel projects, SR#2 is sent to Factory fixing issues in staging project. - SR#2 gets grouped with SR#1 and the corresponding package in the staging project is replaced with link to the version in SR#2. New submissions replace the old ones. According to the procedure explained above, this is how 3 works: - When everything is fixed, there are in fact no local changes in the staging project except for things corresponding to the state of SRs in group. - Group will get accepted (all SR at the same time) and staging project deleted (as it is now merged into Factory). A last remark about the whole process: - if a package is involved in a staging project, submit requests to Factory that doesn't contain fix for the staging project are declined. It might be few exceptions like really dangerous zero day security exploit or data eating monster, but certainly not ordinary version bumps. This is just a proposal on how staging projects could work. The main goal of this proposal is to partially move the responsibility of the integration process into the community of packagers so: - Staging project maintainer has to cooperate with package maintainer to get his SR in. - Package maintainers have motivation to fix their stuff, otherwise new versions don't get included. The main advantages if this approach are: - Everything has to go through package maintainer to make sure changes don't get lost, package maintainers knows about the problems and agrees with the solution - We move from one good state to another, accepting only changes together with fixes. What do you think? What do you like/dislike? Opinions? -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
"Michal" == Michal Hrusecky <mhrusecky@suse.cz> writes:
Michal> Hi, Michal> we are having a nice torrent of ideas in this mailing list since coolo Michal> presented this proposal last Thursday: Michal> https://progress.opensuse.org/workflow/factory-proposal.html Michal> Some of them are related to staging projects, so I will try to Michal> expand further one of the options regarding what could be Michal> happening inside that dashed box in the diagram. Thanks for the explanation, as the boxes in the proposal was not easy to follow with my eyes ;) Michal> 2.2) The submitter coordinates with people who maintain packages Michal> that got broken. That means several people working on several Michal> packages (very often from different devel projects) all together Michal> to fix the new staging project. The fixes can be done in the devel Michal> projects or directly in the staging project. I am not sure I understand how this one work. For example in one of the gcc staging phase, there was a list of broken packages sent to factory list. I am thinking we want to automate this, rather than sending a list of packages that is manually compiled and send to factory. So in this new approach how is it planned to coordinate effort to fix the failing packages ? Michal> This is just a proposal on how staging projects could work. The Michal> main goal of this proposal is to partially move the responsibility Michal> of the integration process into the community of packagers so: - Michal> Staging project maintainer has to cooperate with package Michal> maintainer to get his SR in. - Package maintainers have motivation Michal> to fix their stuff, otherwise new versions don't get included. What is the delegation of work if the package maintainer is not responding in a timely manner ? Michal> The main advantages if this approach are: Michal> - Everything has to go through package maintainer to make sure Michal> changes don't get lost, package maintainers knows about the problems Michal> and agrees with the solution Michal> - We move from one good state to another, accepting only changes Michal> together with fixes. Michal> What do you think? What do you like/dislike? Opinions? Overall sounds fine to me. Togan -- Life is endless possibilities -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Togan Muftuoglu - 20:26 2.12.13 wrote:
"Michal" == Michal Hrusecky <mhrusecky@suse.cz> writes:
Michal> Hi, Michal> we are having a nice torrent of ideas in this mailing list since coolo Michal> presented this proposal last Thursday: Michal> https://progress.opensuse.org/workflow/factory-proposal.html
Michal> Some of them are related to staging projects, so I will try to Michal> expand further one of the options regarding what could be Michal> happening inside that dashed box in the diagram.
Thanks for the explanation, as the boxes in the proposal was not easy to follow with my eyes ;)
Michal> 2.2) The submitter coordinates with people who maintain packages Michal> that got broken. That means several people working on several Michal> packages (very often from different devel projects) all together Michal> to fix the new staging project. The fixes can be done in the devel Michal> projects or directly in the staging project.
I am not sure I understand how this one work. For example in one of the gcc staging phase, there was a list of broken packages sent to factory list. I am thinking we want to automate this, rather than sending a list of packages that is manually compiled and send to factory.
In case of gcc new staging project will get created and we will wait a reaaaaallly long time as everything depends on it :-D Than we will have staging project where there are quite some packages failing. We can list only failing packages, but we can simply point people to the web monitor of this staging project.
So in this new approach how is it planned to coordinate effort to fix the failing packages ?
It is just an idea, one of many possibilities. In gcc case, we should probably call for project wide help, but gcc maintainer will probably know the most frequent cases and probably could do some mass submission to fix them.
Michal> This is just a proposal on how staging projects could work. The Michal> main goal of this proposal is to partially move the responsibility Michal> of the integration process into the community of packagers so: - Michal> Staging project maintainer has to cooperate with package Michal> maintainer to get his SR in. - Package maintainers have motivation Michal> to fix their stuff, otherwise new versions don't get included.
What is the delegation of work if the package maintainer is not responding in a timely manner ?
Well, we should probably detect and handle inactive maintainers in general, but that would be out of scope of staging projects. We should somehow handle packages that nobody fixes, packages with security bugs nobody cares about and similar. Currently there is a group called confusingly 'factory-maintainers' that acts as backup for maintainers, but in general, I would leave this open and we can address inactive contributors later in discussion more generally ;-)
Michal> The main advantages if this approach are: Michal> - Everything has to go through package maintainer to make sure Michal> changes don't get lost, package maintainers knows about the problems Michal> and agrees with the solution Michal> - We move from one good state to another, accepting only changes Michal> together with fixes.
Michal> What do you think? What do you like/dislike? Opinions?
Overall sounds fine to me.
Glad you like it :-) -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 02 December 2013 19:52:47 Michal Hrusecky wrote:
Hi,
we are having a nice torrent of ideas in this mailing list since coolo presented this proposal last Thursday: https://progress.opensuse.org/workflow/factory-proposal.html
Some of them are related to staging projects, so I will try to expand further one of the options regarding what could be happening inside that dashed box in the diagram.
The main goal of staging projects is preventing Factory from breaking so often by creating a "temporary and alternative Factory" where dangerous and conflicting submissions have to learn to live in peace and harmony before entering the real Factory. The original mail from coolo contains a great example with new automake.
So let's look closer into all those boxes, arrows and decision boxes:
1) Developer sends SR#1 to Factory and review team decides it needs staging, so staging project gets created (let's assume the decision is made by the review team, even if this is currently being discussed in the mailing list). The staging project initially contains only the package that is being sent and rebuilds everything from Factory that depend on it.
The art here is how to make that decisions. If it would be based on past experience, we would always recommend to stage glibc :-) On the other hand, there's udev, which doesn't break builds but peoples running systems. So staging udev wouldn't reveal anything until people actually test the staged udev. Though this is a counter-example, I like the general idea. The Factory maintainers and the review team could come up with a list of packages that are likely to cause issues. For those we could just demand staging, no matter what. We would publish / version the list somewhere and just add a check to factory-auto. Another category would be packages where the submitter demands staging. I expect this to happen far less frequently but we should emphasize that this option is available. Responsible submitters may want to use it if they bring major changes to the distro.
2) [snip]
This is just a proposal on how staging projects could work. The main goal of this proposal is to partially move the responsibility of the integration process into the community of packagers so:
- Staging project maintainer has to cooperate with package maintainer to get his SR in.
To whom does the SR belong here? Is it the staging prj maintainer that wants it or the packager? IMO its usually the latter wanting the SR to get in while the former wants to get rid of the staging project again.
- Package maintainers have motivation to fix their stuff, otherwise new versions don't get included. I think this was discussed elsewhere already, but not-so-core community members are often asking why they should fix their package if breakage came from elsewhere (e.g. new glibc vs. random game pkg). But it's
The main advantages if this approach are: - Everything has to go through package maintainer to make sure changes don't get lost, package maintainers knows about the problems and agrees with the solution - We move from one good state to another, accepting only changes together with fixes.
I think you are correct that we have to bring the packagers into the fix-it boat. But let's not be overambitious, this won't solve everything :-)
What do you think? What do you like/dislike? Opinions?
-- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
Sascha Peilicke - 10:17 3.12.13 wrote:
On Monday 02 December 2013 19:52:47 Michal Hrusecky wrote:
Hi,
we are having a nice torrent of ideas in this mailing list since coolo presented this proposal last Thursday: https://progress.opensuse.org/workflow/factory-proposal.html
Some of them are related to staging projects, so I will try to expand further one of the options regarding what could be happening inside that dashed box in the diagram.
The main goal of staging projects is preventing Factory from breaking so often by creating a "temporary and alternative Factory" where dangerous and conflicting submissions have to learn to live in peace and harmony before entering the real Factory. The original mail from coolo contains a great example with new automake.
So let's look closer into all those boxes, arrows and decision boxes:
1) Developer sends SR#1 to Factory and review team decides it needs staging, so staging project gets created (let's assume the decision is made by the review team, even if this is currently being discussed in the mailing list). The staging project initially contains only the package that is being sent and rebuilds everything from Factory that depend on it.
The art here is how to make that decisions. If it would be based on past experience, we would always recommend to stage glibc :-)
And probably rightfully. Although I would say that bugfix releases not changing API/ABI probably don't need to go to such an extent.
On the other hand, there's udev, which doesn't break builds but peoples running systems. So staging udev wouldn't reveal anything until people actually test the staged udev.
Well, apart from staging projects, there is need QA part in the proposed workflow ;-)
Though this is a counter-example, I like the general idea.
Glad to hear so.
The Factory maintainers and the review team could come up with a list of packages that are likely to cause issues. For those we could just demand staging, no matter what. We would publish / version the list somewhere and just add a check to factory-auto.
Well, I would make it a little softer as part of the review - if you see that there is just typo in man fixed, no need for staging, if all headers got renamed, we need staging for sure. These to being just extremes, but what I'm saying is that I would like to make it more fuzzy...
Another category would be packages where the submitter demands staging. I expect this to happen far less frequently but we should emphasize that this option is available. Responsible submitters may want to use it if they bring major changes to the distro.
Sure, packages should be able to ask for it as they should know much better how drastic are the changes.
2) [snip]
This is just a proposal on how staging projects could work. The main goal of this proposal is to partially move the responsibility of the integration process into the community of packagers so:
- Staging project maintainer has to cooperate with package maintainer to get his SR in.
To whom does the SR belong here? Is it the staging prj maintainer that wants it or the packager? IMO its usually the latter wanting the SR to get in while the former wants to get rid of the staging project again.
Actually both. It was meant that the guy maintaining staging project has to work together with other people to get rid of his staging project and get stuff in, but it works also other way around. Packager need to cooperate with staging project maintainer to get his new and cool version in.
- Package maintainers have motivation to fix their stuff, otherwise new versions don't get included. I think this was discussed elsewhere already, but not-so-core community members are often asking why they should fix their package if breakage came from elsewhere (e.g. new glibc vs. random game pkg). But it's
Well, it works other way around as well, why should glibc maintainer fix something that uses API that was deprecated long time ago. The important part is that they should work it out together, otherwise neither of them can in.
The main advantages if this approach are: - Everything has to go through package maintainer to make sure changes don't get lost, package maintainers knows about the problems and agrees with the solution - We move from one good state to another, accepting only changes together with fixes.
I think you are correct that we have to bring the packagers into the fix-it boat. But let's not be overambitious, this won't solve everything :-)
Well, this should solve the failing packages in Factory and make Factory buildable/rebuildable all the time. To make it stable and working, we need QA, that was in another box :-D -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tuesday 03 December 2013 10:41:59 Michal Hrusecky wrote:
Sascha Peilicke - 10:17 3.12.13 wrote:
On Monday 02 December 2013 19:52:47 Michal Hrusecky wrote:
Hi,
we are having a nice torrent of ideas in this mailing list since coolo
presented this proposal last Thursday: https://progress.opensuse.org/workflow/factory-proposal.html
Some of them are related to staging projects, so I will try to expand further one of the options regarding what could be happening inside that dashed box in the diagram.
The main goal of staging projects is preventing Factory from breaking so often by creating a "temporary and alternative Factory" where dangerous and conflicting submissions have to learn to live in peace and harmony before entering the real Factory. The original mail from coolo contains a great example with new automake.
So let's look closer into all those boxes, arrows and decision boxes:
1) Developer sends SR#1 to Factory and review team decides it needs
staging, so staging project gets created (let's assume the decision is made by the review team, even if this is currently being discussed in the mailing list). The staging project initially contains only the package that is being sent and rebuilds everything from Factory that depend on it.
The art here is how to make that decisions. If it would be based on past experience, we would always recommend to stage glibc :-)
And probably rightfully. Although I would say that bugfix releases not changing API/ABI probably don't need to go to such an extent.
On the other hand, there's udev, which doesn't break builds but peoples running systems. So staging udev wouldn't reveal anything until people actually test the staged udev.
Well, apart from staging projects, there is need QA part in the proposed workflow ;-)
Ah ok, I have to admit that this thread became TL;DR for me already. So who will do the QA part? Before we propose anything there we should ask our stable release maintenance guys how pending updates are tested (if at all).
Though this is a counter-example, I like the general idea.
Glad to hear so.
The Factory maintainers and the review team could come up with a list of packages that are likely to cause issues. For those we could just demand staging, no matter what. We would publish / version the list somewhere and just add a check to factory-auto.
Well, I would make it a little softer as part of the review - if you see that there is just typo in man fixed, no need for staging, if all headers got renamed, we need staging for sure. These to being just extremes, but what I'm saying is that I would like to make it more fuzzy...
Another category would be packages where the submitter demands staging. I expect this to happen far less frequently but we should emphasize that this option is available. Responsible submitters may want to use it if they bring major changes to the distro.
Sure, packages should be able to ask for it as they should know much better how drastic are the changes.
2) [snip]
This is just a proposal on how staging projects could work. The main goal of this proposal is to partially move the responsibility of the
integration process into the community of packagers so: - Staging project maintainer has to cooperate with package maintainer
to get his SR in.
To whom does the SR belong here? Is it the staging prj maintainer that wants it or the packager? IMO its usually the latter wanting the SR to get in while the former wants to get rid of the staging project again.
Actually both. It was meant that the guy maintaining staging project has to work together with other people to get rid of his staging project and get stuff in, but it works also other way around. Packager need to cooperate with staging project maintainer to get his new and cool version in.
- Package maintainers have motivation to fix their stuff, otherwise
new versions don't get included.
I think this was discussed elsewhere already, but not-so-core community members are often asking why they should fix their package if breakage came from elsewhere (e.g. new glibc vs. random game pkg). But it's
Well, it works other way around as well, why should glibc maintainer fix something that uses API that was deprecated long time ago. The important part is that they should work it out together, otherwise neither of them can in.
I always agree if someone says "people should talk to each other". But the in reality we are talking someone has to fix something after it broke. Usually that's the guys that are either paid for it or who are very concerned about such matters. Often these are the same people :-)
The main advantages if this approach are: - Everything has to go through package maintainer to make sure
changes don't get lost, package maintainers knows about the problems and agrees with the solution
- We move from one good state to another, accepting only changes
together with fixes.
I think you are correct that we have to bring the packagers into the fix-it boat. But let's not be overambitious, this won't solve everything :-)
Well, this should solve the failing packages in Factory and make Factory buildable/rebuildable all the time. To make it stable and working, we need QA, that was in another box :-D
What we can improve though is the notification part of it. While we have coolo's reminder mails about new breakage, that's sometimes still overlooked by people (like me). -- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
On 12/03/2013 11:52 AM, Sascha Peilicke wrote:
On Tuesday 03 December 2013 10:41:59 Michal Hrusecky wrote:
Sascha Peilicke - 10:17 3.12.13 wrote:
On the other hand, there's udev, which doesn't break builds but peoples running systems. So staging udev wouldn't reveal anything until people actually test the staged udev. Well, apart from staging projects, there is need QA part in the proposed workflow ;-) Ah ok, I have to admit that this thread became TL;DR for me already. So who will do the QA part?
In the proposal, that would be the "QA Team". With heavy usage of a revamped openQA.
Before we propose anything there we should ask our stable release maintenance guys how pending updates are tested (if at all).
[...]
I think you are correct that we have to bring the packagers into the fix-it boat. But let's not be overambitious, this won't solve everything :-) Well, this should solve the failing packages in Factory and make Factory buildable/rebuildable all the time. To make it stable and working, we need QA, that was in another box :-D What we can improve though is the notification part of it. While we have coolo's reminder mails about new breakage, that's sometimes still overlooked by people (like me).
We are working in a heavy redesign of notifications in OBS. Expect good news about it in the short term. Cheers -- Ancor González Sosa openSUSE Team at Suse Linux GmbH -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 12/03/2013 04:41 AM, Michal Hrusecky wrote:
Sascha Peilicke - 10:17 3.12.13 wrote: <snip>
2) [snip]
This is just a proposal on how staging projects could work. The main goal of this proposal is to partially move the responsibility of the integration process into the community of packagers so:
- Staging project maintainer has to cooperate with package maintainer to get his SR in.
To whom does the SR belong here? Is it the staging prj maintainer that wants it or the packager? IMO its usually the latter wanting the SR to get in while the former wants to get rid of the staging project again.
Actually both. It was meant that the guy maintaining staging project has to work together with other people to get rid of his staging project and get stuff in,
Correct. I agree that having basically one person doing this (coolo today) is not a good approach and we need to fix this problem. What I am not so confident about is that those touching the very core packages have the interest/knowledge/energy/time to chaperone a staging branch. If AJ pulls a new glibc from upstream and tons of stuff breaks in the staging project we are basically asking AJ to be coolo and run after all the package maintainers that now have broken stuff. From my point of view that doesn't really resolve the basic problem, it just shifts the problem onto the shoulders of someone else. That someone else most likely has less time to chase all the broken stuff than coolo. I am not advocating to stay with what we have, I just fail to see how the staging of updates improves the situation overall. Certainly coolo will do less chasing and that's a good thing, but will the distribution as a whole improve or are we more likely to get stuck with older versions because the new chaperones of the staging branches do not have the time/energy etc. to chase everything that happens to break?
but it works also other way around. Packager need to cooperate with staging project maintainer to get his new and cool version in.
- Package maintainers have motivation to fix their stuff, otherwise new versions don't get included. I think this was discussed elsewhere already, but not-so-core community members are often asking why they should fix their package if breakage came from elsewhere (e.g. new glibc vs. random game pkg). But it's
Well, it works other way around as well, why should glibc maintainer fix something that uses API that was deprecated long time ago. The important part is that they should work it out together, otherwise neither of them can in.
That's not quite correct. The broken package is already in Factory, using the deprecated API. Thus glibc is blocked by something that is "nominally" broken already. The developer that submitted the glibc changes has to chase the packagers with broken stuff, as mentioned above. But there is also a good chance that the packager with the broken stuff is not intimately familiar with the code and thus has no idea how to fix it in the first place. Yes, these are all problems that rest on coolo's shoulders today and that's not good. However moving them onto someone else' shoulders does not make the problem go away. I think what we are actually doing is discouraging changes for packages that have a large impact such as gcc, glibc and others. As I mentioned in another thread, we do not want to encourage "dump and run" but this appears to go toward the other extreme. Basically if one is not willing to chaperone a staging project better not send any things that are potentially disruptive. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Robert Schweikert - 12:59 3.12.13 wrote:
...
Correct. I agree that having basically one person doing this (coolo today) is not a good approach and we need to fix this problem. What I am not so confident about is that those touching the very core packages have the interest/knowledge/energy/time to chaperone a staging branch.
If AJ pulls a new glibc from upstream and tons of stuff breaks in the staging project we are basically asking AJ to be coolo and run after all the package maintainers that now have broken stuff. From my point of view that doesn't really resolve the basic problem, it just shifts the problem onto the shoulders of someone else. That someone else most likely has less time to chase all the broken stuff than coolo. I am not advocating to stay with what we have, I just fail to see how the staging of updates improves the situation overall. Certainly coolo will do less chasing and that's a good thing, but will the distribution as a whole improve or are we more likely to get stuck with older versions because the new chaperones of the staging branches do not have the time/energy etc. to chase everything that happens to break?
You missed the second part. No new version of the broken package will get in unless it fixes the staging project. So now both maintainers are stuck with old versions that work together. And it is in the interest of both of them to resolve the issues.
...
I think what we are actually doing is discouraging changes for packages that have a large impact such as gcc, glibc and others. As I mentioned in another thread, we do not want to encourage "dump and run" but this appears to go toward the other extreme. Basically if one is not willing to chaperone a staging project better not send any things that are potentially disruptive.
No, we are postponing the integration of changes till we are sure that integration will result in something that would work :-) -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 12/03/2013 01:21 PM, Michal Hrusecky wrote:
Robert Schweikert - 12:59 3.12.13 wrote:
...
Correct. I agree that having basically one person doing this (coolo today) is not a good approach and we need to fix this problem. What I am not so confident about is that those touching the very core packages have the interest/knowledge/energy/time to chaperone a staging branch.
If AJ pulls a new glibc from upstream and tons of stuff breaks in the staging project we are basically asking AJ to be coolo and run after all the package maintainers that now have broken stuff. From my point of view that doesn't really resolve the basic problem, it just shifts the problem onto the shoulders of someone else. That someone else most likely has less time to chase all the broken stuff than coolo. I am not advocating to stay with what we have, I just fail to see how the staging of updates improves the situation overall. Certainly coolo will do less chasing and that's a good thing, but will the distribution as a whole improve or are we more likely to get stuck with older versions because the new chaperones of the staging branches do not have the time/energy etc. to chase everything that happens to break?
You missed the second part. No new version of the broken package will get in unless it fixes the staging project.
Hmmm that would require that everyone that is a staging tree chaperone knows about all the broken packages in all staging projects. If the chaperone's do not poses this knowledge a new version of the broken package can enter factory through an unrelated staging branch.
So now both maintainers are stuck with old versions that work together. And it is in the interest of both of them to resolve the issues.
Nope, it is in the interest of the user that they resolve the issue, not necessarily in the interest of the packager.
...
I think what we are actually doing is discouraging changes for packages that have a large impact such as gcc, glibc and others. As I mentioned in another thread, we do not want to encourage "dump and run" but this appears to go toward the other extreme. Basically if one is not willing to chaperone a staging project better not send any things that are potentially disruptive.
No, we are postponing the integration of changes till we are sure that integration will result in something that would work :-)
True, but the process chosen to get there requires more stage tree chaperones, i.e. we need to clone coolo. Why do we believe that is going to work? Why do we think that everyone that maintains some low level package is ready, willing, and able to chaperone a staging project with potentially many broken things? But we do not really need to discuss this in a theoretical vacuum. @Richard are you going to chase all the package maintainers that have broken packages when your gcc update cannot get merged to factory and is stuck in a staging branch? I do not want to put words in Richard's mouth, but historically the answer to that question has been NO. coolo and others have done most of the chasing and fixing. Thus, having the staging branch addresses the "factory" is broken problem, but all the other issues, i.e. we only have a few people chasing and fixing things remain the same as they are today. Who is going to be willing, able, and resourceful enough to be a chaperone for a staging branch and has the necessary time to do the job well? Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU SUSE-IBM Software Integration Center LINUX Tech Lead Public Cloud Architect rjschwei@suse.com rschweik@ca.ibm.com 781-464-8147 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Robert Schweikert - 14:23 3.12.13 wrote:
On 12/03/2013 01:21 PM, Michal Hrusecky wrote:
Robert Schweikert - 12:59 3.12.13 wrote:
...
Correct. I agree that having basically one person doing this (coolo today) is not a good approach and we need to fix this problem. What I am not so confident about is that those touching the very core packages have the interest/knowledge/energy/time to chaperone a staging branch.
If AJ pulls a new glibc from upstream and tons of stuff breaks in the staging project we are basically asking AJ to be coolo and run after all the package maintainers that now have broken stuff. From my point of view that doesn't really resolve the basic problem, it just shifts the problem onto the shoulders of someone else. That someone else most likely has less time to chase all the broken stuff than coolo. I am not advocating to stay with what we have, I just fail to see how the staging of updates improves the situation overall. Certainly coolo will do less chasing and that's a good thing, but will the distribution as a whole improve or are we more likely to get stuck with older versions because the new chaperones of the staging branches do not have the time/energy etc. to chase everything that happens to break?
You missed the second part. No new version of the broken package will get in unless it fixes the staging project.
Hmmm that would require that everyone that is a staging tree chaperone knows about all the broken packages in all staging projects. If the chaperone's do not poses this knowledge a new version of the broken package can enter factory through an unrelated staging branch.
No, staging project maintainer needs has a staging project and OBS shows him, which packages are broken. He can ask OBS who are the maintainers and try to ping them and help them to fix their stuff. He doesn't need to know anything about other packages, he just needs to know something about his and how to communicate.
So now both maintainers are stuck with old versions that work together. And it is in the interest of both of them to resolve the issues.
Nope, it is in the interest of the user that they resolve the issue, not necessarily in the interest of the packager.
Well, only if packager doesn't care if he has a package in Factory... What it doesn't solve is inactive - presume dead - maintainers. This is general issue, that should be addressed in different discussion. -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Dec 2, 2013 at 1:52 PM, Michal Hrusecky <mhrusecky@suse.cz> wrote:
Hi,
we are having a nice torrent of ideas in this mailing list since coolo presented this proposal last Thursday: https://progress.opensuse.org/workflow/factory-proposal.html
Some of them are related to staging projects, so I will try to expand further one of the options regarding what could be happening inside that dashed box in the diagram.
The main goal of staging projects is preventing Factory from breaking so often by creating a "temporary and alternative Factory" where dangerous and conflicting submissions have to learn to live in peace and harmony before entering the real Factory. The original mail from coolo contains a great example with new automake.
So let's look closer into all those boxes, arrows and decision boxes:
1) Developer sends SR#1 to Factory and review team decides it needs staging, so staging project gets created (let's assume the decision is made by the review team, even if this is currently being discussed in the mailing list). The staging project initially contains only the package that is being sent and rebuilds everything from Factory that depend on it.
A couple questions/comments: === - how SR batches are handled? For instance, if I have even just 2 SRs that I know depend on each other but I want them staged because I don't know what else may depend on them. How do I tell OBS to create a staging tree and include both of my SRs. Obviously for KDE or Gnome, these might be very large SR batches. === - I like the idea of every SR (or small SR batch) going through a staging project. But I think we need 2 solutions: short term (hours) staging projects and long term (days/weeks) staging projects. -- long term staging projects - this has been the focus of discussion, so I will ignore it here. -- short term (hours) staging projects I propose every SR (or SR batch) not tagged for long term staging be evaluated for short term staging viability. The criteria could be "causes 5 or less package builds and all of the package builds take less than 1 hour each". Once a SR / SR batch is tagged for short term staging: --- create short term staging project - A BtrFS read/write snapshot of factory is created. This should only take a couple seconds --- Update the affected packages based on the SR / SR batch - I assume a few more seconds --- Run the build logic and build the updated packages and anything that depends on them - this is the time consumer, put hopefully this class of build can be given top priority. --- If no build errors, forward original SR / SR batch to factory reviewers and destroy BtrFS snapshot. --- if build errors, hoover up the build logs from all the builds and attach them to the SR, reject the SR, and destroy the BtrFS snapshot. If the above is totally automated, it seems the BtrFS snapshot may only live for a an hour or less typically. With that process it seems the overhead of creating a staging projects is minimal and even single package SRs of leaf packages could be run through a quick staging compile / dependency compile. === -- How many "leaf" SRs per day are there anyway? -- How many simultaneous BtrFS snapshots can exist and still get reasonable performance? --- If one assumes 10 simultaneous BtrFS snapshots is reasonable to maintain, then a queue could be established to push leaf package staging projects through. If the average leaf staging project lives for 1 hour, then that's 240 SR / SR batches per day that could be treated this way.. Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Greg Freemyer - 9:34 3.12.13 wrote:
=== - how SR batches are handled? For instance, if I have even just 2 SRs that I know depend on each other but I want them staged because I don't know what else may depend on them. How do I tell OBS to create a staging tree and include both of my SRs. Obviously for KDE or Gnome, these might be very large SR batches.
You write in SR of the lower package (the one that the other one depends on) please create Staging for this and in the second one that it depends on the first). AFAIK currently OBS doesn't handle dependencies between SRs.
=== - I like the idea of every SR (or small SR batch) going through a staging project. But I think we need 2 solutions: short term (hours) staging projects and long term (days/weeks) staging projects.
-- long term staging projects - this has been the focus of discussion, so I will ignore it here.
-- short term (hours) staging projects
I propose every SR (or SR batch) not tagged for long term staging be evaluated for short term staging viability. The criteria could be "causes 5 or less package builds and all of the package builds take less than 1 hour each".
Once a SR / SR batch is tagged for short term staging:
From here
--- create short term staging project - A BtrFS read/write snapshot of factory is created. This should only take a couple seconds
--- Update the affected packages based on the SR / SR batch - I assume a few more seconds
--- Run the build logic and build the updated packages and anything that depends on them - this is the time consumer, put hopefully this class of build can be given top priority.
to here we don't need special workflow as it would be handled by OBS
--- If no build errors, forward original SR / SR batch to factory reviewers and destroy BtrFS snapshot.
--- if build errors, hoover up the build logs from all the builds and attach them to the SR, reject the SR, and destroy the BtrFS snapshot.
If the above is totally automated, it seems the BtrFS snapshot may only live for a an hour or less typically.
With that process it seems the overhead of creating a staging projects is minimal and even single package SRs of leaf packages could be run through a quick staging compile / dependency compile. ===
Not a good idea to hack a new OBS from scratches living next to the official one. And I don't see what should be the difference between the short term and long term ones. Short ones are rejected right away without striving for solution in staging?
-- How many "leaf" SRs per day are there anyway?
No idea.
-- How many simultaneous BtrFS snapshots can exist and still get reasonable performance?
No idea and doesn't matter.
--- If one assumes 10 simultaneous BtrFS snapshots is reasonable to maintain, then a queue could be established to push leaf package staging projects through. If the average leaf staging project lives for 1 hour, then that's 240 SR / SR batches per day that could be treated this way..
Scheduling, building takes some time, there are other stuff, overall I think that we don't need staging for everything (although it might be nice). -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Quoting Michal Hrusecky <mhrusecky@suse.cz>:
Greg Freemyer - 9:34 3.12.13 wrote:
You write in SR of the lower package (the one that the other one depends on) please create Staging for this and in the second one that it depends on the first). AFAIK currently OBS doesn't handle dependencies between SRs.
AFAIK, OBS doesn't handle it 'natively' (well no webui or osc direct). but there is a plugin: osc group this is the 'magic' currently used on openSUSE:Factory by the release team. Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Dominique Leuenberger a.k.a. Dimstar - 15:39 3.12.13 wrote:
Quoting Michal Hrusecky <mhrusecky@suse.cz>:
Greg Freemyer - 9:34 3.12.13 wrote:
You write in SR of the lower package (the one that the other one depends on) please create Staging for this and in the second one that it depends on the first). AFAIK currently OBS doesn't handle dependencies between SRs.
AFAIK, OBS doesn't handle it 'natively' (well no webui or osc direct).
but there is a plugin: osc group
this is the 'magic' currently used on openSUSE:Factory by the release team.
Well, I know about it, but it requires some special rights and doesn't handle dependencies as much as allows grouping of requests, which is possible workaround around missing dependencies handling :-) -- Michal HRUSECKY SUSE LINUX, s.r.o. openSUSE Team Lihovarska 1060/12 PGP 0xFED656F6 19000 Praha 9 mhrusecky[at]suse.cz Czech Republic http://michal.hrusecky.net http://www.suse.cz -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (7)
-
Ancor Gonzalez Sosa -
Dominique Leuenberger a.k.a. Dimstar -
Greg Freemyer -
Michal Hrusecky -
Robert Schweikert -
Sascha Peilicke -
Togan Muftuoglu