[opensuse-packaging] Rebuilding in OBS
Hello, For the past months it's been really difficult to contribute to package updates since I only have time to help out on weekends. This boils down to commits going in to oS:F, and my pet project (GNOME:Factory) is basically always blocked waiting for oS:F to build. Trying to build a package locally, is not all that easy, as recently, osc tells me, more often than not, that it can't find a package on a mirror nor in OBS so the local build is aborted. Now, I understand that we want to rebuild packages etc, and I also understand that weekends are the most appropriate time to this. The way OBS rebuilds packages seems a little bit paranoid to me though and I would like to ask the question if we can do it better. So, this is the workflow (AFAIK): 1. I branch a package from G:F, it ends up in my home and basically the content of it is a _link (since I have not changed anything yet) 2. I update the package and commit (now, I have a project.diff) 3. I submit the package to G:F 4. The package is accepted into G:F (G:F will now contain _link and project.diff, while the package in my home is back to being a _link) 5. The package is submitted from G:F to oS:F (and lets say accepted into oS:F). G:F will now be _link, and the package in my home is _link. In oS:F, it's all the files this package contains of Now, what will happen is: * When I do 1, OBS wants to build the package in my home * In step 4 (when package have been accepted), OBS will want to build the package in both my home and in G:F * In step 5, OBS will want to build the package in oS:F, G:F and my home My thinking: In this scenario, my home, G:F and oS:F all use the same build dependencies, so when a package is a _link, can't OBS just copy the content from the linked package instead of having to rebuild it completely in all these different places? Cheers, Magnus -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Sat, 22 Aug 2009, Magnus Boman wrote:
For the past months it's been really difficult to contribute to package updates since I only have time to help out on weekends. This boils down to commits going in to oS:F, and my pet project (GNOME:Factory) is basically always blocked waiting for oS:F to build.
One way out of this dilemma is to have a repository not build against oS:F/standard/ (that is constantly changing), but only against oS:F/snapshot/ . The latter changes only at certain sync points (when the standard/ part has completely rebuilt once). If you do that you risk a breaking build when submitting to oS:F proper (because up until then you built against a slightly older version), but you at least can get work done. So, if there would be a shadow repo of G:F building against ../snapshot/ you could work against that shadow repo, until satisfied, and then submit to G:F proper (and eventually to oS:F then). Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Mon, 2009-08-24 at 12:21 +0200, Michael Matz wrote:
Hi,
On Sat, 22 Aug 2009, Magnus Boman wrote:
For the past months it's been really difficult to contribute to package updates since I only have time to help out on weekends. This boils down to commits going in to oS:F, and my pet project (GNOME:Factory) is basically always blocked waiting for oS:F to build.
One way out of this dilemma is to have a repository not build against oS:F/standard/ (that is constantly changing), but only against oS:F/snapshot/ . The latter changes only at certain sync points (when the standard/ part has completely rebuilt once). If you do that you risk a breaking build when submitting to oS:F proper (because up until then you built against a slightly older version), but you at least can get work done.
It doesn't really help here since I have to build against GNOME:Factory and not oS:F as such. Basically all packages in G:F are blocked once G:F have been submitted to oS:F.
So, if there would be a shadow repo of G:F building against ../snapshot/ you could work against that shadow repo, until satisfied, and then submit to G:F proper (and eventually to oS:F then).
If this doesn't get me banned from OBS, I suspect that we end up delaying oS:F rebuilds even more... :-) Cheers, Magnus -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hi, On Mon, 24 Aug 2009, Magnus Boman wrote:
One way out of this dilemma is to have a repository not build against oS:F/standard/ (that is constantly changing), but only against oS:F/snapshot/ . The latter changes only at certain sync points (when the standard/ part has completely rebuilt once). If you do that you risk a breaking build when submitting to oS:F proper (because up until then you built against a slightly older version), but you at least can get work done.
It doesn't really help here since I have to build against GNOME:Factory and not oS:F as such. Basically all packages in G:F are blocked once G:F have been submitted to oS:F.
That's why I have said that G:F (or a variant thereof) needs to be based on oS:F/snapshot/ and you would work against that variant.
So, if there would be a shadow repo of G:F building against ../snapshot/ you could work against that shadow repo, until satisfied, and then submit to G:F proper (and eventually to oS:F then).
If this doesn't get me banned from OBS, I suspect that we end up delaying oS:F rebuilds even more... :-)
Well, if that's what it takes to improve the scheduler ... If it works satisfyingly you could do away with the above work-arounds. But until then you should use the workaround to get work done IMO. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Montag 24 August 2009 schrieb Michael Matz:
Hi,
On Mon, 24 Aug 2009, Magnus Boman wrote:
One way out of this dilemma is to have a repository not build against oS:F/standard/ (that is constantly changing), but only against oS:F/snapshot/ . The latter changes only at certain sync points (when the standard/ part has completely rebuilt once). If you do that you risk a breaking build when submitting to oS:F proper (because up until then you built against a slightly older version), but you at least can get work done.
It doesn't really help here since I have to build against GNOME:Factory and not oS:F as such. Basically all packages in G:F are blocked once G:F have been submitted to oS:F.
That's why I have said that G:F (or a variant thereof) needs to be based on oS:F/snapshot/ and you would work against that variant.
Yes, this is _exactly_ what should be done. Building against snapshot should be actually the preferred solution for all but a very limited set of projects out there. This also spreads the load a bit because on weekends openSUSE:Factory/standard would rebuild without blocking anything else and over the week GNOME:Factory would build _once_ (but it does that anyway :). Forbidding or mass replacing standard with snapshot is a bit too much IMO, because some repos _need_ to be in sync e.g. with the kernel ABI of openSUSE:Factory, but maybe it's the right approach to mass replace and then change that couple back. And then there is question when to sync snapshot. Right now it's done (if I remember) before full rebuilds, but it's possible to couple it with the publishing of /factory/repo/oss (in theory, I have no clue about the code in question :) Greetings, Stephan -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le mardi 25 août 2009, à 17:32 +0200, Stephan Kulow a écrit :
Yes, this is _exactly_ what should be done. Building against snapshot should be actually the preferred solution for all but a very limited set of projects out there.
I'll change G:F to build against snapshot once we'll be ready to start a fun new "let's rebuild everything in G:F" cycle. Should be sometimes today or tomorrow.
And then there is question when to sync snapshot. Right now it's done (if I remember) before full rebuilds, but it's possible to couple it with the publishing of /factory/repo/oss (in theory, I have no clue about the code in question :)
Yeah, it'd be nice to do this change. This should avoid some potential issues where some ABI in standard and snapshots are different (in a library that is not correctly versioned, eg). Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le mercredi 26 août 2009, à 14:42 +0200, Vincent Untz a écrit :
Le mardi 25 août 2009, à 17:32 +0200, Stephan Kulow a écrit :
Yes, this is _exactly_ what should be done. Building against snapshot should be actually the preferred solution for all but a very limited set of projects out there.
I'll change G:F to build against snapshot once we'll be ready to start a fun new "let's rebuild everything in G:F" cycle. Should be sometimes today or tomorrow.
Done.
And then there is question when to sync snapshot. Right now it's done (if I remember) before full rebuilds, but it's possible to couple it with the publishing of /factory/repo/oss (in theory, I have no clue about the code in question :)
Yeah, it'd be nice to do this change. This should avoid some potential issues where some ABI in standard and snapshots are different (in a library that is not correctly versioned, eg).
How can I help make this happen? :-) Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Freitag 28 August 2009 schrieb Vincent Untz:
How can I help make this happen? :-)
Michael talked about adding it to the scheduler, right now the snapshot is done behind the scheduler and then he's signalled. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Freitag, 28. August 2009 10:44:22 schrieb Vincent Untz:
Le mercredi 26 août 2009, à 14:42 +0200, Vincent Untz a écrit :
Le mardi 25 août 2009, à 17:32 +0200, Stephan Kulow a écrit :
Yes, this is _exactly_ what should be done. Building against snapshot should be actually the preferred solution for all but a very limited set of projects out there.
I'll change G:F to build against snapshot once we'll be ready to start a fun new "let's rebuild everything in G:F" cycle. Should be sometimes today or tomorrow.
Done.
And then there is question when to sync snapshot. Right now it's done (if I remember) before full rebuilds, but it's possible to couple it with the publishing of /factory/repo/oss (in theory, I have no clue about the code in question :)
Yeah, it'd be nice to do this change. This should avoid some potential issues where some ABI in standard and snapshots are different (in a library that is not correctly versioned, eg).
How can I help make this happen? :-)
I am currently thinking about some concept how to do this every repo, without the need to create another one like snapshot. However, this will end up in more package builds, when we have it. So we will be less often in "blocked" but way more in "scheduled" due to the increased job numbers :/ So I am not that sure that this will be helpfull at all. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Dienstag 25 August 2009 schrieb Stephan Kulow:
Yes, this is _exactly_ what should be done. Building against snapshot should be actually the preferred solution for all but a very limited set of projects out there.
I now also redirect queries for snapshot repo to the factory repo, so that you can build faster with osc build as it will have a much better chance to find the right rpms for snapshot on mirrors than for standard. All you have to do is to osc build snapshot - osc build will pick standard by default. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Samstag, 22. August 2009 01:59:55 schrieb Magnus Boman:
Hello,
For the past months it's been really difficult to contribute to package updates since I only have time to help out on weekends. This boils down to commits going in to oS:F, and my pet project (GNOME:Factory) is basically always blocked waiting for oS:F to build. Trying to build a package locally, is not all that easy, as recently, osc tells me, more often than not, that it can't find a package on a mirror nor in OBS so the local build is aborted.
Now, I understand that we want to rebuild packages etc, and I also understand that weekends are the most appropriate time to this.
The way OBS rebuilds packages seems a little bit paranoid to me though and I would like to ask the question if we can do it better.
So, this is the workflow (AFAIK):
1. I branch a package from G:F, it ends up in my home and basically the content of it is a _link (since I have not changed anything yet) 2. I update the package and commit (now, I have a project.diff) 3. I submit the package to G:F 4. The package is accepted into G:F (G:F will now contain _link and project.diff, while the package in my home is back to being a _link) 5. The package is submitted from G:F to oS:F (and lets say accepted into oS:F). G:F will now be _link, and the package in my home is _link. In oS:F, it's all the files this package contains of
Now, what will happen is:
* When I do 1, OBS wants to build the package in my home
* In step 4 (when package have been accepted), OBS will want to build the package in both my home and in G:F
* In step 5, OBS will want to build the package in oS:F, G:F and my home
My thinking:
In this scenario, my home, G:F and oS:F all use the same build dependencies, so when a package is a _link, can't OBS just copy the content from the linked package instead of having to rebuild it completely in all these different places?
No, because the project repos may have a different setup, regarding their project config (build against other repos or have flags in prjconf). Also the rpm headers and signature will be different for sure. What could get avoided is the rebuild when the sources get merged in linked packages. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le jeudi 27 août 2009, à 15:00 +0200, Adrian Schröter a écrit :
What could get avoided is the rebuild when the sources get merged in linked packages.
That'd be great, but I guess one potential issue is that, at least for oS:F, the spec file can be tweaked a bit when a request gets accepted. Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Freitag, 28. August 2009 14:27:17 schrieb Vincent Untz:
Le jeudi 27 août 2009, à 15:00 +0200, Adrian Schröter a écrit :
What could get avoided is the rebuild when the sources get merged in linked packages.
That'd be great, but I guess one potential issue is that, at least for oS:F, the spec file can be tweaked a bit when a request gets accepted.
this would never happen in openSUSE:Factory since there is never a source link. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Le vendredi 28 août 2009, à 14:55 +0200, Adrian Schröter a écrit :
Am Freitag, 28. August 2009 14:27:17 schrieb Vincent Untz:
Le jeudi 27 août 2009, à 15:00 +0200, Adrian Schröter a écrit :
What could get avoided is the rebuild when the sources get merged in linked packages.
That'd be great, but I guess one potential issue is that, at least for oS:F, the spec file can be tweaked a bit when a request gets accepted.
this would never happen in openSUSE:Factory since there is never a source link.
Sure, but it happens in all devel projects: + Magnus makes a change to gnome-main-menu in G:F + gnome-main-menu builds fine + a request is created from G:F/gnome-main-menu to oS:F/gnome-main-menu + the request is accepted + two possibilities: - the spec file gets tweaked: gnome-main-menu should be rebuilt in G:F anyway - the spec file doesn't change compared to what was in G:F: no need to rebuild Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Freitag, 28. August 2009 15:00:28 schrieb Vincent Untz:
Le vendredi 28 août 2009, à 14:55 +0200, Adrian Schröter a écrit :
Am Freitag, 28. August 2009 14:27:17 schrieb Vincent Untz:
Le jeudi 27 août 2009, à 15:00 +0200, Adrian Schröter a écrit :
What could get avoided is the rebuild when the sources get merged in linked packages.
That'd be great, but I guess one potential issue is that, at least for oS:F, the spec file can be tweaked a bit when a request gets accepted.
this would never happen in openSUSE:Factory since there is never a source link.
Sure, but it happens in all devel projects:
+ Magnus makes a change to gnome-main-menu in G:F + gnome-main-menu builds fine + a request is created from G:F/gnome-main-menu to oS:F/gnome-main-menu + the request is accepted + two possibilities: - the spec file gets tweaked: gnome-main-menu should be rebuilt in G:F anyway - the spec file doesn't change compared to what was in G:F: no need to rebuild
True, in that case there is no choice. However, I want anyway to get rid of the factory only special hacks. But we need to make some other stuff happen before. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
participants (5)
-
Adrian Schröter -
Magnus Boman -
Michael Matz -
Stephan Kulow -
Vincent Untz