[opensuse-buildservice] Build a whole project with osc
Hi, As requested by darix, I'm emailing oS-buildservice to ask for a new osc feature - capability to build a whole project (eg. KDE:KDE4:UNSTABLE:Desktop) with one single command. Currently there is only possible to do things like $ osc build PLATFORM ARCH BUILD_DESCR Which restricts builds to a given PRJCT:PGK so if one wants to build a whole project the following won't work: $ osc build PLATFORM ARCH PROJECT With my osc feature request something like it would be possible. For what I have understood, darix pointed out that to do that some API changes/exposing would be needed or so, which seems to concern him a little. - Build service/osc team, what do you thing about this request? Do you thing it's possible to implement? Any limitations? - Users, please let developers know if you also like this idea! The discussion has just began...! ;-) -- Regards, Carlos Goncalves --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hello,
Which restricts builds to a given PRJCT:PGK so if one wants to build a whole project the following won't work: $ osc build PLATFORM ARCH PROJECT
With my osc feature request something like it would be possible.
For what I have understood, darix pointed out that to do that some API changes/exposing would be needed or so, which seems to concern him a little.
- Build service/osc team, what do you thing about this request? Do you thing it's possible to implement? Any limitations? - Users, please let developers know if you also like this idea!
Would be fine. And "osc build PROJECT" and "osc build ARCH PROJECT" as well. These would allow to make a complete test run before server upload. Ciao -- http://www.dstoecker.eu/ (PGP key available) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, May 09, 2008 at 08:15:25PM +0100, Carlos Goncalves wrote:
Hi,
As requested by darix, I'm emailing oS-buildservice to ask for a new osc feature - capability to build a whole project (eg. KDE:KDE4:UNSTABLE:Desktop) with one single command.
Currently there is only possible to do things like $ osc build PLATFORM ARCH BUILD_DESCR
Which restricts builds to a given PRJCT:PGK so if one wants to build a whole project the following won't work: $ osc build PLATFORM ARCH PROJECT
With my osc feature request something like it would be possible.
For what I have understood, darix pointed out that to do that some API changes/exposing would be needed or so, which seems to concern him a little.
- Build service/osc team, what do you thing about this request? Do you thing it's possible to implement? Any limitations? - Users, please let developers know if you also like this idea!
The discussion has just began...! ;-)
At the moment I wouldn't know how to implement this in osc. osc would need the information from the backend which determines the order of package builds. Otherwise, it sounds like a very useful idea. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On 2008-05-09 20:15:25 +0100, Carlos Goncalves wrote:
Hi,
As requested by darix, I'm emailing oS-buildservice to ask for a new osc feature - capability to build a whole project (eg. KDE:KDE4:UNSTABLE:Desktop) with one single command.
Currently there is only possible to do things like $ osc build PLATFORM ARCH BUILD_DESCR
Which restricts builds to a given PRJCT:PGK so if one wants to build a whole project the following won't work: $ osc build PLATFORM ARCH PROJECT
With my osc feature request something like it would be possible.
For what I have understood, darix pointed out that to do that some API changes/exposing would be needed or so, which seems to concern him a little.
Hmm I think we need one additional api call to retrieve the buildorder of a project. This new call could look like this: 'POST:/build/$project/$repository/$arch/_buildorder' Then we could upload all spec files to the backend, the backend parses all specs and calculates the buildorder. Calculating the buildorder isn't too hard but the real problem is how to transfer all the spec files to the backend? I attached a potential method to calculate the buildorder (it retrieves the spec files and other needed data in one big xml file) (note: this method isn't complete, it should just show that the whole thing shouldn't be too hard to implement). So once the backend generates a buildorder xml file we can start to tweak the clients to build a complete project. Any opinions?:)
- Build service/osc team, what do you thing about this request? Do you thing it's possible to implement? Any limitations? - Users, please let developers know if you also like this idea!
The discussion has just began...! ;-)
Marcus sub getbuildorder_post { my ($cgi, $projid, $repoid, $arch) = @_; # TODO: error checking, expand links, handle dsc's etc. my $fn = "$uploaddir/$$"; die("upload failed\n") unless BSServer::read_file($fn); # BSXML::specs dtd: (<data /> contains the complete spec file) # our $specs = [ # 'specs' => # 'project', # [[ 'package' => # 'name', # 'data' => (), # ]], # ]; my %deps; my %map; my %cycles; my @pacs; my $config = BSRPC::rpc("$BSConfig::srcserver/getconfig", undef, "project=$projid", "repository=$repoid"); my $bconf = Build::read_config($arch, [split("\n", $config)]); # parse the xml file which contains several specfiles and build # needed data structures for BSSort::sortpacks() my $specs = readxml($fn, $BSXML::specs, 1); if ($specs) { foreach (@{$specs->{'package'}}) { my $pac = $_->{'name'}; push @pacs, $pac; # Build::Rpm::parse should also parse "Provides:" my $spec = Build::Rpm::parse($bconf, [split('\n', $_->{'data'})] ); $deps{$pac} = $spec->{'deps'}; foreach (@{$spec->{'subpacks'}}) { $map{$_} = $pac; } } } # calculate buildorder of our project # moved sortpacks() from bs_sched into a new module my @ret = BSSort::sortpacks( \%deps, \%map, undef, undef, \%cycles, @pacs); my $ret; if (%cycles) { $ret->{'error'} = 'cycling deps'; } $ret->{'package'} = \@ret; # BSXML::buildorder dtd: # our $buildorder = [ # 'buildorder' => # 'error', # [], # [ 'package' ] # ]; return ($ret, $BSXML::buildorder); } --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Wed, May 21, 2008 at 05:49:43PM +0200, Marcus Hüwe wrote:
On 2008-05-09 20:15:25 +0100, Carlos Goncalves wrote:
Hi,
As requested by darix, I'm emailing oS-buildservice to ask for a new osc feature - capability to build a whole project (eg. KDE:KDE4:UNSTABLE:Desktop) with one single command.
Currently there is only possible to do things like $ osc build PLATFORM ARCH BUILD_DESCR
Which restricts builds to a given PRJCT:PGK so if one wants to build a whole project the following won't work: $ osc build PLATFORM ARCH PROJECT
With my osc feature request something like it would be possible.
For what I have understood, darix pointed out that to do that some API changes/exposing would be needed or so, which seems to concern him a little.
Hmm I think we need one additional api call to retrieve the buildorder of a project. This new call could look like this:
'POST:/build/$project/$repository/$arch/_buildorder'
Then we could upload all spec files to the backend, the backend parses all specs and calculates the buildorder. Calculating the buildorder isn't too hard but the real problem is how to transfer all the spec files to the backend?
There actually is kind of a plan how this bit could be solved. A long time ago, we planned to complete the local build support by sending (from osc to the backend) the requires and provides of locally preferred packages (so the api can take that into account). We would actually send the local spec, plus provides and requires of the other packages cpio'ed to the api. (I actually implemented the cpio part at the time) I'm not sure if this is already enough to calculate the build order, but if not it could probably achieved the same way.
I attached a potential method to calculate the buildorder (it retrieves the spec files and other needed data in one big xml file) (note: this method isn't complete, it should just show that the whole thing shouldn't be too hard to implement).
So once the backend generates a buildorder xml file we can start to tweak the clients to build a complete project.
Any opinions?:)
Yes, it would be quite interesting. Just for the record, if the build order is known, something like for i in pkg1 pkg2 pkg3; do osc build ..... -k /tmp/keepdir -p /tmp/keepdir done works around the lack of the functionality. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
Marcus Hüwe wrote:
Hi,
As requested by darix, I'm emailing oS-buildservice to ask for a new osc feature - capability to build a whole project (eg. KDE:KDE4:UNSTABLE:Desktop) with one single command. Hmm I think we need one additional api call to retrieve the buildorder of a
On 2008-05-09 20:15:25 +0100, Carlos Goncalves wrote: project. This new call could look like this:
'POST:/build/$project/$repository/$arch/_buildorder'
Then we could upload all spec files to the backend, the backend parses all specs and calculates the buildorder. Calculating the buildorder isn't too hard
Note that the build order might change as you build packages and new dependencies show up. It's probably not too hard but it isn't that easy either ;-). Basically osc would have to send the specfiles + provides/requires/failures of already built packages + maybe something else each time a build finishes. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Marcus Hüwe wrote:
Hi,
As requested by darix, I'm emailing oS-buildservice to ask for a new osc feature - capability to build a whole project (eg. KDE:KDE4:UNSTABLE:Desktop) with one single command. Hmm I think we need one additional api call to retrieve the buildorder of a
On 2008-05-09 20:15:25 +0100, Carlos Goncalves wrote: project. This new call could look like this:
'POST:/build/$project/$repository/$arch/_buildorder'
Then we could upload all spec files to the backend, the backend parses all specs and calculates the buildorder. Calculating the buildorder isn't too hard
Note that the build order might change as you build packages and new dependencies show up. It's probably not too hard but it isn't that easy either ;-). Hmm no. My plan was to send _all_ spec files (or just only BuildRequires and Provides (we also need to associate this information to a particular
On 2008-05-21 18:05:18 +0200, Michal Marek wrote: package)) to the backend. Imagine the following: - package "foo" requires "bar-devel" - package "bar" was added locally ("bar" also provides "bar-devel") Now if we send the spec files for package "foo" and for package "bar" to the server we can calculate the buildorder without any problem. We will only encounter a problem when we're trying to retrieve the buildinfo for package "foo" because we have to tell the backend somehow that the package "bar-devel" exists only locally.
Basically osc would have to send the specfiles + provides/requires/failures of already built packages + maybe something else each time a build finishes.
Marcus --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hi,
As requested by darix, I'm emailing oS-buildservice to ask for a new osc feature - capability to build a whole project (eg. KDE:KDE4:UNSTABLE:Desktop) with one single command. Hmm I think we need one additional api call to retrieve the buildorder of a
On 21.05.2008 at 19:23, Marcus Hüwe
wrote: On 2008-05-21 18:05:18 +0200, Michal Marek wrote: Marcus Hüwe wrote: On 2008-05-09 20:15:25 +0100, Carlos Goncalves wrote: project. This new call could look like this: 'POST:/build/$project/$repository/$arch/_buildorder'
Then we could upload all spec files to the backend, the backend parses all specs and calculates the buildorder. Calculating the buildorder isn't too hard
Note that the build order might change as you build packages and new dependencies show up. It's probably not too hard but it isn't that easy either ;-). Hmm no. My plan was to send _all_ spec files (or just only BuildRequires and Provides (we also need to associate this information to a particular package)) to the backend.
Imagine the following: - package "foo" requires "bar-devel" - package "bar" was added locally ("bar" also provides "bar-devel")
the provides of bar-devel is normally not detected by BS. If you check in whatever spec file that creates a bar-devel rpm, and another that Requires: bar-devel, the latter one will stay on Missing dependencies until bar has built a single time and it will start using this. Dominique --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2008-05-21 18:23:37 +0200, Dominique Leuenberger wrote:
Hi,
As requested by darix, I'm emailing oS-buildservice to ask for a new osc feature - capability to build a whole project (eg. KDE:KDE4:UNSTABLE:Desktop) with one single command. Hmm I think we need one additional api call to retrieve the buildorder of a
On 21.05.2008 at 19:23, Marcus Hüwe
wrote: On 2008-05-21 18:05:18 +0200, Michal Marek wrote: Marcus Hüwe wrote: On 2008-05-09 20:15:25 +0100, Carlos Goncalves wrote: project. This new call could look like this: 'POST:/build/$project/$repository/$arch/_buildorder'
Then we could upload all spec files to the backend, the backend parses all specs and calculates the buildorder. Calculating the buildorder isn't too hard
Note that the build order might change as you build packages and new dependencies show up. It's probably not too hard but it isn't that easy either ;-). Hmm no. My plan was to send _all_ spec files (or just only BuildRequires and Provides (we also need to associate this information to a particular package)) to the backend.
Imagine the following: - package "foo" requires "bar-devel" - package "bar" was added locally ("bar" also provides "bar-devel")
the provides of bar-devel is normally not detected by BS. If you check in whatever spec file that creates a bar-devel rpm, and another that Requires: bar-devel, the latter one will stay on Missing dependencies until bar has built a single time and it will start using this.
Yes basically you're right but it mainly depends on the implementation of the buildorder calculation. It is possible that the BS "knows" about subpackages so that they can be used to calculate the buildorder (without having a "missing dependencies" status). If you want to know how it exactly works have a look at the code which I attached to my first mail in this thread (the interesting thing is the %map hash which maps subpackages to the corresponding buildservice package). Marcus --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Marcus Hüwe wrote:
Note that the build order might change as you build packages and new dependencies show up. It's probably not too hard but it isn't that easy either ;-). Hmm no. My plan was to send _all_ spec files (or just only BuildRequires and Provides (we also need to associate this information to a particular
On 2008-05-21 18:05:18 +0200, Michal Marek wrote: package)) to the backend.
What I was trying to say was that you can't statically extract all dependency information from a specfile. Dependencies are also generated at build time. Michal --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Michal Marek wrote:
Marcus Hüwe wrote:
Note that the build order might change as you build packages and new dependencies show up. It's probably not too hard but it isn't that easy either ;-). Hmm no. My plan was to send _all_ spec files (or just only BuildRequires and Provides (we also need to associate this information to a particular
On 2008-05-21 18:05:18 +0200, Michal Marek wrote: package)) to the backend.
What I was trying to say was that you can't statically extract all dependency information from a specfile. Dependencies are also generated at build time.
To give an example: A.spec: BuildRequires: B # no Requires B.spec # no (Build)Requires, but %post -f post.sh C.spec # no (Build)Requires %files /usr/bin/* No locally built packages. Now osc sends the three specs to the backend, which sends back this buildorder: B, A, C. Osc builds B.spec, which results in a B.rpm with following %post scriptlet: #!/usr/bin/c hello world (/usr/bin/c being a part of the C package). Without asking the backend again, osc is lost, because B suddenly has unresolved dependencies. When building inside the bs, A would be set to "expansion error, nothing provides /usr/bin/c needed by B" and C would be built instead (actually C would be built alongside with B). A successful build of C would then resolve the expansion error in A. So osc needs to inform the bs about changes after each build. Michal --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2008-05-21 19:02:54 +0200, Michal Marek wrote:
Michal Marek wrote:
Marcus Hüwe wrote:
Note that the build order might change as you build packages and new dependencies show up. It's probably not too hard but it isn't that easy either ;-). Hmm no. My plan was to send _all_ spec files (or just only BuildRequires and Provides (we also need to associate this information to a particular
On 2008-05-21 18:05:18 +0200, Michal Marek wrote: package)) to the backend.
What I was trying to say was that you can't statically extract all dependency information from a specfile. Dependencies are also generated at build time.
(/usr/bin/c being a part of the C package). Without asking the backend again, osc is lost, because B suddenly has unresolved dependencies.
Ah now I see your point - thanks for this example.
When building inside the bs, A would be set to "expansion error, nothing provides /usr/bin/c needed by B" and C would be built instead (actually C would be built alongside with B). A successful build of C would then resolve the expansion error in A.
So osc needs to inform the bs about changes after each build.
Hmm yes - this was something I wanted to avoid but now I agree with you that we have to inform the BS after each build. Marcus --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (6)
-
Carlos Goncalves
-
Dirk Stoecker
-
Dominique Leuenberger
-
Marcus Hüwe
-
Michal Marek
-
Peter Poeml