[opensuse-buildservice] inject local rpm data for buildinfo calculation
Hi, I wrote a small patch to inject rpm/deb/whatever data to the buildinfo calculation. This means one can collect the rpm/deb provides and requires of local packages and send them to the obs which uses them to calculate the buildinfo for local builds. Usecase: * you submitted a new package to the obs: before you can build a local package which depends on this new package you have to wait until the obs has built it (now you could simply do sth. like "osc build ... -p /path/to/local/pkgs" and it'll directly start the local build). How it works: A client like osc collects the requires and provides of some local packages, stores these information in a xml file and finally creates a cpio archive which contains the xml + a spec/dsc file and sends it to the obs. (Note: I decided to store the data in a xml because I didn't want to introduce yet another metadata format - this results in less additional code) Example xml: <localdeps> <package name="foobar" version="3.0.1" release="0" arch="i586"> <provides>foobar = 3.0.1-0</provides> <requires>glibc</requires> <requires>libfoo-devel >= 2.0</requires> </package> ... </localdeps> Any feedback, corrections etc. are welcome:) If there are no objections I'll commit it within the next few days. Marcus
Am Montag, 23. Februar 2009 01:54:12 schrieb Marcus Hüwe:
Hi,
I wrote a small patch to inject rpm/deb/whatever data to the buildinfo calculation. This means one can collect the rpm/deb provides and requires of local packages and send them to the obs which uses them to calculate the buildinfo for local builds.
cool stuff, the patch looks good to me as well. However, please wait for mls judgement :) bye adrian
Usecase: * you submitted a new package to the obs: before you can build a local package which depends on this new package you have to wait until the obs has built it (now you could simply do sth. like "osc build ... -p /path/to/local/pkgs" and it'll directly start the local build).
How it works: A client like osc collects the requires and provides of some local packages, stores these information in a xml file and finally creates a cpio archive which contains the xml + a spec/dsc file and sends it to the obs. (Note: I decided to store the data in a xml because I didn't want to introduce yet another metadata format - this results in less additional code) Example xml: <localdeps> <package name="foobar" version="3.0.1" release="0" arch="i586"> <provides>foobar = 3.0.1-0</provides> <requires>glibc</requires> <requires>libfoo-devel >= 2.0</requires> </package> ... </localdeps>
Any feedback, corrections etc. are welcome:) If there are no objections I'll commit it within the next few days.
Marcus
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Mon, Feb 23, 2009 at 01:54:12AM +0100, Marcus Hüwe wrote:
I wrote a small patch to inject rpm/deb/whatever data to the buildinfo calculation.
Marvellous... very very useful.
of local packages and send them to the obs which uses them to calculate the buildinfo for local builds.
Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Mon, Feb 23, 2009 at 01:54:12AM +0100, Marcus Hüwe wrote: > I wrote a small patch to inject rpm/deb/whatever data to the buildinfo > calculation. This means one can collect the rpm/deb provides and requires > of local packages and send them to the obs which uses them to calculate > the buildinfo for local builds. Heh, this is a feature that was actually already discussed with Peter but never reached the implementation phase. > Usecase: > * you submitted a new package to the obs: before you can build a local package > which depends on this new package you have to wait until the obs has built it > (now you could simply do sth. like "osc build ... -p /path/to/local/pkgs" > and it'll directly start the local build). > > How it works: > A client like osc collects the requires and provides of some local packages, > stores these information in a xml file and finally creates a cpio archive > which contains the xml + a spec/dsc file and sends it to the obs. > (Note: I decided to store the data in a xml because I didn't want to introduce > yet another metadata format - this results in less additional code) How about using Build.pm's internal format? > Example xml: >> > [...] @@ -1370,6 +1372,7 @@ if (!$cgi->{'internal'}) { for (@bdeps) { my $p = $dep2rpm{$_}->{'path'}; + next unless $p; push @bdepsp, "$reporoot/$dep2prp{$_}/$arch/:full/$p"; } @bdepsp = getbinarydata(@bdepsp); @@ -1381,6 +1384,7 @@ ($_->{'project'}, $_->{'repository'}) = split('/', $dep2prp{$_->{'name'}} , 2); } my $r = $bdepsp{$_->{'name'}}; + $r = $ldeps->{$_->{'name'}} if (defined($ldeps) && !defined($r)); if ($r) { $_->{'version'} = $r->{'version'}; $_->{'release'} = $r->{'release'} if exists $r->{'release'}; > [...] > + my %localdeps; > + # FIXME: is there a to check for the cpio magic? Of course there is, it must start with "070701". You can use this for autodetection, i.e. no need for the "cpio" switch. Have you already patched osc so that it sends the extra data? Cheers, Micha. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org> > ... >foobar = 3.0.1-0 >glibc >libfoo-devel >= 2.0 >
On 2009-02-25 14:59:38 +0100, Michael Schroeder wrote:
On Mon, Feb 23, 2009 at 01:54:12AM +0100, Marcus Hüwe wrote:
I wrote a small patch to inject rpm/deb/whatever data to the buildinfo calculation. This means one can collect the rpm/deb provides and requires of local packages and send them to the obs which uses them to calculate the buildinfo for local builds.
Heh, this is a feature that was actually already discussed with Peter but never reached the implementation phase.
Yes Peter told me the same:)
How about using Build.pm's internal format?
Indeed that's a good idea.
+ # FIXME: is there a to check for the cpio magic?
Of course there is, it must start with "070701". You can use this for autodetection, i.e. no need for the "cpio" switch.
The problem is that there's no easy way to check for the magic. We can't use the existing BSServer::read_data() method because it reads directly from the socket. So if we do something like: if (BSServer::read_data(6, 1) eq '070701') { BSServer::read_cpio(...) } the read_cpio() will complain because it doesn't find the magic anymore. The following should work in theory: BSServer::read_file($fn); # store cpio archive _or_ plain text file in a file # (it reads all the POSTed data) Now we can easily check if $fn starts with 070701. The only problem with this approach is that we would need to outsource parts from BSHTTP::cpio_receiver() into a more "generic" method (basically we only need to pass a ref to method which reads the data (either from a file or from a socket)). Or is there an easy way which I didn't found?:)
Have you already patched osc so that it sends the extra data?
No I didn't start with the osc implementation yet (but Peter already checked in a module which can be used to create a cpio archive). Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Wed, Feb 25, 2009 at 05:24:58PM +0100, Marcus Hüwe wrote:
Now we can easily check if $fn starts with 070701. The only problem with this approach is that we would need to outsource parts from BSHTTP::cpio_receiver() into a more "generic" method (basically we only need to pass a ref to method which reads the data (either from a file or from a socket)).
Or is there an easy way which I didn't found?:)
Hah, see my commit a couple of hours ago. You can now do: open(F, '<', $cpiofile) || die("$cpiofile: $!\n"); my $uploaded = BSHTTP::cpio_receiver(BSHTTP::fd2hdr(\*F), {'directory' => $dir}); close F; Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2009-02-25 17:28:09 +0100, Michael Schroeder wrote:
On Wed, Feb 25, 2009 at 05:24:58PM +0100, Marcus Hüwe wrote:
Now we can easily check if $fn starts with 070701. The only problem with this approach is that we would need to outsource parts from BSHTTP::cpio_receiver() into a more "generic" method (basically we only need to pass a ref to method which reads the data (either from a file or from a socket)).
Or is there an easy way which I didn't found?:)
Hah, see my commit a couple of hours ago. You can now do:
Ah I would have needed it before sunday in order to save some time:)
open(F, '<', $cpiofile) || die("$cpiofile: $!\n"); my $uploaded = BSHTTP::cpio_receiver(BSHTTP::fd2hdr(\*F), {'directory' => $dir}); close F;
Awesome! Thanks! I'll incorporate all your suggestions and post a new patch afterwards. Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2009-02-25 14:59:38 +0100, Michael Schroeder wrote:
How about using Build.pm's internal format?
I've tried that now but faced some "problems". If I understand it correctly the format which is used by Build::readdeps() looks like this: P:<id>: <list of provides separated by a whitespace> R:<id>: <list of requires separated by a whitespace> And <id> looks like <pkgname><some other info> - at least that's the impression I got after reading the code + looking at e.g. createyastdeps. So an example entry would look like this: P:foo-bar-gui-qt: foo-bar-gui-qt = 1.0 R:foo-bar-gui-qt: libfoo P:foo-bar-gui-gtk: foo-bar-gui-gtk = 1.0 R:foo-bar-gui-gtk: libbar These are valid package names but aren't parsed correctly by Build::readdeps() which is due to ... if ($s =~ s/^P:(.*):$/$1/) { my $pkgid = $s; $s =~ s/-[^-]+-[^-]+-[^-]+$//; $provides{$s} = \@ss; ... # the same for provides The problem is that I didn't get the idea of this substitute (apart from causing problems). Additionally what's the idea of the $pkgidp hashref which can be passed to Build::readdeps()? Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Feb 26, 2009 at 01:16:49PM +0100, Marcus Hüwe wrote:
the format which is used by Build::readdeps() looks like this: P:<id>: <list of provides separated by a whitespace> R:<id>: <list of requires separated by a whitespace> And <id> looks like <pkgname><some other info> - at least that's the impression I got after reading the code + looking at e.g. createyastdeps. So an example entry would look like this:
P:foo-bar-gui-qt: foo-bar-gui-qt = 1.0 R:foo-bar-gui-qt: libfoo P:foo-bar-gui-gtk: foo-bar-gui-gtk = 1.0 R:foo-bar-gui-gtk: libbar
These are valid package names but aren't parsed correctly by Build::readdeps() which is due to ... if ($s =~ s/^P:(.*):$/$1/) { my $pkgid = $s; $s =~ s/-[^-]+-[^-]+-[^-]+$//; $provides{$s} = \@ss; ... # the same for provides
Yep, Build::readdeps() is broken. It still is only able to parse the old autobuild format while the Build.pm code nowadays writes some slight different format. Luckily this part of readdeps() isn't used at all, thus we're free to fix it.
The problem is that I didn't get the idea of this substitute (apart from causing problems). Additionally what's the idea of the $pkgidp hashref which can be passed to Build::readdeps()?
It can be used to record the package->pkgid relation. Used in legacy autobuild code. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2009-02-26 14:24:02 +0100, Michael Schroeder wrote:
On Thu, Feb 26, 2009 at 01:16:49PM +0100, Marcus Hüwe wrote:
These are valid package names but aren't parsed correctly by Build::readdeps() which is due to ... if ($s =~ s/^P:(.*):$/$1/) { my $pkgid = $s; $s =~ s/-[^-]+-[^-]+-[^-]+$//; $provides{$s} = \@ss; ... # the same for provides
Yep, Build::readdeps() is broken. It still is only able to parse the old autobuild format while the Build.pm code nowadays writes some slight different format. Luckily this part of readdeps() isn't used at all, thus we're free to fix it.
Ok this sounds good. Yet another question: is the current format used somewhere else? Because otherwise it would be cool if it's possible to slightly change the format. At the moment it has the following structure: ... I:<pkgid>: <name>-<version>-<release> <buildtime> ... My proposal is to separate <name>, <version> and <release> with a whitespace instead of the dash because it would be easier to parse the name version and release of a package (I need this info in order to send back a "clean" buildinfo). Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Marcus Hüwe wrote:
On 2009-02-26 14:24:02 +0100, Michael Schroeder wrote:
Yep, Build::readdeps() is broken. It still is only able to parse the old autobuild format while the Build.pm code nowadays writes some slight different format. Luckily this part of readdeps() isn't used at all, thus we're free to fix it.
Ok this sounds good. Yet another question: is the current format used somewhere else?
Files with that format are created by create*deps scripts and parsed by expanddeps.
Because otherwise it would be cool if it's possible to slightly change the format. At the moment it has the following structure: ... I:<pkgid>: <name>-<version>-<release> <buildtime> ... My proposal is to separate <name>, <version> and <release> with a whitespace instead of the dash because it would be easier to parse the name version and release of a package (I need this info in order to send back a "clean" buildinfo).
That line is (optionally) used by init_buildsystem to speed up installation. See RPMIDFMT in there. It is possible to change the format but existing srcfiles.cache in build roots have to be rebuilt then. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Feb 26, 2009 at 08:43:35PM +0100, Marcus Hüwe wrote:
My proposal is to separate <name>, <version> and <release> with a whitespace instead of the dash because it would be easier to parse the name version and release of a package (I need this info in order to send back a "clean" buildinfo).
We could do that, but it really isn't that hard to split the version/release with the current version... Cheers, Micha. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2009-02-27 11:00:01 +0100, Michael Schroeder wrote:
On Thu, Feb 26, 2009 at 08:43:35PM +0100, Marcus Hüwe wrote:
My proposal is to separate <name>, <version> and <release> with a whitespace instead of the dash because it would be easier to parse the name version and release of a package (I need this info in order to send back a "clean" buildinfo).
We could do that, but it really isn't that hard to split the version/release with the current version...
Here's a new version of the patch: build/Build.pm: * rewrote some parts of readdeps(), it parses the current format which is used by build and also supports debian requires/provides (foo | bar (>> 3.1)...) * removed $strip_versions module variable * the former $pkgidp ref of readdeps() is now used to store package information (version, release, arch) (the self provides of a package is used to determine this information) backend/bs_repserver: * some minor changes with regard to the last patch * local packages are preferred Feedback, objections etc. are welcome:) Marcus
On 2009-03-02 02:51:41 +0100, Marcus Hüwe wrote:
Here's a new version of the patch:
Any news on this? I'm just asking because someone asked for such a feature in #opensuse-buildservice. Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (5)
-
Adrian Schröter -
Ludwig Nussel -
Marcus Hüwe -
Michael Schroeder -
Peter Poeml