Mailinglist Archive: opensuse-buildservice (269 mails)

< Previous Next >
Re: [opensuse-buildservice] download packages on demand (hackweek project)
  • From: Marcus Hüwe <suse-tux@xxxxxx>
  • Date: Mon, 8 Sep 2008 13:23:04 +0200
  • Message-id: <20080908112304.GA3222@xxxxxxxxxxxxxxxxxxx>
On 2008-09-03 09:03:09 +0200, Adrian Schröter wrote:
On Montag 01 September 2008 20:16:43 Marcus Hüwe wrote:
Hi,

during the last hackweek I worked on the obs so that it is able to resolve
dependencies using metadata and that it download certain packages on
demand.

What and Why:
If you install a local obs you have to setup the initial distributions. If
you want to build against a distribution like openSUSE which is already
part of the "official" obs you can use the remoteurl feature. In case your
preferred distribution isn't hosted on the "official" obs you have to setup
the base distributions on your own. The first step is that you have
download the complete package tree. IMHO it is really annyoing to download
_lots_ of GBs if you just want to test something. Another reason why I did
it is that I wanted to get some more knowledge about the backend:)
So during the hackweek I implemented something which just downloads the
required packages on demand (precondition: the backend needs access to the
internet).

cool stuff. We may can deliver our packages of OBS with some preconfigured
projects using this feature. It makes sense esp. for frequently changing
repos. Do you have an "is up 2 date" check inside ?

Hmm I think the best would be to write a small script which checks if the
metadata have changed and if so it'll "ping" the scheduler to rescan
the changed project. This script can be executed via a cronjob every
n minutes. Or would you prefer a different implementation?

I would like to see support for rp-md repos before of course ;)
(esp. for Packman)

I just added rpmmd support (see attached patches). One remark about the
Rpmmd.pm module: it isn't possible to use the XML/Structured.pm module to parse
the primary.xml file because loading a ~56M file as a tree into memory will
take a _very_ long time. That's why I'm using a stream based solution which
will consume less memory.

Any remarks, comments etc. are appreciated:)


Marcus
--- meta.diff.old 2008-09-08 13:02:14.000000000 +0200
+++ meta.diff 2008-09-08 13:02:44.000000000 +0200
@@ -1,17 +1,19 @@
-Index: build/Meta.pm
+Index: Meta.pm
===================================================================
---- build/Meta.pm (Revision 0)
-+++ build/Meta.pm (Revision 0)
-@@ -0,0 +1,12 @@
+--- Meta.pm (Revision 0)
++++ Meta.pm (Revision 0)
+@@ -0,0 +1,14 @@
+package Build::Meta;
+
+use strict;
+use warnings;
++use Build::Meta::Rpmmd;
+use Build::Meta::Deb;
+
+sub parse {
-+ my ($fn, $type) = @_;
-+ return Build::Meta::Deb::parse($fn) if $type eq 'deb';
++ my ($fn, $type, $opts) = @_;
++ return Build::Meta::Deb::parse($fn, $opts) if $type eq 'deb';
++ return Build::Meta::Rpmmd::parse($fn, $opts) if $type eq 'rpmmd';
+}
+
+1;
Index: build/Meta/Rpmmd.pm
===================================================================
--- build/Meta/Rpmmd.pm (Revision 0)
+++ build/Meta/Rpmmd.pm (Revision 0)
@@ -0,0 +1,106 @@
+package Build::Meta::Rpmmd;
+use strict;
+use warnings;
+use XML::Parser;
+
+sub parse {
+ my ($fn, $opts) = @_;
+ my $h = rpmmdhandler->new(@{$opts->{'arch'}});
+ my $p = XML::Parser->new(Handlers => {
+ Start => sub { return $h->start_handler(@_); },
+ End => sub { return $h->end_handler(@_); },
+ Char => sub { return $h->char_handler(@_); },
+ }, ErrorContext => 2);
+ eval {
+ $p->parsefile($fn);
+ };
+ die("parse: $@") if $@;
+ return $h->getrepodata();
+}
+
+1;
+
+package rpmmdhandler;
+use strict;
+use warnings;
+
+sub new {
+ my ($class, @arch) = @_;
+ my $self = {};
+ $self->{'repodata'} = {};
+ $self->{'pack'} = {};
+ $self->{'arch'} = [ @arch ]; # XXX: are there cases where we want to mix
i586 and i686?
+ $self->{'reqprov'} = ();
+ $self->{'elems'} = [ qw(name arch version location size rpm:sourcerpm
rpm:provides rpm:requires rpm:entry) ];
+ return bless($self, $class);
+}
+
+sub addversrel {
+ my ($self, $attrs) = @_;
+ $self->{'pack'}->{'version'} = $attrs->{'ver'};
+ $self->{'pack'}->{'release'} = $attrs->{'rel'};
+}
+
+sub addreqprov {
+ my ($self, $attrs) = @_;
+ my %flags = ( 'EQ' => '=', 'LE' => '<=', 'GE' => '>=', 'LT' => '<', 'GT' =>
'>' );
+ my $name = $attrs->{'name'};
+ unless ($name =~ /^(rpmlib\(|\/)/) {
+ $name .= exists $attrs->{'flags'} ? " $flags{$attrs->{'flags'}} " : "";
+ $name .= exists $attrs->{'ver'} ? $attrs->{'ver'} : "";
+ $name .= exists $attrs->{'rel'} ? "-$attrs->{'rel'}" : "";
+ push @{$self->{'reqprov'}}, $name;
+ }
+}
+
+sub addlocation {
+ my ($self, $attrs) = @_;
+ $self->{'pack'}->{'path'} = $attrs->{'href'};
+}
+
+sub addsize {
+ my ($self, $attrs) = @_;
+ $self->{'pack'}->{'id'} = "-1/$attrs->{'package'}/-1"; # XXX: the <time />
tag provides time etc. but do we really need it?
+}
+
+sub getrepodata {
+ my ($self) = @_;
+ return $self->{'repodata'};
+}
+
+# XML::Parser handlers
+
+sub start_handler {
+ my ($self, $e, $name, %attrs) = @_;
+ $self->{'pack'}->{'hdrmd5'} = "0" if $name eq 'package';
+ return unless grep { $name eq $_ } qw(version location size rpm:entry);
+ $self->addversrel(\%attrs) if $name eq 'version';
+ $self->addreqprov(\%attrs) if $name eq 'rpm:entry';
+ $self->addlocation(\%attrs) if $name eq 'location';
+ $self->addsize(\%attrs) if $name eq 'size';
+}
+
+sub end_handler {
+ my ($self, $e, $name) = @_;
+ if ($name =~ /rpm:(provides|requires)/) {
+ $name =~ s/rpm://;
+ $self->{'pack'}->{$name} = $self->{'reqprov'};
+ $self->{'reqprov'} = ();
+ }
+ $self->{'repodata'}->{$self->{'pack'}->{'name'}} = $self->{'pack'} if $name
eq 'package' && grep { $self->{'pack'}->{'arch'} eq $_ } @{$self->{'arch'}};
+ $self->{'pack'} = {} if $name eq 'package';
+}
+
+sub char_handler {
+ my ($self, $e, $text) = @_;
+ return unless grep { $e->{'Context'}[-1] eq $_ } @{$self->{'elems'}};
+ my $tag = $e->{'Context'}[-1];
+ if ($tag eq 'rpm:sourcerpm') {
+ $tag = 'source';
+ # stolen from Build/Rpm.pm
+ $text =~ s/-[^-]*-[^-]*\.[^\.]*\.rpm//;
+ }
+ $self->{'pack'}->{$tag} = $text;
+}
+
+1;
< Previous Next >