Mailinglist Archive: opensuse-buildservice (269 mails)

< Previous Next >
[opensuse-buildservice] download packages on demand (hackweek project)
  • From: Marcus Hüwe <suse-tux@xxxxxx>
  • Date: Mon, 1 Sep 2008 20:16:43 +0200
  • Message-id: <20080901181643.GB3229@xxxxxxxxxxxxxxxxxxx>
Hi,

during the last hackweek I worked on the obs so that it is able to resolve
dependencies using metadata and that it download certain packages on demand.

What and Why:
If you install a local obs you have to setup the initial distributions. If you
want to build against a distribution like openSUSE which is already part of
the "official" obs you can use the remoteurl feature. In case your preferred
distribution isn't hosted on the "official" obs you have to setup the base
distributions on your own. The first step is that you have download the
complete package tree. IMHO it is really annyoing to download _lots_ of GBs if
you just want to test something. Another reason why I did it is that I wanted
to get some more knowledge about the backend:)
So during the hackweek I implemented something which just downloads the
required packages on demand (precondition: the backend needs access to the
internet).

How does it work (currently only debian metadata support is available):
As debian stores all relevant information for dependency resolving in one big
file (for instance
http://ftp.de.debian.org/debian/dists/sid/main/binary-i386/Packages.bz2)
the scheduler simply parses this file and builds up all the internal
datastructures based on this single file.
Now if a worker or an user makes a request for a certain package the repserver
checks if the file was already downloaded. If it doesn't exist the requested
package will be downloaded and forwarded to the requester. In case the
requested file already exists it'll be directly delivered (without downloading
it again).

Install instructions:
Using this feature is quite simple, first of all you have to apply the attached
patch.
Then you have to setup the base distribution (for instance debian sid):

mkdir -p /srv/obs/build/Debian:SID/standard/i586/:full
# download and install the Packages file
curl http://ftp.de.debian.org/debian/dists/sid/main/binary-i386/Packages.bz2 |
bunzip2 > /srv/obs/build/Debian:SID/standard/i586/:full/Packages
# fix the permissions
chown -R obsrun:obsrun /srv/obs/build

# install the project config etc. (we're reusing the Debian:Etch file)
osc meta prjconf Debian:Etch > /srv/obs/projects/Debian:SID.conf
# now open this file with $EDITOR and add the following line
Preinstall: debconf
# this is just a workaround...

osc meta prj Debian:Etch > /srv/obs/projects/Debian:SID.xml
# now open this file with $EDITOR and add the following line
<download baseurl="http://ftp.de.debian.org/debian"; metafile="Packages"
mtype="deb" />
# additionally replace all occurrences of "Etch" with "SID"

# fix the permissions
chown -R obsrun:obsrun /srv/obs/projects

# finally import the new base distribution into the database
RAILS_ENV="production_slave" ruby /srv/www/obs/frontend/script/import

That's it:)


By the way if the <download /> element is found in a project's metadata file
the obs will perform the things described above.
This new feature should have no impact on existing projects it's
just optional.


Feedback, bugreports etc. are welcome - I hope there's someone who finds it
useful:)


Marcus
Index: frontend/app/models/db_project.rb
===================================================================
--- frontend/app/models/db_project.rb (Revision 4848)
+++ frontend/app/models/db_project.rb (Arbeitskopie)
@@ -65,6 +65,21 @@
self.description = project.description.to_s
self.remoteurl = project.has_element?(:remoteurl) ?
project.remoteurl.to_s : nil
self.remoteproject = project.has_element?(:remoteproject) ?
project.remoteproject.to_s : nil
+ self.baseurl = nil
+ self.metafile = nil
+ self.mtype = nil
+ # XXX: normally this should be done during the xml validation...
+ if project.has_element?(:download)
+ if project.download.has_attribute?(:baseurl) &&
project.download.has_attribute?(:metafile) \
+ && project.download.has_attribute?(:mtype)
+ self.baseurl = project.download.baseurl.to_s
+ self.metafile = project.download.metafile.to_s
+ self.mtype = project.download.mtype.to_s
+ else
+ raise SaveError, "download element is incomplete"
+ end
+ end
+
self.save!

#--- update users ---#
@@ -311,6 +326,7 @@
project.description( description )
project.remoteurl(remoteurl) unless remoteurl.blank?
project.remoteproject(remoteproject) unless remoteproject.blank?
+ project.download( :baseurl => baseurl, :metafile => metafile, :mtype =>
mtype ) unless baseurl.blank? # this check is enough

each_user do |u|
project.person( :userid => u.login, :role => u.role_name )
Index: backend/BSXML.pm
===================================================================
--- backend/BSXML.pm (Revision 4817)
+++ backend/BSXML.pm (Arbeitskopie)
@@ -83,6 +83,11 @@
'role',
'userid',
]],
+ [ 'download' =>
+ 'baseurl',
+ 'metafile',
+ 'mtype',
+ ],
@flags,
[ $repo ],
];
@@ -245,6 +250,11 @@
'patternmd5',
'remoteurl',
'remoteproject',
+ [ 'download' =>
+ 'baseurl',
+ 'metafile',
+ 'mtype',
+ ],
@flags,
[ $repo ],
[[ 'package' =>
Index: backend/bs_repserver
===================================================================
--- backend/bs_repserver (Revision 4817)
+++ backend/bs_repserver (Arbeitskopie)
@@ -146,27 +146,53 @@
return \%bins;
}

+sub getfilename {
+ my $f = shift;
+ $f =~ s/.*\///;
+ return $f;
+}
+
+# naive implementation but it's sufficient for our needs
+sub getdirname {
+ my $d = shift;
+ my @dirname = split("/", $d);
+ return join("/", @dirname[0..$#dirname-1]);
+}
+
sub getbinarydata {
my @bins = @_;

my @res;
+ my %cache; # keeps the repodata caches (for each project)
+ my %fp; # map filename to package name
for my $bin (@bins) {
my $filename = $bin;
$filename =~ s/.*\///;
+ my $dirname = getdirname($bin);
+ my $data = undef;
local *F;
if (!open(F, '<', $bin)) {
- push @res, {'filename' => $filename, 'error' => "$bin: $!"};
- next;
+ if (-s "$dirname.cache") {
+ $cache{$dirname} = Storable::retrieve("$dirname.cache") unless exists
$cache{$dirname};
+ $fp{$dirname} = { map { getfilename($cache{$dirname}->{$_}->{'path'})
=> $_ } keys %{$cache{$dirname}} } unless exists $fp{$dirname};
+ $data = $cache{$dirname}->{$fp{$dirname}->{$filename}};
+ }
+ unless ($data) {
+ push @res, {'filename' => $filename, 'error' => "$bin: $!"};
+ next;
+ }
}
- my @s = stat(F);
- my $data = Build::query([$bin, \*F], 'evra' => 1);
- close(F);
- if (!$data) {
- push @res, {'filename' => $filename, 'error' => 'bad binary package'};
- next;
+ unless ($data) {
+ my @s = stat(F);
+ $data = Build::query([$bin, \*F], 'evra' => 1);
+ close(F);
+ if (!$data) {
+ push @res, {'filename' => $filename, 'error' => 'bad binary package'};
+ next;
+ }
+ $data->{'mtime'} = $s[9];
}
$data->{'filename'} = $filename;
- $data->{'mtime'} = $s[9];
delete $data->{'hdrmd5'};
delete $data->{'provides'};
delete $data->{'requires'};
@@ -207,8 +233,15 @@
my @send;
for my $n (@qbins) {
if ($bins->{$n} eq '_gone') {
- push @send, {'name' => $n, 'error' => 'not available'};
- next;
+ my $file = '';
+ if ($cgi->{'url'}) {
+ $file = fetchbinary("$reporoot/$prp/$arch/:full.cache",
"$reporoot/$prp/$arch/:full", $n, $cgi->{'url'});
+ ($bins->{$n} = $file) =~ s/$reporoot\/// if $file;
+ }
+ unless ($file) {
+ push @send, {'name' => $n, 'error' => 'not available'};
+ next;
+ }
}
my $r = "$reporoot/$bins->{$n}";
if ($r =~ /\.rpm$/) {
@@ -239,6 +272,7 @@
my @args;
push @args, "view=$view";
push @args, map {"binary=$_"} @{$cgi->{'binary'} || []};
+ push @args, "url=$cgi->{'url'}" if $cgi->{'url'};
BSHandoff::handoff($ajaxsocket,
"/build/$projid/$repoid/$arch/_repository", undef, @args);
exit(0);
}
@@ -273,9 +307,17 @@
next;
}
my $fd = gensym;
- if (!open($fd, '<', "$reporoot/$prp/$arch/:full/$c->{'path'}")) {
- push @files, {'name' => $bin, 'error' => 'not available'};
- next;
+ my $fname = getfilename($c->{'path'});
+ if (!open($fd, '<', "$reporoot/$prp/$arch/:full/$fname")) {
+ my $file = '';
+ if ($cgi->{'url'}) {
+ $file = fetchbinary("$reporoot/$prp/$arch/:full.cache",
"$reporoot/$prp/$arch/:full", $bin, $cgi->{'url'});
+ open($fd, '<', "$file") or warn("open: $!\n");
+ }
+ unless ($file) {
+ push @files, {'name' => $bin, 'error' => 'not available'};
+ next;
+ }
}
my $n = $bin;
$n .= $1 if $c->{'path'} =~ /(\.rpm|\.deb)$/;
@@ -457,6 +499,27 @@
return undef;
}

+sub fetchbinary {
+ my ($cache, $dstdir, $bin, $url) = @_;
+ $cache = Storable::retrieve("$cache");
+ return '' unless exists $cache->{$bin};
+ my $file = "$dstdir/" . getfilename($cache->{$bin}->{'path'});
+ print "fetching: $url/$cache->{$bin}->{'path'}\n";
+ eval {
+ #$data = BSRPC::rpc("$url/$cache->{$bin}->{'path'}");
+ BSRPC::rpc({
+ uri => "$url/$cache->{$bin}->{'path'}",
+ filename => $file,
+ receiver => \&BSHTTP::file_receiver
+ }, undef, ());
+ };
+ if ($@) {
+ print "failed: $@\n";
+ return '';
+ }
+ return $file;
+}
+
sub getbinary {
my ($cgi, $projid, $repoid, $arch, $packid, $bin) = @_;
if ($packid eq '_repository' && $bin eq '_buildconfig') {
@@ -467,6 +530,12 @@
my $path = "$reporoot/$projid/$repoid/$arch/$packid/$bin";
if ($packid eq ':full' && ! -f $path) {
my $bins = findbinaries("$projid/$repoid", $arch, $bin);
+ if ($bins->{$bin} eq '_gone' && exists $cgi->{'url'}) {
+ my $file = fetchbinary("$reporoot/$projid/$repoid/$arch/$packid.cache",
+ "$reporoot/$projid/$repoid/$arch/$packid", $bin,
$cgi->{'url'});
+ die("no such binary '$bin'\n") unless $file;
+ ($bins->{$bin} = $file) =~ s/$reporoot\///;
+ }
die("no such binary '$bin'\n") if $bins->{$bin} eq '_gone';
$path = $bin = "$reporoot/$bins->{$bin}";
$bin =~ s/.*\///;
@@ -1274,6 +1343,7 @@
for my $file (grep {!/^\./} ls($dir)) {
if ($file eq 'Build' && -d "$dir/$file") {
for my $file2 (grep {!/^\./} ls("$dir/Build")) {
+ next if -d "$dir/Build/$file2"; # do not send the new Meta stuff (just a
quick hack)
push @send, {'name' => "$file2", 'filename' => "$dir/Build/$file2"};
}
}
@@ -1659,13 +1729,13 @@
'/' => \&hello,

'POST:/build/$project/$repository/$arch/_repository match:' => \&postrepo,
- '/build/$project/$repository/$arch/$package:package_repository view:?
binary:filename*' => \&getbinarylist,
+ '/build/$project/$repository/$arch/$package:package_repository view:?
binary:filename* url:?' => \&getbinarylist,
'POST:/build/$project/$repository/$arch/$package_repository/_buildinfo add:*
internal:bool? deps:bool?' => \&getbuildinfo_post,
'/build/$project/$repository/$arch/$package/_buildinfo add:* internal:bool?
debug:bool? deps:bool?' => \&getbuildinfo,
'/build/$project/$repository/$arch/$package/_status' => \&getbuildstatus,
'/build/$project/$repository/$arch/$package/_history' => \&gethistory,
'/build/$project/$repository/$arch/$package/_log nostream:bool? start:num?
end:num? handoff:bool?' => \&getlogfile,
- '/build/$project/$repository/$arch/$package:package_repository/$filename' =>
\&getbinary,
+ '/build/$project/$repository/$arch/$package:package_repository/$filename
url:?' => \&getbinary,
'PUT:/build/$project/$repository/$arch/_repository/$filename
ignoreolder:bool? wipe:bool?' => \&putbinary,
'/search/published/binary/id $match:' => \&search_published_binary_id,
'/search/published/pattern/id $match:' => \&search_published_pattern_id,
@@ -1678,7 +1748,7 @@
'/getbuildcode' => \&getbuildcode,
'/getworkercode' => \&getworkercode,
'/putjob $arch $job $jobid:md5 $code:?' => \&putjob,
- '/getbinaries $project $repository $arch binaries: nometa:bool?' =>
\&getbinaries,
+ '/getbinaries $project $repository $arch binaries: nometa:bool? url:?' =>
\&getbinaries,
'/getbinaryversions $project $repository $arch binaries:' =>
\&getbinaryversions,

# published files
@@ -1701,7 +1771,7 @@
'/' => \&hello,
'/ajaxstatus' => \&getajaxstatus,
'/build/$project/$repository/$arch/$package/_log nostream:bool? start:num?
end:num?' => \&getlogfile,
- '/build/$project/$repository/$arch/$package:package_repository view:?
binary:filename*' => \&getbinarylist,
+ '/build/$project/$repository/$arch/$package:package_repository view:?
binary:filename* url:?' => \&getbinarylist,
'/_result $prpa+ oldstate:md5? package* code:* withbinarylist:bool?' =>
\&getresult,
];

Index: backend/bs_sched
===================================================================
--- backend/bs_sched (Revision 4817)
+++ backend/bs_sched (Arbeitskopie)
@@ -45,6 +45,7 @@
use BSBuild;
use Build;
use BSDB;
+use Build::Meta;

use strict;

@@ -281,6 +282,8 @@
return $repobins;
}

+my $projpacks; # global project/package data
+
sub findbins {
my ($prp) = @_;
local *D;
@@ -313,6 +316,13 @@
if (!opendir(D, $dir)) {
return findbins_remote($prp);
}
+ my @prp = split("/", $prp);
+ my $projid = $prp[0];
+ if (exists $projpacks->{$projid}->{'download'}) {
+ $repobins =
Build::Meta::parse("$dir/$projpacks->{$projid}->{'download'}->{'metafile'}",
"$projpacks->{$projid}->{'download'}->{'mtype'}");
+ $cnt = keys %$repobins;
+ }
+
my @bins = grep {/\.(?:rpm|deb|iso|raw|raw.install)$/} readdir(D);
closedir D;
if (!@bins && -s "$dir.subdirs") {
@@ -379,7 +389,6 @@



-my $projpacks; # global project/package data

# this is basically getconfig from the source server
# we do not need any macros, just the config
Index: backend/bs_worker
===================================================================
--- backend/bs_worker (Revision 4817)
+++ backend/bs_worker (Arbeitskopie)
@@ -540,6 +540,8 @@
$ddir = $dir;
# get only missing packages
push @args, "binaries=".join(',', @todo);
+ my $proj = XMLin($BSXML::proj,
BSRPC::rpc("$srcserver/project/$repo->{'project'}", undef, ()));
+ push @args, "url=$proj->{'download'}->{'baseurl'}" if exists
$proj->{'download'};
}
mkdir_p($ddir) || die("mkdir_p $ddir: $!\n");
push @args, "project=$repo->{'project'}";
Index: backend/bs_srcserver
===================================================================
--- backend/bs_srcserver (Revision 4817)
+++ backend/bs_srcserver (Arbeitskopie)
@@ -890,7 +890,7 @@
@packages = findpackages($projid) unless $cgi->{'nopackages'};
next if $repoids && !grep {$repoids->{$_->{'name'}}}
@{$proj->{'repository'} || []};
next if $packids && !grep {$packids->{$_}} @packages;
- for (qw{title description build publish debuginfo useforbuild remoteurl
remoteproject}) {
+ for (qw{title description build publish debuginfo useforbuild remoteurl
remoteproject download}) {
$jinfo->{$_} = $proj->{$_} if exists $proj->{$_};
}
# Check build flags in project meta data
@@ -2218,7 +2218,7 @@
sub getbinarylist {
my ($cgi, $projid, $repoid, $arch, $packid) = @_;

- checkprojrepoarch($projid, $repoid, $arch);
+ my $proj = checkprojrepoarch($projid, $repoid, $arch);
my $view = $cgi->{'view'};
my @args;
push @args, "view=$view" if $view;
@@ -2228,6 +2228,7 @@
BSHandoff::handoff($ajaxsocket, "/build/$projid/$repoid/$arch/$packid",
undef, @args);
exit(0);
}
+ push @args, "url=$proj->{'download'}->{'baseurl'}" if exists
$proj->{'download'};
my $param = {
'uri' => "$BSConfig::reposerver/build/$projid/$repoid/$arch/$packid",
'ignorestatus' => 1,
@@ -2261,7 +2262,7 @@
'ignorestatus' => 1,
'receiver' => \&BSServer::reply_receiver,
};
- BSWatcher::rpc($param);
+ BSWatcher::rpc($param, undef, exists $proj->{'download'} ?
"url=$proj->{'download'}->{'baseurl'}" : ());
return undef;
}

@@ -2975,7 +2976,8 @@
# work around api bug: only get 50 packages at a time
@fetch = splice(@fetch, 0, 50) if !$cpio && @fetch > 50;

- $cpio ||= BSWatcher::rpc($param, undef, "view=cpio", map {"binary=$_"}
@fetch);
+ my $proj = checkprojrepoarch($projid, $repoid, $arch, 1);
+ $cpio ||= BSWatcher::rpc($param, undef, "view=cpio", (exists
$proj->{'download'} ? "url=$proj->{'download'}->{'baseurl'}" : '', map
{"binary=$_"} @fetch));
return undef if $BSStdServer::isajax && !$cpio;
for my $f (@{$cpio || []}) {
my $bin = $f->{'name'};
Index: build/Meta/Deb.pm
===================================================================
--- build/Meta/Deb.pm (Revision 0)
+++ build/Meta/Deb.pm (Revision 0)
@@ -0,0 +1,68 @@
+package Build::Meta::Deb;
+use strict;
+use warnings;
+use Data::Dumper;
+
+my %tagmap = (
+ 'Package' => 'name',
+ 'Version' => 'version',
+ 'Provides' => 'provides',
+ 'Depends' => 'requires',
+ 'Pre-Depends' => 'requires',
+ 'Filename' => 'path',
+ 'Source' => 'source',
+ 'MD5sum' => 'hdrmd5', # XXX:
+ 'Architecture' => 'arch',
+);
+
+sub parse {
+ my $fn = shift;
+
+ my %packs = ();
+ my $cur = {};
+ open(F, '<', $fn) or die("open: $!\n");
+ while (<F>) {
+ chomp;
+ next unless
/^(Package|Version|Provides|Depends|Pre-Depends|Filename|MD5sum|Source|Architecture|Size):\s(.*)/;
+ my ($tag, $what) = ($1, $2);
+ if ($tag =~ /^[\w-]*Depends|Provides/) {
+ my @m = $what =~ /([^\s,]+)(\s[^,]*)?[\s,]*/g;
+ my @l = ();
+ while (@m) {
+ my ($pack, $vers) = splice(@m, 0, 2);
+ $pack .= $vers if defined $vers;
+ push @l, $pack;
+ }
+ # stolen from the Build/Deb.pm
+ s/\(([^\)]*)\)/$1/g for @l;
+ s/<</</g for @l;
+ s/>>/>/g for @l;
+
+ push @{$cur->{$tagmap{$tag}}}, @l;
+ next;
+ }
+ # Size is the last entry in a package section
+ if ($tag eq 'Size') {
+ $cur->{'id'} = "-1/$what/-1";
+ my $rel = exists $cur->{'release'} ? "-$cur->{'release'}" : '';
+ push @{$cur->{'provides'}}, "$cur->{'name'} = $cur->{'version'}$rel";
+ $cur->{'requires'} = [] unless exists $cur->{'requires'};
+ $cur->{'source'} = $cur->{'name'} unless exists $cur->{'source'};
+ $packs{$cur->{'name'}} = $cur;
+ $cur = {};
+ next;
+ }
+ $cur->{$tagmap{$tag}} = $what;
+ if ($tag eq 'Version') {
+ # stolen from Build/Deb.pm
+ if ($what =~ /^(.*)-(.*?)$/) {
+ $cur->{'version'} = $1;
+ $cur->{'release'} = $2;
+ }
+ }
+ }
+ close(F);
+ return \%packs;
+}
+
+1;
Index: build/Meta.pm
===================================================================
--- build/Meta.pm (Revision 0)
+++ build/Meta.pm (Revision 0)
@@ -0,0 +1,12 @@
+package Build::Meta;
+
+use strict;
+use warnings;
+use Build::Meta::Deb;
+
+sub parse {
+ my ($fn, $type) = @_;
+ return Build::Meta::Deb::parse($fn) if $type eq 'deb';
+}
+
+1;
< Previous Next >
Follow Ups