[opensuse-buildservice] download packages on demand (hackweek project)
Hi, during the last hackweek I worked on the obs so that it is able to resolve dependencies using metadata and that it download certain packages on demand. What and Why: If you install a local obs you have to setup the initial distributions. If you want to build against a distribution like openSUSE which is already part of the "official" obs you can use the remoteurl feature. In case your preferred distribution isn't hosted on the "official" obs you have to setup the base distributions on your own. The first step is that you have download the complete package tree. IMHO it is really annyoing to download _lots_ of GBs if you just want to test something. Another reason why I did it is that I wanted to get some more knowledge about the backend:) So during the hackweek I implemented something which just downloads the required packages on demand (precondition: the backend needs access to the internet). How does it work (currently only debian metadata support is available): As debian stores all relevant information for dependency resolving in one big file (for instance http://ftp.de.debian.org/debian/dists/sid/main/binary-i386/Packages.bz2) the scheduler simply parses this file and builds up all the internal datastructures based on this single file. Now if a worker or an user makes a request for a certain package the repserver checks if the file was already downloaded. If it doesn't exist the requested package will be downloaded and forwarded to the requester. In case the requested file already exists it'll be directly delivered (without downloading it again). Install instructions: Using this feature is quite simple, first of all you have to apply the attached patch. Then you have to setup the base distribution (for instance debian sid): mkdir -p /srv/obs/build/Debian:SID/standard/i586/:full # download and install the Packages file curl http://ftp.de.debian.org/debian/dists/sid/main/binary-i386/Packages.bz2 | bunzip2 > /srv/obs/build/Debian:SID/standard/i586/:full/Packages # fix the permissions chown -R obsrun:obsrun /srv/obs/build # install the project config etc. (we're reusing the Debian:Etch file) osc meta prjconf Debian:Etch > /srv/obs/projects/Debian:SID.conf # now open this file with $EDITOR and add the following line Preinstall: debconf # this is just a workaround... osc meta prj Debian:Etch > /srv/obs/projects/Debian:SID.xml # now open this file with $EDITOR and add the following line <download baseurl="http://ftp.de.debian.org/debian" metafile="Packages" mtype="deb" /> # additionally replace all occurrences of "Etch" with "SID" # fix the permissions chown -R obsrun:obsrun /srv/obs/projects # finally import the new base distribution into the database RAILS_ENV="production_slave" ruby /srv/www/obs/frontend/script/import That's it:) By the way if the <download /> element is found in a project's metadata file the obs will perform the things described above. This new feature should have no impact on existing projects it's just optional. Feedback, bugreports etc. are welcome - I hope there's someone who finds it useful:) Marcus
On Montag 01 September 2008 20:16:43 Marcus Hüwe wrote:
Hi,
during the last hackweek I worked on the obs so that it is able to resolve dependencies using metadata and that it download certain packages on demand.
What and Why: If you install a local obs you have to setup the initial distributions. If you want to build against a distribution like openSUSE which is already part of the "official" obs you can use the remoteurl feature. In case your preferred distribution isn't hosted on the "official" obs you have to setup the base distributions on your own. The first step is that you have download the complete package tree. IMHO it is really annyoing to download _lots_ of GBs if you just want to test something. Another reason why I did it is that I wanted to get some more knowledge about the backend:) So during the hackweek I implemented something which just downloads the required packages on demand (precondition: the backend needs access to the internet).
cool stuff. We may can deliver our packages of OBS with some preconfigured projects using this feature. It makes sense esp. for frequently changing repos. Do you have an "is up 2 date" check inside ? I would like to see support for rp-md repos before of course ;) (esp. for Packman) bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2008-09-03 09:03:09 +0200, Adrian Schröter wrote:
On Montag 01 September 2008 20:16:43 Marcus Hüwe wrote:
Hi,
during the last hackweek I worked on the obs so that it is able to resolve dependencies using metadata and that it download certain packages on demand.
What and Why: If you install a local obs you have to setup the initial distributions. If you want to build against a distribution like openSUSE which is already part of the "official" obs you can use the remoteurl feature. In case your preferred distribution isn't hosted on the "official" obs you have to setup the base distributions on your own. The first step is that you have download the complete package tree. IMHO it is really annyoing to download _lots_ of GBs if you just want to test something. Another reason why I did it is that I wanted to get some more knowledge about the backend:) So during the hackweek I implemented something which just downloads the required packages on demand (precondition: the backend needs access to the internet).
cool stuff. We may can deliver our packages of OBS with some preconfigured projects using this feature. It makes sense esp. for frequently changing repos. Do you have an "is up 2 date" check inside ?
Currently there's no up 2 date check but this would be a nice feature.
I would like to see support for rp-md repos before of course ;) (esp. for Packman)
Ok I'll have a look at it:) Marcus --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2008-09-03 09:03:09 +0200, Adrian Schröter wrote:
On Montag 01 September 2008 20:16:43 Marcus Hüwe wrote:
Hi,
during the last hackweek I worked on the obs so that it is able to resolve dependencies using metadata and that it download certain packages on demand.
What and Why: If you install a local obs you have to setup the initial distributions. If you want to build against a distribution like openSUSE which is already part of the "official" obs you can use the remoteurl feature. In case your preferred distribution isn't hosted on the "official" obs you have to setup the base distributions on your own. The first step is that you have download the complete package tree. IMHO it is really annyoing to download _lots_ of GBs if you just want to test something. Another reason why I did it is that I wanted to get some more knowledge about the backend:) So during the hackweek I implemented something which just downloads the required packages on demand (precondition: the backend needs access to the internet).
cool stuff. We may can deliver our packages of OBS with some preconfigured projects using this feature. It makes sense esp. for frequently changing repos. Do you have an "is up 2 date" check inside ?
Hmm I think the best would be to write a small script which checks if the metadata have changed and if so it'll "ping" the scheduler to rescan the changed project. This script can be executed via a cronjob every n minutes. Or would you prefer a different implementation?
I would like to see support for rp-md repos before of course ;) (esp. for Packman)
I just added rpmmd support (see attached patches). One remark about the Rpmmd.pm module: it isn't possible to use the XML/Structured.pm module to parse the primary.xml file because loading a ~56M file as a tree into memory will take a _very_ long time. That's why I'm using a stream based solution which will consume less memory. Any remarks, comments etc. are appreciated:) Marcus
participants (2)
-
Adrian Schröter -
Marcus Hüwe