[opensuse-buildservice] How can I configure download on demand to use an RPM repository?
Hi, all!
I'm trying to configure an own RPM repository to use download on demand, but I don't know how to do it, because into the OBS wiki[1] there is only an example to use debian repository.
My repository is a classic repomd, like the Fedora repository.
Can anyone help me?
Regards.
[1] http://en.opensuse.org/openSUSE:Build_Service_Concept_CrossDevelopment
On 2010-09-28 12:03:58 +0200, Francisco Javier Cuadrado wrote:
I'm trying to configure an own RPM repository to use download on demand, but I don't know how to do it, because into the OBS wiki[1] there is only an example to use debian repository.
My repository is a classic repomd, like the Fedora repository.
First you need to download the "primary.xml.gz" file. Afterwards uncompress it and store it to /srv/obs/build/<project>/<repo>/<arch>/:full/primary.xml Next you need to edit the project meta for instance via osc: "osc meta prj <project> -e" and add the following:
<download baseurl="http://<url>/<path" metafile="primary.xml" mtype="rpmmd" arch="<arch>" />
Choose "<path>" so that the rpms are located in "http://<url>/<path>/<arch>/*" and "http://<url>/<path>/noarch/*".
Also make sure that dod is enabled in /usr/lib/obs/server/BSConfig.pm: our $enable_download_on_demand = 1;
Afterwards restart the scheduler.
Marcus
On Tue, Sep 28, 2010 at 06:24, Marcus Hüwe suse-tux@gmx.de wrote:
On 2010-09-28 12:03:58 +0200, Francisco Javier Cuadrado wrote:
I'm trying to configure an own RPM repository to use download on demand, but I don't know how to do it, because into the OBS wiki[1] there is only an example to use debian repository.
My repository is a classic repomd, like the Fedora repository.
First you need to download the "primary.xml.gz" file. Afterwards uncompress it and store it to /srv/obs/build/<project>/<repo>/<arch>/:full/primary.xml Next you need to edit the project meta for instance via osc: "osc meta prj <project> -e" and add the following:
<download baseurl="http://<url>/<path" metafile="primary.xml" mtype="rpmmd" arch="<arch>" />
Choose "<path>" so that the rpms are located in "http://<url>/<path>/<arch>/*" and "http://<url>/<path>/noarch/*".
Also make sure that dod is enabled in /usr/lib/obs/server/BSConfig.pm: our $enable_download_on_demand = 1;
Afterwards restart the scheduler.
This got my attention...
So would this let us get RPMs that are continually updated, like from a repo such as Fedora Updates?
On Tue, Sep 28, 2010 at 11:14:12AM -0500, Robert Xu wrote:
On Tue, Sep 28, 2010 at 06:24, Marcus Hüwe suse-tux@gmx.de wrote:
On 2010-09-28 12:03:58 +0200, Francisco Javier Cuadrado wrote:
Afterwards restart the scheduler.
This got my attention...
So would this let us get RPMs that are continually updated, like from a repo such as Fedora Updates?
Absolutely not. In fact, this seriously breaks and wont work at all due to some limitations of DoD. The repo you pull from has to be completely static. The build service never re-downloads the repo metadata, and caches a lot of info. Basically, some fairly substantial changes need to be done to DoD to make this work. I think this is on the feature schedule, but people arent currently working on it. I had one of our interns look at it last summer, but he didnt get to it before he left because of other priority things he was working. -- Michael
On 2010-09-28 11:31:38 -0500, Michael E Brown wrote:
On Tue, Sep 28, 2010 at 11:14:12AM -0500, Robert Xu wrote:
On Tue, Sep 28, 2010 at 06:24, Marcus Hüwe suse-tux@gmx.de wrote:
On 2010-09-28 12:03:58 +0200, Francisco Javier Cuadrado wrote:
Afterwards restart the scheduler.
This got my attention...
So would this let us get RPMs that are continually updated, like from a repo such as Fedora Updates?
Absolutely not. In fact, this seriously breaks and wont work at all due to some limitations of DoD. The repo you pull from has to be completely static. The build service never re-downloads the repo metadata, and caches a lot of info.
Can you summarize which features do you miss? If I understand you correctly the obs should fetch the required metadata on its own?
In order to support non static metadata (without "hacks" like removing :full* and triggering a scanrepo event) we would just need to compare the old metadata with the new metadata and remove some entries from :full.solv (and their correspondent binaries from the :full/ dir). I think this shouldn't be too hard to implement:)
Marcus
On Wed, Sep 29, 2010 at 04:42:56AM -0500, Marcus Hüwe wrote:
On 2010-09-28 11:31:38 -0500, Michael E Brown wrote:
On Tue, Sep 28, 2010 at 11:14:12AM -0500, Robert Xu wrote:
On Tue, Sep 28, 2010 at 06:24, Marcus Hüwe suse-tux@gmx.de wrote:
So would this let us get RPMs that are continually updated, like from a repo such as Fedora Updates?
Absolutely not. In fact, this seriously breaks and wont work at all due to some limitations of DoD. The repo you pull from has to be completely static. The build service never re-downloads the repo metadata, and caches a lot of info.
Can you summarize which features do you miss? If I understand you correctly the obs should fetch the required metadata on its own?
Correct. Today you have to manually download primary.xml.gz and uncompress even though the information exists where OBS could/should get this itself. It should download repomd.xml and parse that to get the url for primary.xml.gz, then download/uncompress.
In order to support non static metadata (without "hacks" like removing :full* and triggering a scanrepo event) we would just need to compare the old metadata with the new metadata and remove some entries from :full.solv (and their correspondent binaries from the :full/ dir). I think this shouldn't be too hard to implement:)
Well, if you put it that way, maybe. But you still need to teach OBS to download the metadata in the first place.
Pat Dignan wrote up a wiki page with the concept on it, which I cant lookup right now as I'm on my way to a meeting...
-- Michael
On 2010-09-29 13:57:11 -0500, Michael E Brown wrote:
On Wed, Sep 29, 2010 at 04:42:56AM -0500, Marcus Hüwe wrote:
On 2010-09-28 11:31:38 -0500, Michael E Brown wrote:
On Tue, Sep 28, 2010 at 11:14:12AM -0500, Robert Xu wrote:
On Tue, Sep 28, 2010 at 06:24, Marcus Hüwe suse-tux@gmx.de wrote:
So would this let us get RPMs that are continually updated, like from a repo such as Fedora Updates?
Absolutely not. In fact, this seriously breaks and wont work at all due to some limitations of DoD. The repo you pull from has to be completely static. The build service never re-downloads the repo metadata, and caches a lot of info.
Can you summarize which features do you miss? If I understand you correctly the obs should fetch the required metadata on its own?
Correct. Today you have to manually download primary.xml.gz and uncompress even though the information exists where OBS could/should get this itself. It should download repomd.xml and parse that to get the url for primary.xml.gz, then download/uncompress.
In order to support non static metadata (without "hacks" like removing :full* and triggering a scanrepo event) we would just need to compare the old metadata with the new metadata and remove some entries from :full.solv (and their correspondent binaries from the :full/ dir). I think this shouldn't be too hard to implement:)
Well, if you put it that way, maybe. But you still need to teach OBS to download the metadata in the first place.
I just wrote a small patch to support the "update dod meta" feature. If you set up an initial dod repo the scheduler downloads the required metadata automatically. Additionally you can create a so called "refreshdodmeta" event so that the scheduler updates the dod metadata and cleans up the :full dir (removes unneeded binaries).
TODO: * api support for the refreshdodmeta event (trivial) * add download support for susetags and debmd (probably also trivial) * support project meta changes (for instance update the dodmeta if the <download /> tag has changed, remove binaries if the <download /> tag is removed? etc.)
Feedback, objections to the patch/the current design are welcome:)
Marcus
On Sun, Oct 03, 2010 at 02:57:25PM -0500, Marcus Hüwe wrote:
I just wrote a small patch to support the "update dod meta" feature. If you set up an initial dod repo the scheduler downloads the required metadata automatically. Additionally you can create a so called "refreshdodmeta" event so that the scheduler updates the dod metadata and cleans up the :full dir (removes unneeded binaries).
TODO:
- api support for the refreshdodmeta event (trivial)
- add download support for susetags and debmd (probably also trivial)
- support project meta changes (for instance update the dodmeta if the <download /> tag has changed, remove binaries if the <download /> tag is removed? etc.)
Feedback, objections to the patch/the current design are welcome:)
This is pretty nice. I'm due to rebuild my local dev instance, so I'll see if I can fit some time in to try this patch. It sounds workable for what we do, ie. cron rsync of update repos. We can update the repos and add a step to run the refresh OBS too. -- Michael
Michael E Brown wrote:
On Sun, Oct 03, 2010 at 02:57:25PM -0500, Marcus Hüwe wrote:
I just wrote a small patch to support the "update dod meta" feature. If you set up an initial dod repo the scheduler downloads the required metadata automatically. Additionally you can create a so called "refreshdodmeta" event so that the scheduler updates the dod metadata and cleans up the :full dir (removes unneeded binaries).
TODO:
- api support for the refreshdodmeta event (trivial)
- add download support for susetags and debmd (probably also trivial)
- support project meta changes (for instance update the dodmeta if the <download /> tag has changed, remove binaries if the <download /> tag is removed? etc.)
Feedback, objections to the patch/the current design are welcome:)
This is pretty nice. I'm due to rebuild my local dev instance, so I'll see if I can fit some time in to try this patch. It sounds workable for what we do, ie. cron rsync of update repos. We can update the repos and add a step to run the refresh OBS too. -- Michael
I think you mean update support for susetags and debmd, basically DoD works for all 3 Meta Tag formats.
On 2010-10-04 10:29:35 -0500, Michael E Brown wrote:
On Sun, Oct 03, 2010 at 02:57:25PM -0500, Marcus Hüwe wrote:
I just wrote a small patch to support the "update dod meta" feature. If you set up an initial dod repo the scheduler downloads the required metadata automatically. Additionally you can create a so called "refreshdodmeta" event so that the scheduler updates the dod metadata and cleans up the :full dir (removes unneeded binaries).
TODO:
- api support for the refreshdodmeta event (trivial)
- add download support for susetags and debmd (probably also trivial)
- support project meta changes (for instance update the dodmeta if the <download /> tag has changed, remove binaries if the <download /> tag is removed? etc.)
Feedback, objections to the patch/the current design are welcome:)
This is pretty nice. I'm due to rebuild my local dev instance, so I'll see if I can fit some time in to try this patch. It sounds workable for what we do, ie. cron rsync of update repos. We can update the repos and add a step to run the refresh OBS too.
Ok. To update the metadata simply call something like: curl -X POST -d '' 'http://<srcserver>/build/<project>?cmd=refreshdodmeta&repository=<repo>&nopackages=1' (you can also specify the arch with &arch=<myarch>).
Marcus
On Mon, Oct 4, 2010 at 14:44, Marcus Hüwe suse-tux@gmx.de wrote:
On 2010-10-04 10:29:35 -0500, Michael E Brown wrote:
On Sun, Oct 03, 2010 at 02:57:25PM -0500, Marcus Hüwe wrote:
I just wrote a small patch to support the "update dod meta" feature. If you set up an initial dod repo the scheduler downloads the required metadata automatically. Additionally you can create a so called "refreshdodmeta" event so that the scheduler updates the dod metadata and cleans up the :full dir (removes unneeded binaries).
TODO: * api support for the refreshdodmeta event (trivial) * add download support for susetags and debmd (probably also trivial) * support project meta changes (for instance update the dodmeta if the <download /> tag has changed, remove binaries if the <download /> tag is removed? etc.)
Feedback, objections to the patch/the current design are welcome:)
This is pretty nice. I'm due to rebuild my local dev instance, so I'll see if I can fit some time in to try this patch. It sounds workable for what we do, ie. cron rsync of update repos. We can update the repos and add a step to run the refresh OBS too.
Ok. To update the metadata simply call something like: curl -X POST -d '' 'http://<srcserver>/build/<project>?cmd=refreshdodmeta&repository=<repo>&nopackages=1' (you can also specify the arch with &arch=<myarch>).
Can this get merged into 2.1?
Hi, again!
2010/9/28 Marcus Hüwe suse-tux@gmx.de:
On 2010-09-28 12:03:58 +0200, Francisco Javier Cuadrado wrote:
I'm trying to configure an own RPM repository to use download on demand, but I don't know how to do it, because into the OBS wiki[1] there is only an example to use debian repository.
My repository is a classic repomd, like the Fedora repository.
First you need to download the "primary.xml.gz" file. Afterwards uncompress it and store it to /srv/obs/build/<project>/<repo>/<arch>/:full/primary.xml Next you need to edit the project meta for instance via osc: "osc meta prj <project> -e" and add the following:
<download baseurl="http://<url>/<path" metafile="primary.xml" mtype="rpmmd" arch="<arch>" />
Choose "<path>" so that the rpms are located in "http://<url>/<path>/<arch>/*" and "http://<url>/<path>/noarch/*".
I've got a new doubt about DoD.
Can I have multiple primary files used into different download tags but for the same architecture? For example:
<download baseurl="http://repo-1/" metafile="primary-1.xml" arch="i586" ... /> <download baseurl="http://repo-2/" metafile="primary-2.xml" arch="i586" ... />
And into /srv/obs/build/<project>/i586/:full I will have primary-1.xml and primary-2.xml files.
So I can use multiple repositories for one architecture, in this case I know that there aren't duplicated packages, one repo doesn't contain any package which is into the other repo.
Also make sure that dod is enabled in /usr/lib/obs/server/BSConfig.pm: our $enable_download_on_demand = 1;
Afterwards restart the scheduler.
Marcus
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2010-10-04 13:30:15 +0200, Francisco Javier Cuadrado wrote:
2010/9/28 Marcus Hüwe suse-tux@gmx.de:
On 2010-09-28 12:03:58 +0200, Francisco Javier Cuadrado wrote:
I'm trying to configure an own RPM repository to use download on demand, but I don't know how to do it, because into the OBS wiki[1] there is only an example to use debian repository.
My repository is a classic repomd, like the Fedora repository.
First you need to download the "primary.xml.gz" file. Afterwards uncompress it and store it to /srv/obs/build/<project>/<repo>/<arch>/:full/primary.xml Next you need to edit the project meta for instance via osc: "osc meta prj <project> -e" and add the following:
<download baseurl="http://<url>/<path" metafile="primary.xml" mtype="rpmmd" arch="<arch>" />
Choose "<path>" so that the rpms are located in "http://<url>/<path>/<arch>/*" and "http://<url>/<path>/noarch/*".
I've got a new doubt about DoD.
Can I have multiple primary files used into different download tags but for the same architecture? For example:
<download baseurl="http://repo-1/" metafile="primary-1.xml" arch="i586" ... /> <download baseurl="http://repo-2/" metafile="primary-2.xml" arch="i586" ... />
No that's not supported at the moment. In this case the scheduler will only consider the first <download /> tag. But you can "work around" it: - create a new project and add the second download tag to it - afterwards you can include the new repo in your first project via "<repository>..."
Marcus
2010/10/4 Marcus Hüwe suse-tux@gmx.de:
On 2010-10-04 13:30:15 +0200, Francisco Javier Cuadrado wrote:
2010/9/28 Marcus Hüwe suse-tux@gmx.de:
On 2010-09-28 12:03:58 +0200, Francisco Javier Cuadrado wrote:
I'm trying to configure an own RPM repository to use download on demand, but I don't know how to do it, because into the OBS wiki[1] there is only an example to use debian repository.
My repository is a classic repomd, like the Fedora repository.
First you need to download the "primary.xml.gz" file. Afterwards uncompress it and store it to /srv/obs/build/<project>/<repo>/<arch>/:full/primary.xml Next you need to edit the project meta for instance via osc: "osc meta prj <project> -e" and add the following:
<download baseurl="http://<url>/<path" metafile="primary.xml" mtype="rpmmd" arch="<arch>" />
Choose "<path>" so that the rpms are located in "http://<url>/<path>/<arch>/*" and "http://<url>/<path>/noarch/*".
I've got a new doubt about DoD.
Can I have multiple primary files used into different download tags but for the same architecture? For example:
<download baseurl="http://repo-1/" metafile="primary-1.xml" arch="i586" ... /> <download baseurl="http://repo-2/" metafile="primary-2.xml" arch="i586" ... />
No that's not supported at the moment. In this case the scheduler will only consider the first <download /> tag. But you can "work around" it:
- create a new project and add the second download tag to it
- afterwards you can include the new repo in your first project via
"<repository>..."
Could this feature be implemented? It can be useful.
Marcus
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2010-10-04 15:19:24 +0200, Francisco Javier Cuadrado wrote:
Could this feature be implemented? It can be useful.
the solution provided by Marcus Hüwe would be cleaner imho.
darix
On Fri, Oct 8, 2010 at 10:31, Marcus Rueckert darix@opensu.se wrote:
On 2010-10-04 15:19:24 +0200, Francisco Javier Cuadrado wrote:
Could this feature be implemented? It can be useful.
the solution provided by Marcus Hüwe would be cleaner imho.
Yes, can we get this merged somehow? Even build.o.o would be able to probably profit from this (like ex. Fedora update repos) (and hopefully get this into 2.1 before it is too late)
2010/10/4 Marcus Hüwe suse-tux@gmx.de:
On 2010-10-04 13:30:15 +0200, Francisco Javier Cuadrado wrote:
2010/9/28 Marcus Hüwe suse-tux@gmx.de:
On 2010-09-28 12:03:58 +0200, Francisco Javier Cuadrado wrote:
I'm trying to configure an own RPM repository to use download on demand, but I don't know how to do it, because into the OBS wiki[1] there is only an example to use debian repository.
My repository is a classic repomd, like the Fedora repository.
First you need to download the "primary.xml.gz" file. Afterwards uncompress it and store it to /srv/obs/build/<project>/<repo>/<arch>/:full/primary.xml Next you need to edit the project meta for instance via osc: "osc meta prj <project> -e" and add the following:
<download baseurl="http://<url>/<path" metafile="primary.xml" mtype="rpmmd" arch="<arch>" />
Choose "<path>" so that the rpms are located in "http://<url>/<path>/<arch>/*" and "http://<url>/<path>/noarch/*".
I've got a new doubt about DoD.
Can I have multiple primary files used into different download tags but for the same architecture? For example:
<download baseurl="http://repo-1/" metafile="primary-1.xml" arch="i586" ... /> <download baseurl="http://repo-2/" metafile="primary-2.xml" arch="i586" ... />
No that's not supported at the moment. In this case the scheduler will only consider the first <download /> tag. But you can "work around" it:
- create a new project and add the second download tag to it
- afterwards you can include the new repo in your first project via
"<repository>..."
Sorry for answering so late, but now I'm trying to create a project using that explanation and I don't know if I'm doing it wrong or I didn't understand you.
I create a project with the following XML config:
<download arch="i586" metafile="primary.xml" mtype="rpmmd" baseurl="......"/> <repository name="standard"> <path project="openSUSE.org:Fedora:12" repository="standard"/> <arch>i586</arch> </repository>
But my project haven't got any build target, is my configuration wrong?
Thanks again.
Marcus
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
2010/10/18 Francisco Javier Cuadrado francisco.cuadrado@iunika.net:
2010/10/4 Marcus Hüwe suse-tux@gmx.de:
On 2010-10-04 13:30:15 +0200, Francisco Javier Cuadrado wrote:
2010/9/28 Marcus Hüwe suse-tux@gmx.de:
On 2010-09-28 12:03:58 +0200, Francisco Javier Cuadrado wrote:
I'm trying to configure an own RPM repository to use download on demand, but I don't know how to do it, because into the OBS wiki[1] there is only an example to use debian repository.
My repository is a classic repomd, like the Fedora repository.
First you need to download the "primary.xml.gz" file. Afterwards uncompress it and store it to /srv/obs/build/<project>/<repo>/<arch>/:full/primary.xml Next you need to edit the project meta for instance via osc: "osc meta prj <project> -e" and add the following:
<download baseurl="http://<url>/<path" metafile="primary.xml" mtype="rpmmd" arch="<arch>" />
Choose "<path>" so that the rpms are located in "http://<url>/<path>/<arch>/*" and "http://<url>/<path>/noarch/*".
I've got a new doubt about DoD.
Can I have multiple primary files used into different download tags but for the same architecture? For example:
<download baseurl="http://repo-1/" metafile="primary-1.xml" arch="i586" ... /> <download baseurl="http://repo-2/" metafile="primary-2.xml" arch="i586" ... />
No that's not supported at the moment. In this case the scheduler will only consider the first <download /> tag. But you can "work around" it:
- create a new project and add the second download tag to it
- afterwards you can include the new repo in your first project via
"<repository>..."
Sorry for answering so late, but now I'm trying to create a project using that explanation and I don't know if I'm doing it wrong or I didn't understand you.
I create a project with the following XML config:
<download arch="i586" metafile="primary.xml" mtype="rpmmd" baseurl="......"/>
<repository name="standard"> <path project="openSUSE.org:Fedora:12" repository="standard"/> <arch>i586</arch> </repository>
But my project haven't got any build target, is my configuration wrong?
Thanks again.
Sorry for my previous e-mail, but I only have to wait, and then my build target has appeared, I think OBS was updating something.
Marcus
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- Francisco Javier Cuadrado
SPAIN-CHINA-HONG KONG-POLAND
Spain Main: +34-910005436 Centro Empresarial Eisenhower C/ Cañada Real de Merinas, 25 Edificio 7 - oficina 1ºC 28042 Madrid
buildservice@lists.opensuse.org
-
Francisco Javier Cuadrado -
Marcus Hüwe -
Marcus Rueckert -
Martin Mohring -
Michael E Brown -
Robert Xu