[opensuse-buildservice] obs_mirror_project
I had some trouble with the obs_mirror_project in the new build of obs-server, so I rewrote it in Python. It's called exactly the same way but it differs in the following ways: - Uses the osc library directly to perform api operations - Instead of downloading every single RPM on every invocation, it only downloads RPMs that have not already been downloaded - RPMs are downloaded to a temporary directly and moved to the local project once completed. If a download fails, there won't be any half-downloaded RPMs causing buildroot failures - RPMs that are no longer on the api server are cleaned up - Download progress is displayed -- James Oakley jfunk@funktronics.ca
On Thursday 16 August 2007 02:38:27 wrote James Oakley:
I had some trouble with the obs_mirror_project in the new build of obs-server, so I rewrote it in Python. It's called exactly the same way but it differs in the following ways:
- Uses the osc library directly to perform api operations
- Instead of downloading every single RPM on every invocation, it only downloads RPMs that have not already been downloaded
- RPMs are downloaded to a temporary directly and moved to the local project once completed. If a download fails, there won't be any half-downloaded RPMs causing buildroot failures
- RPMs that are no longer on the api server are cleaned up
- Download progress is displayed
great stuff :) I think I will drop my ruby script and package yours instead ... Would a GPLv2 license okay with you ? Would it be okay when we relicense it at any later point under a later GPL license ? Peter, do you think that this might become part of osc instead of the obs packages ? bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Aug 16, 2007 at 10:13:18AM +0200, Adrian Schröter wrote:
On Thursday 16 August 2007 02:38:27 wrote James Oakley:
I had some trouble with the obs_mirror_project in the new build of obs-server, so I rewrote it in Python. It's called exactly the same way but it differs in the following ways:
- Uses the osc library directly to perform api operations
- Instead of downloading every single RPM on every invocation, it only downloads RPMs that have not already been downloaded
- RPMs are downloaded to a temporary directly and moved to the local project once completed. If a download fails, there won't be any half-downloaded RPMs causing buildroot failures
- RPMs that are no longer on the api server are cleaned up
- Download progress is displayed
great stuff :)
I think I will drop my ruby script and package yours instead ...
Would a GPLv2 license okay with you ? Would it be okay when we relicense it at any later point under a later GPL license ?
Peter, do you think that this might become part of osc instead of the obs packages ?
Yes. It could also become a osc subcommand I think. In fact, I have started looking at it 20 minutes ago, and I'm trying to add the missing hooks to the osc core module to make this task easier. For example, a "binary list" command has been on the TODO for a while. I'm browsing https://api.opensuse.org/apidocs/, to find possible use cases and to come up with a user interface. There seem to be several possible ways to list binaries (per package vs. repository wide, for instance). For this obs script, maybe it would be interesting to have a (more general) osc command to download/mirror all binaries of a project into a directory. An obs-specific wrapper could then look pretty simple. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
Hi, and an additional request: currently, obs_mirror_project overwrites <project>.xml and <project>.conf in /srv/obs/projects. This can be quite confusing if you update the packages from the server. Also, does the server somehow provide checksums to check for consistency of the rpm base? Seems that also other people have jumped on this... Martin Dr. Peter Poeml wrote:
On Thu, Aug 16, 2007 at 10:13:18AM +0200, Adrian Schröter wrote:
On Thursday 16 August 2007 02:38:27 wrote James Oakley:
I had some trouble with the obs_mirror_project in the new build of obs-server, so I rewrote it in Python. It's called exactly the same way but it differs in the following ways:
- Uses the osc library directly to perform api operations
- Instead of downloading every single RPM on every invocation, it only downloads RPMs that have not already been downloaded
- RPMs are downloaded to a temporary directly and moved to the local project once completed. If a download fails, there won't be any half-downloaded RPMs causing buildroot failures
- RPMs that are no longer on the api server are cleaned up
- Download progress is displayed
great stuff :)
I think I will drop my ruby script and package yours instead ...
Would a GPLv2 license okay with you ? Would it be okay when we relicense it at any later point under a later GPL license ?
Peter, do you think that this might become part of osc instead of the obs packages ?
Yes.
It could also become a osc subcommand I think.
In fact, I have started looking at it 20 minutes ago, and I'm trying to add the missing hooks to the osc core module to make this task easier.
For example, a "binary list" command has been on the TODO for a while. I'm browsing https://api.opensuse.org/apidocs/, to find possible use cases and to come up with a user interface. There seem to be several possible ways to list binaries (per package vs. repository wide, for instance).
For this obs script, maybe it would be interesting to have a (more general) osc command to download/mirror all binaries of a project into a directory. An obs-specific wrapper could then look pretty simple.
Peter
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday 16 August 2007 5:13 am, Adrian Schröter wrote:
Would a GPLv2 license okay with you ? Would it be okay when we relicense it at any later point under a later GPL license ?
Absolutely, I submitted it for potential inclusion. -- James Oakley jfunk@funktronics.ca --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Aug 16, 2007 at 09:45:30AM -0300, James Oakley wrote:
On Thursday 16 August 2007 5:13 am, Adrian Schröter wrote:
Would a GPLv2 license okay with you ? Would it be okay when we relicense it at any later point under a later GPL license ?
Absolutely, I submitted it for potential inclusion.
Great. Thank you! I have been playing with an osc "mirror_binaries" command. Looks promising. Can download binaries project-wide or per package. But has the same problems you also mentioned, like not really checking for uptodate-ness. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Thu, Aug 16, 2007 at 04:18:26PM +0200, Dr. Peter Poeml wrote:
On Thu, Aug 16, 2007 at 09:45:30AM -0300, James Oakley wrote:
On Thursday 16 August 2007 5:13 am, Adrian Schröter wrote:
Would a GPLv2 license okay with you ? Would it be okay when we relicense it at any later point under a later GPL license ?
Absolutely, I submitted it for potential inclusion.
Great. Thank you!
I have been playing with an osc "mirror_binaries" command. Looks promising. Can download binaries project-wide or per package. But has the same problems you also mentioned, like not really checking for uptodate-ness.
Alright, I just committed all what's needed in osc the core to list and download binaries. I also committed an extension to the "ls" command to list binaries. Then I have a command to mirror binaries to a local directory. It is attached. (Try it by putting it to .osc-plugins/) Thanks, Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
James, I have added a get_binary_list() to osc (just checked it into svn). I also hacked up a "ls --binary" extension to osc (which uses this new method). Not in svn yet. I am attaching a patch to the script you posted. It uses the new osc function, and I simplified two places where I used two existing functions to pull stuff from the api, and the permissions bitmask fix I mentioned before. I would like to integrate more stuff into osc, if possible. The script as such is not 100% suitable, because it requires some actions to be done as root (chown obsrun, as required by the build service backend). Maybe the download part could be integrated into osc, while the chown business could be deferred to another script... or should osc be run directly by the obsrun user, maybe? But an interesting part to integrate would be the "get binary" part, from your script -- your agreement provided. What do you think? I'm thinking of a command to download a binary, one command to download all binaries of a package, and of a command to mirror a project like you do. I am also interested in feedback about friendlyness of using the osc module from external scripts. I tend to rather add things inside osc, by adding a subcommand, which means that using osc from externally is not so well tested. It may not be optimal therefore. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Thursday 16 August 2007 7:50 am, Dr. Peter Poeml wrote:
James,
I have added a get_binary_list() to osc (just checked it into svn).
I also hacked up a "ls --binary" extension to osc (which uses this new method). Not in svn yet.
I am attaching a patch to the script you posted. It uses the new osc function, and I simplified two places where I used two existing functions to pull stuff from the api, and the permissions bitmask fix I mentioned before.
I would like to integrate more stuff into osc, if possible.
It would be nice to be able to get some more metadata from the binary list. It appears that binaries built from projects are listed as <name>.rpm instead of <name>-<version>-<release>.<arch>.rpm. There's no way to tell from the filename alone if it's different from a local copy. An md5sum would solve that problem.
The script as such is not 100% suitable, because it requires some actions to be done as root (chown obsrun, as required by the build service backend). Maybe the download part could be integrated into osc, while the chown business could be deferred to another script... or should osc be run directly by the obsrun user, maybe?
Yeah, there are better solutions to that problem. os.setuid(), for example.
But an interesting part to integrate would be the "get binary" part, from your script -- your agreement provided. What do you think?
I'm thinking of a command to download a binary, one command to download all binaries of a package, and of a command to mirror a project like you do.
Yeah, that would be great.
I am also interested in feedback about friendlyness of using the osc module from external scripts.
It works great, in my experience. This isn't the first script I've written with it. The initialisation isn't as intuitive as it could be, however: conf.get_config() conf.config['apiurl'] = conf.config['scheme'] + '://' + conf.config['apisrv'] conf.config['user'] = conf.config['auth_dict'][conf.config['apisrv']]['user'] conf.init_basicauth(conf.config) It would be nicer if there was one function to call, eg: conf.init(apisrv=None) where the default api server will be used if apisrv is None. Some simple examples in the documentation would help those looking to learn the library as well. -- James Oakley jfunk@funktronics.ca --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Aug 16, 2007 at 10:43:27AM -0300, James Oakley wrote:
On Thursday 16 August 2007 7:50 am, Dr. Peter Poeml wrote:
James,
I have added a get_binary_list() to osc (just checked it into svn).
I also hacked up a "ls --binary" extension to osc (which uses this new method). Not in svn yet.
I am attaching a patch to the script you posted. It uses the new osc function, and I simplified two places where I used two existing functions to pull stuff from the api, and the permissions bitmask fix I mentioned before.
I would like to integrate more stuff into osc, if possible.
It would be nice to be able to get some more metadata from the binary list. It appears that binaries built from projects are listed as <name>.rpm instead of <name>-<version>-<release>.<arch>.rpm. There's no way to tell from the filename alone if it's different from a local copy. An md5sum would solve that problem.
It might be possible that there is a way. When asking for binaries in the backend, they are unversioned. When asking for published binaries though, they are versioned: % osc-stable req published/Apache/SLE_10/x86_64/ <directory> <entry name="GeoIP-1.4.2-4.2.x86_64.rpm" /> <entry name="GeoIP-debuginfo-1.4.2-4.2.x86_64.rpm" /> ... I'm not sure at the moment, if the size, mtime or md5sum can be obtained somehow. I don't find it in the api docs. Andreas? Michael?
I am also interested in feedback about friendlyness of using the osc module from external scripts.
It works great, in my experience. This isn't the first script I've written with it.
Cool. And good to know. I usually try not to break existing usage of the code... being aware of users increases the motivation for that, and makes me know that it is worth my time :-)
The initialisation isn't as intuitive as it could be, however:
conf.get_config() conf.config['apiurl'] = conf.config['scheme'] + '://' + conf.config['apisrv'] conf.config['user'] = conf.config['auth_dict'][conf.config['apisrv']]['user'] conf.init_basicauth(conf.config)
It would be nicer if there was one function to call, eg:
conf.init(apisrv=None)
where the default api server will be used if apisrv is None.
Some simple examples in the documentation would help those looking to learn the library as well.
Definitely. I'll put this onto the TODO. Thanks! Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Thu, Aug 16, 2007 at 10:43:27AM -0300, James Oakley wrote:
I am also interested in feedback about friendlyness of using the osc module from external scripts.
It works great, in my experience. This isn't the first script I've written with it.
The initialisation isn't as intuitive as it could be, however:
conf.get_config() conf.config['apiurl'] = conf.config['scheme'] + '://' + conf.config['apisrv'] conf.config['user'] = conf.config['auth_dict'][conf.config['apisrv']]['user'] conf.init_basicauth(conf.config)
It would be nicer if there was one function to call, eg:
conf.init(apisrv=None)
where the default api server will be used if apisrv is None.
Thanks for the suggestion. I had something like that in mind, but deferred it until I replace the dictionary-based configuration with a more object-based approach. However, I just looked at the initialization code and it turned out that it was easier to integrate into get_config() than I thought. The function prototype now is: get_config(override_conffile = None, override_http_debug = None, override_apisrv = None) Please let me know if it works for you in the way you suggested. Thanks, Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Wednesday 22 August 2007 7:18 am, Dr. Peter Poeml wrote:
However, I just looked at the initialization code and it turned out that it was easier to integrate into get_config() than I thought. The function prototype now is:
get_config(override_conffile = None, override_http_debug = None, override_apisrv = None)
Please let me know if it works for you in the way you suggested.
That looks perfect. Thanks, -- James Oakley jfunk@funktronics.ca --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Wed, Aug 15, 2007 at 09:38:27PM -0300, James Oakley wrote:
I had some trouble with the obs_mirror_project in the new build of obs-server, so I rewrote it in Python. It's called exactly the same way but it differs in the following ways:
- Uses the osc library directly to perform api operations
- Instead of downloading every single RPM on every invocation, it only downloads RPMs that have not already been downloaded
- RPMs are downloaded to a temporary directly and moved to the local project once completed. If a download fails, there won't be any half-downloaded RPMs causing buildroot failures
- RPMs that are no longer on the api server are cleaned up
- Download progress is displayed
An idea, regardless of the usefulness of a "mirror binaries" tool: How about implementing the import of e.g. 10.2 binaries into a local buildservice instance by _uploading_ the binaries via the api? Instead of writing the prj xml, the prf conf and all the binaries into the local filesystem, it should be possible to do this via the _api_. This way we would get rid of the requirement to chown files to the obsrun user. There are api calls to save prj xml and config, but I'm not sure about a "put binary" path. Does that exist? Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Friday 17 August 2007 00:15, Dr. Peter Poeml wrote:
How about implementing the import of e.g. 10.2 binaries into a local buildservice instance by _uploading_ the binaries via the api?
Instead of writing the prj xml, the prf conf and all the binaries into the local filesystem, it should be possible to do this via the _api_. This way we would get rid of the requirement to chown files to the obsrun user.
Sounds like a good idea to me. -- Cornelius Schumacher <cschum@suse.de> --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Friday 17 August 2007 00:15:56 wrote Dr. Peter Poeml:
On Wed, Aug 15, 2007 at 09:38:27PM -0300, James Oakley wrote:
I had some trouble with the obs_mirror_project in the new build of obs-server, so I rewrote it in Python. It's called exactly the same way but it differs in the following ways:
- Uses the osc library directly to perform api operations
- Instead of downloading every single RPM on every invocation, it only downloads RPMs that have not already been downloaded
- RPMs are downloaded to a temporary directly and moved to the local project once completed. If a download fails, there won't be any half-downloaded RPMs causing buildroot failures
- RPMs that are no longer on the api server are cleaned up
- Download progress is displayed
An idea, regardless of the usefulness of a "mirror binaries" tool:
How about implementing the import of e.g. 10.2 binaries into a local buildservice instance by _uploading_ the binaries via the api?
Instead of writing the prj xml, the prf conf and all the binaries into the local filesystem, it should be possible to do this via the _api_. This way we would get rid of the requirement to chown files to the obsrun user.
There are api calls to save prj xml and config, but I'm not sure about a "put binary" path. Does that exist?
not for the public api by intention for now. We would need at least some restriction enhancements for that, because we explicit disallow binary uploads for .opensuse.org . The script could use however the backend api for now ... bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, Aug 17, 2007 at 02:00:19PM +0200, Adrian Schröter wrote:
There are api calls to save prj xml and config, but I'm not sure about a "put binary" path. Does that exist?
not for the public api by intention for now.
Okay, I suspected that already. But that is a local restriction which doesn't necessarily make sense for every buildservice instance. The API should support it, in general, IMO. A sensible default would be to allow it only for the admin account, I think.
We would need at least some restriction enhancements for that, because we explicit disallow binary uploads for .opensuse.org .
The script could use however the backend api for now ...
Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
participants (5)
-
Adrian Schröter -
Cornelius Schumacher -
Dr. Peter Poeml -
James Oakley -
Martin Mohring