[opensuse-buildservice] "quick" way to checkout a repo
Hello, Is there is a "quick" way to checkout a repository, like mirror w/rsync, aria2c, or something like that? I need to checkout the devel:languages:R:CRAN-packages repository, or at least a very large number of packages, which has some 3400+ packages and checking out w/osc is going to take a long time. Thanks! -- Later, Darin -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 2014-04-15 12:23:27 -0400, Darin Perusich wrote:
Is there is a "quick" way to checkout a repository, like mirror w/rsync, aria2c, or something like that? I need to checkout the devel:languages:R:CRAN-packages repository, or at least a very large number of packages, which has some 3400+ packages and checking out w/osc is going to take a long time.
I assume you want to checkout the source "project" and not the (binary) repository. AFAIK, there is no way to checkout the sources except doing a http request for each single file. I think we could speed up osc's checkout process, if the backend has support for retrieving the source files of a package in a single cpio archive. Adding such a feature shouldn't be too hard (the workers already retrieve the sources this way). Any opinions? Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am 15.04.2014 19:02, schrieb Marcus Hüwe:
On 2014-04-15 12:23:27 -0400, Darin Perusich wrote:
Is there is a "quick" way to checkout a repository, like mirror w/rsync, aria2c, or something like that? I need to checkout the devel:languages:R:CRAN-packages repository, or at least a very large number of packages, which has some 3400+ packages and checking out w/osc is going to take a long time.
I assume you want to checkout the source "project" and not the (binary) repository. AFAIK, there is no way to checkout the sources except doing a http request for each single file.
I think we could speed up osc's checkout process, if the backend has support for retrieving the source files of a package in a single cpio archive. Adding such a feature shouldn't be too hard (the workers already retrieve the sources this way).
Any opinions?
Actually in my experience with checking out large directories, the http transfer is not the only problem with osc. osc also parses and recreates huge XML files all the time. You notice this especially when updating existant checkouts - when almost no files are transferred. But yes, speeding up source downloads ven try would help too (actually, I'm afraid we don't even support compression for these http transfers. Greetings, Stephan -- Ma muaß weiterkämpfen, kämpfen bis zum Umfalln, a wenn die ganze Welt an Arsch offen hat, oder grad deswegn. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 2014-04-16 07:24:16 +0200, Stephan Kulow wrote:
Am 15.04.2014 19:02, schrieb Marcus Hüwe:
On 2014-04-15 12:23:27 -0400, Darin Perusich wrote:
Is there is a "quick" way to checkout a repository, like mirror w/rsync, aria2c, or something like that? I need to checkout the devel:languages:R:CRAN-packages repository, or at least a very large number of packages, which has some 3400+ packages and checking out w/osc is going to take a long time.
I assume you want to checkout the source "project" and not the (binary) repository. AFAIK, there is no way to checkout the sources except doing a http request for each single file.
I think we could speed up osc's checkout process, if the backend has support for retrieving the source files of a package in a single cpio archive. Adding such a feature shouldn't be too hard (the workers already retrieve the sources this way).
Any opinions?
Actually in my experience with checking out large directories, the http transfer is not the only problem with osc. osc also parses and recreates huge XML files all the time.
What do you mean with a large directory? A project with many packages or a package with many files? If it's the latter, do you have an example?
You notice this especially when updating existant checkouts - when almost no files are transferred.
Hmm I would still suspect that this is due to the http requests: even if no files changed, osc does at least one (or two, if the package is an expanded link) http request.
But yes, speeding up source downloads ven try would help too (actually, I'm afraid we don't even support compression for these http transfers.
Ok. I'll have a look at it. Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Darin Perusich -
Marcus Hüwe -
Stephan Kulow