[opensuse] fast rsync mirror for repositories
Hello: I would like to make a local copy of Leap 15.1 repositories before it disappears. Because of the structure of the repositories directory I use rsync with filters to download only 15.1 related stuff. Unfortunately I can't find a fast rsync mirror which has repositories. Earlier I used ftp.gwdg.de site but now when I start rsync on it I only get a "receiving file list ..." message and nothing else for more than an hour (until now). It is stuck or very slow I don't know. This is the command I run (dry-run only to test site): rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://ftp.gwdg.de/pub/opensuse/repositories/ opensuse/repositories Welcome to ftp.gwdg.de receiving file list ... Can someone suggest a fast working rsync mirror that has repositories? Thanks, Istvan
Istvan Gabor wrote:
Hello:
I would like to make a local copy of Leap 15.1 repositories before it disappears. Because of the structure of the repositories directory I use rsync with filters to download only 15.1 related stuff.
Unfortunately I can't find a fast rsync mirror which has repositories.
Earlier I used ftp.gwdg.de site but now when I start rsync on it I only get a "receiving file list ..." message and nothing else for more than an hour (until now). It is stuck or very slow I don't know.
This is the command I run (dry-run only to test site):
rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://ftp.gwdg.de/pub/opensuse/repositories/ opensuse/repositories Welcome to ftp.gwdg.de
receiving file list ...
Can someone suggest a fast working rsync mirror that has repositories?
rsync.opensuse.org ? -- Per Jessen, Zürich (8.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.
Thu, 19 Nov 2020 16:09:22 +0100 időpontban Per Jessen írta:
Istvan Gabor wrote:
Hello:
I would like to make a local copy of Leap 15.1 repositories before it disappears. Because of the structure of the repositories directory I use rsync with filters to download only 15.1 related stuff.
Unfortunately I can't find a fast rsync mirror which has repositories.
Earlier I used ftp.gwdg.de site but now when I start rsync on it I only get a "receiving file list ..." message and nothing else for more than an hour (until now). It is stuck or very slow I don't know.
This is the command I run (dry-run only to test site):
rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://ftp.gwdg.de/pub/opensuse/repositories/ opensuse/repositories Welcome to ftp.gwdg.de
receiving file list ...
Can someone suggest a fast working rsync mirror that has repositories?
rsync.opensuse.org ?
I tried it: rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://rsync.opensuse.org/repositories/ opensuse/repositories This is rsync.opensuse.org, public rsync server of openSUSE.org, limited to 50 connections. If you run a public mirror, please get in contact so we can give you access to the stage rsync server. You'll find conditions for access and further information at http://en.opensuse.org/Mirror_Infrastructure Thanks! admin@opensuse.org @ERROR: Unknown module 'repositories' rsync error: error starting client-server protocol (code 5) at main.c(1650) [Receiver=3.1.0 I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path? Istvan
Istvan Gabor wrote:
rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://rsync.opensuse.org/repositories/ opensuse/repositories
@ERROR: Unknown module 'repositories' rsync error: error starting client-server protocol (code 5) at main.c(1650) [Receiver=3.1.0
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
rsync --list-only rsync://rsync.opensuse.org will show you what is available - to rsync repositories, use "buildservice-repos" or "buildservice-repos-main" -- Per Jessen, Zürich (7.2°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland.
Thu, 19 Nov 2020 17:22:22 +0100 időpontban Per Jessen írta:
Istvan Gabor wrote:
rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://rsync.opensuse.org/repositories/ opensuse/repositories
@ERROR: Unknown module 'repositories' rsync error: error starting client-server protocol (code 5) at main.c(1650) [Receiver=3.1.0
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
rsync --list-only rsync://rsync.opensuse.org will show you what is available -
to rsync repositories, use "buildservice-repos" or "buildservice-repos-main"
Yes, that helped. I did not know this --list-only option, it's very useful. I wrongly supposed that http/ftp and rsync directory structures are the same. Thanks! Istvan
On 20/11/2020 10.12, Istvan Gabor wrote:
Thu, 19 Nov 2020 17:22:22 +0100 időpontban Per Jessen írta:
Istvan Gabor wrote:
rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://rsync.opensuse.org/repositories/ opensuse/repositories
@ERROR: Unknown module 'repositories' rsync error: error starting client-server protocol (code 5) at main.c(1650) [Receiver=3.1.0
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
rsync --list-only rsync://rsync.opensuse.org will show you what is available -
to rsync repositories, use "buildservice-repos" or "buildservice-repos-main"
Yes, that helped. I did not know this --list-only option, it's very useful. I wrongly supposed that http/ftp and rsync directory structures are the same.
Notice that you are not accessing the directory structure, but the "rsync modules" structure. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On 19/11/2020 16.47, Istvan Gabor wrote:
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
Module is "something" in rsync parlance. I don't see a description in "man rsync", only usage. It thinks that your command was trying to access a module on the remote server, but that module does not exist. To create a clone of some repos locally, I think it is best not to use rsync, but some normal download tool like wget. Runs faster. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
Carlos E. R. wrote:
On 19/11/2020 16.47, Istvan Gabor wrote:
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
Module is "something" in rsync parlance. I don't see a description in "man rsync", only usage.
man rsyncd.conf
It thinks that your command was trying to access a module on the remote server, but that module does not exist.
Istvan's invocation: rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://rsync.opensuse.org/repositories/ opensuse/repositories ^^^^^^^^^^^^ That is interpreted as the module.
To create a clone of some repos locally, I think it is best not to use rsync, but some normal download tool like wget. Runs faster.
Nah, use rsync, that is what all our mirrors use. It's the whole reason we provide rsync.opensuse.org for public use. -- Per Jessen, Zürich (8.6°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.
On 19/11/2020 19.24, Per Jessen wrote:
Carlos E. R. wrote:
On 19/11/2020 16.47, Istvan Gabor wrote:
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
Module is "something" in rsync parlance. I don't see a description in "man rsync", only usage.
man rsyncd.conf
Well, it explains how to create one, but does not say what a module is. At least I do not see it, because there are too many hits on the word.
It thinks that your command was trying to access a module on the remote server, but that module does not exist.
Istvan's invocation: rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://rsync.opensuse.org/repositories/ opensuse/repositories ^^^^^^^^^^^^
That is interpreted as the module.
To create a clone of some repos locally, I think it is best not to use rsync, but some normal download tool like wget. Runs faster.
Nah, use rsync, that is what all our mirrors use. It's the whole reason we provide rsync.opensuse.org for public use.
I tried once and I had to give up. IMO, use it to keep the mirror updated, but not for the initial population. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
Carlos E. R. wrote:
On 19/11/2020 19.24, Per Jessen wrote:
Carlos E. R. wrote:
On 19/11/2020 16.47, Istvan Gabor wrote:
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
Module is "something" in rsync parlance. I don't see a description in "man rsync", only usage.
man rsyncd.conf
Well, it explains how to create one, but does not say what a module is.
A module is a named rsync unit that represents a certain directory path - at rsync.o.o we have e.g. "buildservice-repos" which represents a local directory path.
It thinks that your command was trying to access a module on the remote server, but that module does not exist.
Istvan's invocation: rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://rsync.opensuse.org/repositories/ opensuse/repositories ^^^^^^^^^^^^
That is interpreted as the module.
To create a clone of some repos locally, I think it is best not to use rsync, but some normal download tool like wget. Runs faster.
Nah, use rsync, that is what all our mirrors use. It's the whole reason we provide rsync.opensuse.org for public use.
I tried once and I had to give up. IMO, use it to keep the mirror updated, but not for the initial population.
I suspect that was/is a local problem. -- Per Jessen, Zürich (8.3°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2020-11-19 at 20:45 +0100, Per Jessen wrote:
Carlos E. R. wrote:
On 19/11/2020 19.24, Per Jessen wrote:
Carlos E. R. wrote:
On 19/11/2020 16.47, Istvan Gabor wrote:
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
Module is "something" in rsync parlance. I don't see a description in "man rsync", only usage.
man rsyncd.conf
Well, it explains how to create one, but does not say what a module is.
A module is a named rsync unit that represents a certain directory path -
at rsync.o.o we have e.g. "buildservice-repos" which represents a local directory path.
Yes yes, I have a local rsync server and I use modules. I "know" what they are, but not formally. I failed to find a definition in the man pages.
It thinks that your command was trying to access a module on the remote server, but that module does not exist.
Istvan's invocation: rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://rsync.opensuse.org/repositories/ opensuse/repositories ^^^^^^^^^^^^
That is interpreted as the module.
To create a clone of some repos locally, I think it is best not to use rsync, but some normal download tool like wget. Runs faster.
Nah, use rsync, that is what all our mirrors use. It's the whole reason we provide rsync.opensuse.org for public use.
I tried once and I had to give up. IMO, use it to keep the mirror updated, but not for the initial population.
I suspect that was/is a local problem.
No, the transmission was slow. It spent time thinking whether to download files or not. It was faster to use wget and bypass the redirector. I must have the script somewhere, in an external disk probably. Found it - huh, I have to swithc to Alpine, Thunderbird can no longer cope with long lines: !/bin/bash OPCIONES="--limit-rate=500K --no-parent --no-host-directories --timestamping --no-verbose" time wget --limit-rate=500K --no-parent --no-host-directories --timestamping --reject .mirrorlist,.btih,.meta4,.magnet,.metalink -r http://download.opensuse.org/distribution/13.1/repo/non-oss/ time wget --limit-rate=500K --no-parent --no-host-directories --timestamping --reject .mirrorlist,.btih,.meta4,.magnet,.metalink -r http://download.opensuse.org/distribution/13.1/repo/oss/ time wget --limit-rate=500K --no-parent --no-host-directories --timestamping --reject .mirrorlist,.btih,.meta4,.magnet,.metalink -r http://download.opensuse.org/update/13.1/ echo "non-oss" | tee -a wget.log echo | tee -a wget.log time wget --limit-rate=500K --no-parent --no-host-directories --timestamping -r http://download.opensuse.org/distribution/13.1/repo/non-oss/ --append-output=wget.log echo >> wget.log echo "=====================================" >> wget.log echo >> wget.log ... My notes do not contain how/if I tried first with rsync. Possibly I asked about this in the mail list - yes, found it, 2016-01-28, but no record of the rsync command I tried. - -- Cheers, Carlos E. R. (from openSUSE 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCX7bSeBwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVxgkAniar3e1jw3fqJJqXtrDL i5i6JYovAJ4ghV34OBtaruGL8cZxDC4lKaPvWQ== =8lJk -----END PGP SIGNATURE-----
Carlos E. R. wrote:
On Thursday, 2020-11-19 at 20:45 +0100, Per Jessen wrote:
Carlos E. R. wrote:
On 19/11/2020 19.24, Per Jessen wrote:
Carlos E. R. wrote:
On 19/11/2020 16.47, Istvan Gabor wrote:
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
Module is "something" in rsync parlance. I don't see a description in "man rsync", only usage.
man rsyncd.conf
Well, it explains how to create one, but does not say what a module is.
A module is a named rsync unit that represents a certain directory path -
at rsync.o.o we have e.g. "buildservice-repos" which represents a local directory path.
Yes yes, I have a local rsync server and I use modules. I "know" what they are, but not formally. I failed to find a definition in the man pages.
I don't understand what you are looking for. An rsync module is a named config in rsyncd.conf, with a set of specific attributes. -- Per Jessen, Zürich (7.1°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland.
On 19/11/2020 21.29, Per Jessen wrote:
Carlos E. R. wrote:
On Thursday, 2020-11-19 at 20:45 +0100, Per Jessen wrote:
Carlos E. R. wrote:
On 19/11/2020 19.24, Per Jessen wrote:
Carlos E. R. wrote:
On 19/11/2020 16.47, Istvan Gabor wrote: > I don't know what "Unknown module 'repositories'" means. Path? > It yes, what would be the > correct path?
Module is "something" in rsync parlance. I don't see a description in "man rsync", only usage.
man rsyncd.conf
Well, it explains how to create one, but does not say what a module is.
A module is a named rsync unit that represents a certain directory path -
at rsync.o.o we have e.g. "buildservice-repos" which represents a local directory path.
Yes yes, I have a local rsync server and I use modules. I "know" what they are, but not formally. I failed to find a definition in the man pages.
I don't understand what you are looking for.
An rsync module is a named config in rsyncd.conf, with a set of specific attributes.
Just a paragraph in the man page defining what a rsync module is. You don't have to explain to me what a module is, I know what it is and I use them - except that I can not put a definnition in words. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
Carlos E. R. wrote:
Yes yes, I have a local rsync server and I use modules. I "know" what they are, but not formally. I failed to find a definition in the man pages.
I don't understand what you are looking for.
An rsync module is a named config in rsyncd.conf, with a set of specific attributes.
Just a paragraph in the man page defining what a rsync module is.
From the man page:
"After the global parameters you should define a number of modules, each module exports a directory tree as a symbolic name.". I think that's pretty succinct? -- Per Jessen, Zürich (7.1°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland.
On 19/11/2020 21.55, Per Jessen wrote:
Carlos E. R. wrote:
Yes yes, I have a local rsync server and I use modules. I "know" what they are, but not formally. I failed to find a definition in the man pages.
I don't understand what you are looking for.
An rsync module is a named config in rsyncd.conf, with a set of specific attributes.
Just a paragraph in the man page defining what a rsync module is.
From the man page:
"After the global parameters you should define a number of modules, each module exports a directory tree as a symbolic name.".
I think that's pretty succinct?
Well... :-) I hoped to see: Definition of Terms Module. A module is... whatever. Don't worry about it, it's me being grumpy. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
19.11.2020 18:47, Istvan Gabor пишет:
Thu, 19 Nov 2020 16:09:22 +0100 időpontban Per Jessen írta:
Istvan Gabor wrote:
Hello:
I would like to make a local copy of Leap 15.1 repositories before it disappears. Because of the structure of the repositories directory I use rsync with filters to download only 15.1 related stuff.
Unfortunately I can't find a fast rsync mirror which has repositories.
Earlier I used ftp.gwdg.de site but now when I start rsync on it I only get a "receiving file list ..." message and nothing else for more than an hour (until now). It is stuck or very slow I don't know.
This is the command I run (dry-run only to test site):
rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://ftp.gwdg.de/pub/opensuse/repositories/ opensuse/repositories Welcome to ftp.gwdg.de
receiving file list ...
Can someone suggest a fast working rsync mirror that has repositories?
rsync.opensuse.org ?
I tried it:
rsync -av -n --include-from=filters.txt --prune-empty-dirs rsync://rsync.opensuse.org/repositories/ opensuse/repositories
This is rsync.opensuse.org, public rsync server of openSUSE.org, limited to 50 connections.
If you run a public mirror, please get in contact so we can give you access to the stage rsync server. You'll find conditions for access and further information at http://en.opensuse.org/Mirror_Infrastructure
Thanks! admin@opensuse.org
@ERROR: Unknown module 'repositories' rsync error: error starting client-server protocol (code 5) at main.c(1650) [Receiver=3.1.0
I don't know what "Unknown module 'repositories'" means. Path? It yes, what would be the correct path?
https://en.opensuse.org/openSUSE:Mirror_infrastructure#rsync_modules
participants (4)
-
Andrei Borzenkov -
Carlos E. R. -
Istvan Gabor -
Per Jessen