On Mon, Sep 9, 2019 at 10:07 AM Per Jessen <per@opensuse.org> wrote:

I suggest you try:

rsync --av rsync://rsync.opensuse.org/opensuse-full-really-everything-including-repositories

and add excludes/includes for what you want or don't want.

Thanks. Your answer and this another answer helped me: https://serverfault.com/a/788007/357679 Let's say I want to mirror just the home:kamarada:15.1:dev project. I create a filter.txt file with contents: ✂---------------------------------------- + /opensuse/repositories/ + /opensuse/repositories/home:/ + /opensuse/repositories/home:/kamarada:/ + /opensuse/repositories/home:/kamarada:/15.1:/ + /opensuse/repositories/home:/kamarada:/15.1:/dev/*** - /opensuse/** ✂---------------------------------------- And run: # rsync -av --dry-run --filter="merge filter.txt" rsync://rsync.opensuse.org/opensuse-full-really-everything-including-repositories /srv/pub/ (remove --dry-run to actually download files) The problem now is: for some reason I'm not able to mirror the home:kamarada:15.1 project, which is the one I actually want. Following the same pattern, that should work: filter.txt: ✂---------------------------------------- + /opensuse/repositories/ + /opensuse/repositories/home:/ + /opensuse/repositories/home:/kamarada:/ + /opensuse/repositories/home:/kamarada:/15.1/*** - /opensuse/** ✂---------------------------------------- # rsync -av --dry-run --filter="merge filter.txt" rsync://rsync.opensuse.org/opensuse-full-really-everything-including-repositories /srv/pub/ Output is: [...] Thanks! admin@opensuse.org receiving incremental file list ./ opensuse/ opensuse/repositories/ opensuse/repositories/home:/ opensuse/repositories/home:/kamarada:/ sent 223 bytes received 182 bytes 115.71 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) This is the folder I want to mirror: https://download.opensuse.org/repositories/home:/kamarada:/15.1/ What is wrong? Antonio The Linux Kamarada Project http://kamarada.github.io/ -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Hi, On Tue, 10 Sep 2019, Per Jessen wrote:

...

...

rsync --av

rsync://rsync.opensuse.org/opensuse-full-really-everything-including-repositories

...

and add excludes/includes for what you want or don't want.

I'm no expert on rsync include/exclude/filter, but maybe try with this filter:

------------------------ + /opensuse/repositories/ + /opensuse/repositories/home:/ + /opensuse/repositories/home:/kamarada: + /opensuse/repositories/home:/kamarada:/15.1:/*** - /opensuse/** --------------------------

No, what he wants is really not there on the rsync server: % rsync rsync://rsync.opensuse.org/opensuse-full-really-everything-including-repositories/opensuse/repositories/home:/kamarada:/ ... drwxr-xr-x 140 2019/03/19 04:38:06 . drwxr-xr-x 25 2018/04/04 23:41:39 15.0: drwxr-xr-x 25 2019/03/19 04:38:06 15.1: drwxr-xr-x 98 2017/04/09 13:39:51 Playground drwxr-xr-x 26 2015/05/09 14:11:10 Playground: drwxr-xr-x 28 2017/12/03 23:37:14 dev drwxr-xr-x 10 2017/04/09 13:35:40 ikiwiki drwxr-xr-x 10 2017/04/09 13:35:02 opera See? No directory named '15.1' (without the colon). It is there on https://download.opensuse.org/repositories/home:/kamarada:/ . But it's not the colon itself that's breaking things, as the example of 'Playground' and 'Playground:' shows. Ciao, Michael. -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

On Tue, Sep 10, 2019 at 2:31 PM Linux Kamarada <linuxkamarada@gmail.com> wrote:

Probably one of these commands are going to work:

$ rsync -av rsync://opensuse.mirror.liquidtelecom.com/repositories/home:/kamarada:/15.1/ $ lftp -c mirror ftp://opensuse.mirror.liquidtelecom.com/opensuse/repositories/home:/kamarada:/15.1/

I'm not able to test them right now, but I'll inform you if they work when possible.

None of those worked for me. But I found a mirroring method which worked for me, which was wget: $ cd /srv/pub/kamarada/15.1/ $ wget -m -nH --cut-dirs=4 -np --reject 'index.*,*.meta4,*.metalink,*.mirrorlist' -e robots=off https://download.opensuse.org/repositories/home:/kamarada:/15.1/ Reference: https://superuser.com/a/952260/458848 Using those rejects, I download only exactly the same files which appear if I access download.opensuse.org using a web browser. Those *.meta4,*.metalink,*.mirrorlist files are generated by MirrorBrain. For what I want to do, I believe wget is going to suffice. If you find anything easier or more usual with rsync, please share. So, in short, in case anyone else needs, my conclusion is: to mirror an Open Build Service repository (e.g. home:myusername) using wget, run the following. $ cd /local/folder/ $ wget -m -nH --cut-dirs=3 -np --reject 'index.*,*.meta4,*.metalink,*.mirrorlist' -e robots=off https://download.opensuse.org/repositories/home:/myusername/ (mind the cut-dirs argument, which, in that example, says to skip the repositories/home:/myusername/ path when creating the local hierarchy) Thank you for your time and efforts trying to help me! Antonio The Linux Kamarada Project http://kamarada.github.io/ -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

On Tue, Sep 10, 2019 at 10:33 PM Linux Kamarada <linuxkamarada@gmail.com> wrote:

For what I want to do, I believe wget is going to suffice. If you find anything easier or more usual with rsync, please share.

So, in short, in case anyone else needs, my conclusion is: to mirror an Open Build Service repository (e.g. home:myusername) using wget, run the following.

$ cd /local/folder/

$ wget -m -nH --cut-dirs=3 -np --reject 'index.*,*.meta4,*.metalink,*.mirrorlist' -e robots=off https://download.opensuse.org/repositories/home:/myusername/

(mind the cut-dirs argument, which, in that example, says to skip the repositories/home:/myusername/ path when creating the local hierarchy)

A downside of mirroring with wget is that deleted files on OBS are not deleted locally... I'll try to workaround that. In fact, wget seems more as a workaround than as a solution. The best possible solution would be to use rsync... On Wed, Sep 11, 2019 at 1:21 AM Malcolm <malcolmlewis@cableone.net> wrote:

Hi I just use osc getbinaries to pull down packages locally, re-locate as required and then can use createrepo.

Hi, Malcolm, thanks for your suggestion! I imagine it is even scriptable, but it seems to me that it is not the easiest way (recreating the repo), so maybe I'll try it later... I'm now thinking on an algorithm like this: 1 Download everything from OBS (using wget, as above) 2 For each file stored locally: 2.1 Check whether the file exists online (for example, trying to download it and receiving HTTP 304 Not Modified or HTTP 404 Not Found, maybe that is possible using wget too) 2.1.1 If it does not exist online, delete it Antonio The Linux Kamarada Project http://kamarada.github.io/ -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org