Dear Andrii, It's just an idea, but how about doing away with the ssh push sync solution altogether, due to recent vulnerabilities/backdoors in sshd, and also because not all mirror admins like to give this kind of even limited rights to other projects, even if they have IP limited access. Instead, there could be a much simpler and cleaner way to do it. You could create a simple nginx web server with subdirs or txt files with unique IDs if needed for all mirrors, so you could easily monitor - who and when retrieved the URL, in case, if there was an outage. The mirrors would query the txt file on the URL generated for them every minute with wget or curl, the value in the txt file could be 0 or 1, if zero then no update (no changes in the main repo) is needed, if 1 then an update would be triggered (the script you provide or rsync command you recommend etc.). If there is an update to do, then the value could remain 1 for about 3-5 minutes in case of a network problem that prevents the mirror from accessing the URL - the flock will not start the rsync process in multiple instances anyway. Essentially, the same result can be achieved as with push sync, but without installing ssh and other customized packages. By the way, there is no silver bullet in mirror operation, there will be problems that need to be solved by the admin of the mirror and no ready-made solution will help, if these problems cannot be solved by someone, there is nothing to talk about. Sometimes rsync process gets stuck, sometimes systemd does not restart the web server or the rsyncd service, but nothing what cannot be solved with a simple if else script. Other thought: If you want you can also introduce rsync over TLS as an option in addition to plain rsync, just if it does not place too much extra load on the main repo server. Thank you for your hard work and ideas to make things better! Have a very nice day! Cheers, Peter On 2024-10-21 10:21, Andrii Nikitin wrote:
Hej,
On various occasions, I heard multiple requests about having a simple way to set up and maintain an openSUSE mirror. Since then I had some ideas about "projects" as they are defined at mirrors.opensuse.org, and the fact that each project may need a dedicated rsync process with customized individual check intervals and maybe some kind of notification from the server when new content is arrived, etc.
E.g. Leap iso and repo files rarely change, so they might need to be checked less frequently than Leap updates, etc. But when a big sync starts for Leap quarterly updates (.iso files) - it shouldn't delay the sync of updates for long.
With these ideas I implemented opensuse-rsync packages. Ideally users follow the following steps: * Choose a size of the mirror according to the list: https://github.com/andrii-suse/opensuse-rsync/blob/master/README.md#approxim... * Install the corresponding package, e.g. opensuse-rsync-typical (which will require approximately 1.2TB of disk space). * Enable the timers using the provided command. * Optionally add custom filters to a config. * Monitor and manage systemd services for each project.
So, the idea is that such steps might be the default set of actions to set up a mirror (but not required). Further, it will be easier to gather feedback and implement improvements or add functionality.
I would appreciate it if somebody had a chance to review this proposal and share ideas if it looks legit or why it is a bad idea. I will be glad to answer questions about the implementation or design decisions, etc.
If no feedback is provided, I probably will add it as an experimental section to the wiki and wait for somebody to try it.
Github: https://github.com/andrii-suse/opensuse-rsync OBS: https://build.opensuse.org/project/show/home:andriinikitin:opensuse-rsync
Regards, Andrii Nikitin