repo updates for Tumbleweed must be atomic
While mirroring /tumbleweed/repo/oss from another mirror via rsync it became clear that these secondary mirrors do not provide atomic updates. The files are updated in random order. Most likely rsync was invoked without --delay-updates. Perhaps also without --partial and proper --partial-dir=DIR, in case this actually combines well with --delay-updates, just to avoid errors like rsync: send_files failed to open "/tumbleweed/repo/oss/x86_64/.lazarus-2.0.12-2.1.x86_64.rpm.C11vZI" (in opensuse): Permission denied (13) I do not know for how long rsync mirrors are offered. Was there any effort to provide mirror admins with instructions about how to mirror a volatile installation repo properly? It is required to transfer repodata/repomd.xml at the very end to keep the repo itself consistent. It is required to transfer /boot/x86_64 with --delay-updates to make sure the installer does not access incompatible files. This part might be tricky. While I could certainly implement all the required checks myself to have a consistent local mirror, I wonder if this was already solved in a generic way, given the issue exists since a decade. smt-server and rmt-server deal with the repodata part already, but they miss support to mirror the inst-sys. And of course they are unable to fix a bad secondary mirror. Olaf
On 15/10/2021 10.05, Olaf Hering wrote:
While mirroring /tumbleweed/repo/oss from another mirror via rsync it became clear that these secondary mirrors do not provide atomic updates. The files are updated in random order. Most likely rsync was invoked without --delay-updates. Perhaps also without --partial and proper --partial-dir=DIR, in case this actually combines well with --delay-updates, just to avoid errors like rsync: send_files failed to open "/tumbleweed/repo/oss/x86_64/.lazarus-2.0.12-2.1.x86_64.rpm.C11vZI" (in opensuse): Permission denied (13)
I do not know for how long rsync mirrors are offered. Was there any effort to provide mirror admins with instructions about how to mirror a volatile installation repo properly?
I'd recommend at least rsync -ay --delete-delay --delay-updates $source $destination It will use some GB extra diskspace while the transfer is ongoing, so users can still fetch the old versions until all the new ones are there. I guess, we should mention these in https://en.opensuse.org/openSUSE:Mirror_howto I have experienced similar problems when I created the opensuse.zq1.de IPFS mirror that always has consistent snapshots. There I use after rsync https://github.com/bmwiedemann/opensusearchive/blob/master/checkrepo to validate that the sync gave me all files. Otherwise, sync again later. Ciao Bernhard M.
This is some great advice.
I will implement it on my mirror server.
On Fri, Oct 15, 2021 at 10:31 AM Bernhard M. Wiedemann
On 15/10/2021 10.05, Olaf Hering wrote:
While mirroring /tumbleweed/repo/oss from another mirror via rsync it became clear that these secondary mirrors do not provide atomic updates. The files are updated in random order. Most likely rsync was invoked without --delay-updates. Perhaps also without --partial and proper --partial-dir=DIR, in case this actually combines well with --delay-updates, just to avoid errors like rsync: send_files failed to open "/tumbleweed/repo/oss/x86_64/.lazarus-2.0.12-2.1.x86_64.rpm.C11vZI" (in opensuse): Permission denied (13)
I do not know for how long rsync mirrors are offered. Was there any effort to provide mirror admins with instructions about how to mirror a volatile installation repo properly?
I'd recommend at least rsync -ay --delete-delay --delay-updates $source $destination
It will use some GB extra diskspace while the transfer is ongoing, so users can still fetch the old versions until all the new ones are there.
I guess, we should mention these in https://en.opensuse.org/openSUSE:Mirror_howto
I have experienced similar problems when I created the opensuse.zq1.de IPFS mirror that always has consistent snapshots. There I use after rsync https://github.com/bmwiedemann/opensusearchive/blob/master/checkrepo to validate that the sync gave me all files. Otherwise, sync again later.
Ciao Bernhard M.
On Fri, 15 Oct 2021, Bernhard M. Wiedemann wrote:
On 15/10/2021 10.05, Olaf Hering wrote:
While mirroring /tumbleweed/repo/oss from another mirror via rsync it became clear that these secondary mirrors do not provide atomic updates. The files are updated in random order. Most likely rsync was invoked without --delay-updates. Perhaps also without --partial and proper --partial-dir=DIR, in case this actually combines well with --delay-updates, just to avoid errors like rsync: send_files failed to open "/tumbleweed/repo/oss/x86_64/.lazarus-2.0.12-2.1.x86_64.rpm.C11vZI" (in opensuse): Permission denied (13)
I do not know for how long rsync mirrors are offered. Was there any effort to provide mirror admins with instructions about how to mirror a volatile installation repo properly?
I'd recommend at least rsync -ay --delete-delay --delay-updates $source $destination
It will use some GB extra diskspace while the transfer is ongoing, so users can still fetch the old versions until all the new ones are there.
I guess, we should mention these in https://en.opensuse.org/openSUSE:Mirror_howto
I'd also recommend that all sites exclude their tmpdir/tmpfiles on their rsyncd services to not expose those to other sites. For example, if you use --delay-updates with the default --partial-dir, exclude the dir .~tmp~ in your rsyncd config, something like this in rsyncd.conf: filter = -_.~tmp~/ If you use --partial-dir=.mysite filter = -_.mysite/ etc. /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se --------------------------------------------------------------------------- Mr. Worf! Set phasers on DEEP FRY!!!! =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Am Fri, 15 Oct 2021 10:31:42 +0200
schrieb "Bernhard M. Wiedemann"
rsync -ay --delete-delay --delay-updates $source $destination
I was surprised by the recommended rsync command, the docs appear to be stale. But, perhaps it just works fine for mirrorbrain itself.
https://github.com/bmwiedemann/opensusearchive/blob/master/checkrepo to validate that the sync gave me all files. Otherwise, sync again later.
Yes, I will implement something like that to get a consistent snapshot. Like building a list for --file-from=, based on repodata/ and CHECKSUMS. Olaf
participants (4)
-
Bernhard M. Wiedemann -
Eric Kochen -
Niklas Edmundsson -
Olaf Hering