On 30/03/2021 11.08, Gerald Pfeifer wrote:

On Thu 2021-03-18, medwinz wrote:

...

My logic is simple, mirrors cannot catch up with the new release if several major updates happen at nearly the same time.

That, plus the timing of announcements also makes a difference.

Not everyone here probably follows the factory@ list closely (I, too, read project@ more regularly), so allow me to share a change Bernhard has recently made which should also help.

Thank you, Bernhard!

For the record, here is what I found during debugging: After the previous full rebuilds, download.opensuse.org IPv6 address had ~50% packet loss for several hours. My nagios[1] has reported this at 2021-02-16 12:20 UTC (20210212 glibc rebuild published) 2021-03-08 11:25 UTC (20210307 glibc rebuild published) 2021-03-12 18:35 UTC (20210311 KDE update published) 2021-03-13 10:37 UTC (still fallout from 20210311?) 2021-03-13 18:09 UTC (still fallout from 20210311?) On the download.o.o VM, top showed ksoftirqd very busy at 98% CPU and several active rsyncd processes using the CPU. This indicates that syncing to the mirrors was still ongoing and that probably also slowed mirrorbrain mirror scans, so that more traffic was directed towards the downloadcontent.o.o site which happens to be the same machine as mirrorbrain (just another IP). All that syncing, scanning and downloading slowed each other down. Apart from the longer publish delay for large Tumbleweed snapshots, I plan to push the setup of the new widehat.o.o machine, so that it can act as a fallback mirror most of the time and relieve downloadcontent.o.o bandwidth. Ciao Bernhard M. [1] http://nagios.zq1.de/nagios4 => http://nagios.zq1.de/cgi-bin/nagios4/extinfo.cgi?type=1&host=oodownload

==== forwarded message ==== From: Bernhard M. Wiedemann <bernhardout@lsmod.de> To: factory@lists.opensuse.org Date: Sun, 28 Mar 2021 15:09:28 +0200 Subject: factory publish delay [was New Tumbleweed snapshot 20210325]

20210325 was the first large snapshot to trigger my new logic for a longer publish delay (6h instead of 2h).

The publisher became active when the repo age was 22203 s and the repo rsync log had 32114 lines and 2091370 byte.

This time, download.o.o seems to have had a much better time handling the load than with previous full rebuilds. Though this seems to have been only a partial rebuild because there are 62802 entries in factory/repo

Medium-term we can do more fine-tuning to balance publish delay vs load of download.o.o

Ciao Bernhard M.