It was suggested I repost this here. I originally posted it in Project on the openSUSE Discord Server. "I don't know if this is the proper place for this to get attention but I'll post this here anyway. Firstly I want to praise all the people who contribute to the required work to keep Tumbleweed rolling on. The latest snapshot was a huge undertaking and much work was required to make it a success. So thanks to all those people. That said however many people had a painful experience obtaining this snapshot. For whatever reason there was much timing out and many had to manually ride herd on the dup. I don't think it is an issue of sufficiency of mirrors. I think it has more to do with MirrorBrain. It's just not routing people to the closest and most robust mirror. Here on Discord there was much discussion of the multi day difficulty in getting the multi gigabyte snapshot downloaded. Is there something that can be done about this?" This is a reoccurring issue. Steven -- ____________ Apply appropriate technology. Use what works without prejudice. Steven L Hess ARS K6KGE DM05gd22 Owner Flex-1500, Flex-6300, Flex-6400, FT-857D, and FT-817ND 927.0875Mhz and 441.125 analog 441.575 DMR, and 441.600 Fusion Repeaters Taft Ca. openSUSE Tumbleweed KDE Plasma with Packman
Steven Hess wrote:
It was suggested I repost this here. I originally posted it in Project on the openSUSE Discord Server. "I don't know if this is the proper place for this to get attention but I'll post this here anyway. Firstly I want to praise all the people who contribute to the required work to keep Tumbleweed rolling on. The latest snapshot was a huge undertaking and much work was required to make it a success. So thanks to all those people.
That said however many people had a painful experience obtaining this snapshot. For whatever reason there was much timing out and many had to manually ride herd on the dup. I don't think it is an issue of sufficiency of mirrors. I think it has more to do with MirrorBrain. It's just not routing people to the closest and most robust mirror.
'robustness' was never a selection factor :-)
Here on Discord there was much discussion of the multi day difficulty in getting the multi gigabyte snapshot downloaded. Is there something that can be done about this?"
If anyone is going to look at anything, we'll need some more information. For instance, example of Mirrorbrain not picking the right mirrors for people. Timeouts is not something we can do much about. Some example filenames or URLs would also be helpful. Please let us continue this on mirror.lists or heroes.list. -- Per Jessen, Zürich (3.7°C) Member, openSUSE Heroes
And that will accomplish? It would seem you'd need detailed errors from many people and I do not believe that any such logging is done at least on my machine. We do not even know what mirror MirrorBrain selected for us. I am simply raising awareness of an ongoing and persistent problem for many of us. Wait and try later in not really a solution.
Steven Hess wrote:
And that will accomplish?
Uh, what will accomplish what? Reporting the issues I assume.
It would seem you'd need detailed errors from many people and I do not believe that any such logging is done at least on my machine.
Somewhat detailed from at least one or two would be a good starting point.
We do not even know what mirror MirrorBrain selected for us. I am simply raising awareness of an ongoing and persistent problem for many of us. Wait and try later in not really a solution.
However ongoing and persistent it may be, raising awareness to something largely unknown is not conducive to a solution either. :-) IOW, if I don't know what it is, there is little I can do about it. -- Per Jessen, Zürich (3.6°C) Member, openSUSE Heroes
Hello, Am Sonntag, 14. März 2021, 19:37:50 CET schrieb Steven Hess:
And that will accomplish? It would seem you'd need detailed errors from many people and I do not believe that any such logging is done at least on my machine. We do not even know what mirror MirrorBrain selected for us.
It's a bit hidden, but it's there ;-) - check your /var/log/zypper.log (or the YaST log if you use YaST). To give you an idea what you have to look for, I'll pick a random example from my zypper.log - in this case, the package was probably just built and not on any mirror yet, therefore I ended up on downloadcontent.opensuse.org: 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp++] MediaCurl.cc(doGetFileCopyFile):1139 URL: http://download.opensuse.org/repositories/home:/cboltz/openSUSE_Factory/noar... 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1374 HTTP response: 200 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://downloadcontent.opensuse.org/repositories/home:/cboltz/openSUSE_Facto..., ) 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp++] MediaCurl.cc(setupEasy):472 Proxy: not explicitly set 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp++] MediaCurl.cc(setupEasy):473 Proxy: libcurl may look into the environment 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600 2021-03-09 12:52:20 <2> tux.boltz(13697) [zypp] MediaMultiCurl.cc(run):1150 overall result 2021-03-09 12:52:20 <2> tux.boltz(13697) [zypp] MediaMultiCurl.cc(run):1154 #0: state: 4 received: 418994 url: http://downloadcontent.opensuse.org/repositories/home:/cboltz/openSUSE_Facto... And yes, I fully agree with Seife that zypper should print a more useful error message so that you see the failing mirror instead of having to read the logfile. Regards, Christian Boltz -- I have the ideal solution for you to speed up the writing of the manuals: http://www.lipsum.com/ - I am sure almost nobody will notice the difference. ;-) [houghi in opensuse-wiki]
Trying to get useful information that file now. I located made a copy to my documents folder and changed the permissions so I can manipulate it. It's more lines than the defaults on my editor will handle. It will take me a while to process. It's huge. This won't be quick.
On 15/03/2021 23.26, Steven Hess wrote:
Trying to get useful information that file now. I located made a copy to my documents folder and changed the permissions so I can manipulate it. It's more lines than the defaults on my editor will handle. It will take me a while to process. It's huge. This won't be quick.
Use "less" to see the file, in a terminal. Or use "grep" to write entries that contain useful strings to another file. Or to automatically remove lines that do not interest you, producing a smaller file. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
I don't know about the mirror issues but apparently there was a cascade of issues lately with updating TW, from the rebuild of all packages after updating a C library to saturated servers to zypper errors caused by the curl utility. Are we planning on a post-mortem of the situation, tomake sure that we extract everything there is to know about this crisis? I feel like the huge drop in stability scores of TW snapshots (seeing from https://review.tumbleweed.boombatower.com/) justifies taking this seriously
Adrien Glauser wrote:
I don't know about the mirror issues but apparently there was a cascade of issues lately with updating TW, from the rebuild of all packages after updating a C library to saturated servers to zypper errors caused by the curl utility.
A major update of TW will cause some delays in mirrors getting updated.
Are we planning on a post-mortem of the situation, tomake sure that we extract everything there is to know about this crisis?
For starters, I don't even know of any crisis. :-) -- Per Jessen, Zürich (3.2°C) Member, openSUSE Heroes
On 3/15/21 7:36 AM, Per Jessen wrote:
Adrien Glauser wrote:
I don't know about the mirror issues but apparently there was a cascade of issues lately with updating TW, from the rebuild of all packages after updating a C library to saturated servers to zypper errors caused by the curl utility.
A major update of TW will cause some delays in mirrors getting updated.
Are we planning on a post-mortem of the situation, tomake sure that we extract everything there is to know about this crisis?
For starters, I don't even know of any crisis. :-)
Likewise I was unaware of one, but i'll correct a slight misunderstanding, The rebuild of all packages isn't due to an issue its an intentional design decision. A couple of times a year the release managers decide to rebuild the distro to ensure nothing gets missed. Often they time it with an update like glibc because due to the very large number of packages that depend on it most of the distro would have been updated anyway. If there have been serious bugs that have come up, the best approach as always is to figure out if they could have been replicated by the packages unit test suite or openqa and then add appropriate tests so that a similar issue can't be released again. Often this is much easier said then done. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
Yeah the word "crisis" was an exaggeration of mine. I just mean that Steven Hess' concerns about the mirrors are shared by many people, for instance https://www.reddit.com/r/openSUSE/comments/m3vbx2/download_mirrors_confusion.... As for my remark on zypper, see https://www.reddit.com/r/openSUSE/comments/m3uj78/seriously_please_fix_zyppe.... Finally for the idea that TW snapshots are having a tough time, see https://tinyurl.com/yjkb9pgg. So yeah, perhaps not a crisis, but overall reasons for concerns.
On 15/03/2021 00.37, Adrien Glauser wrote:
Yeah the word "crisis" was an exaggeration of mine. I just mean that Steven Hess' concerns about the mirrors are shared by many people, for instance https://www.reddit.com/r/openSUSE/comments/m3vbx2/download_mirrors_confusion.... As for my remark on zypper, see https://www.reddit.com/r/openSUSE/comments/m3uj78/seriously_please_fix_zyppe.... Finally for the idea that TW snapshots are having a tough time, see https://tinyurl.com/yjkb9pgg.
So yeah, perhaps not a crisis, but overall reasons for concerns.
Perhaps those people should speak up the issues here, in the the mail list, rather than elsewhere. That way we all would know about them. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list. I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered. All the errors everyone was getting were similar to this: "Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"
Hi Steven, Am 15.03.21 um 01:54 schrieb Steven Hess:
All the errors everyone was getting were similar to this:
"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"
Maybe a bug report against libzypp / zypper would be appropriate here. Not to fix the download error (they probably can't fix mirror problems in software easily ;-)), but the error message has much room for improvement: * it should show additionally where the request has been redirected to (and where the error probably really occured) * "Curl error 55" -- in addition to the numerical error, a descriptive string would be nice (the number is good because it has no issues with translations etc, but having to look it up is not so nice) But maybe it's really time to get rid of zypp and switch to dnf (if it only could do the equivalent of "zypper dupp --no-allow-vendor-change"). Best regards -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
On Mon, Mar 15, 2021 at 2:39 AM Stefan Seyfried <stefan.seyfried@googlemail.com> wrote:
Hi Steven,
Am 15.03.21 um 01:54 schrieb Steven Hess:
All the errors everyone was getting were similar to this:
"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"
Maybe a bug report against libzypp / zypper would be appropriate here.
Not to fix the download error (they probably can't fix mirror problems in software easily ;-)), but the error message has much room for improvement: * it should show additionally where the request has been redirected to (and where the error probably really occured) * "Curl error 55" -- in addition to the numerical error, a descriptive string would be nice (the number is good because it has no issues with translations etc, but having to look it up is not so nice)
But maybe it's really time to get rid of zypp and switch to dnf (if it only could do the equivalent of "zypper dupp --no-allow-vendor-change").
This is already the default behavior on DNF now: https://build.opensuse.org/package/view_file/openSUSE:Factory/libdnf/libdnf-... -- 真実はいつも一つ！/ Always, there's only one truth!
On 15/03/2021 01.54, Steven Hess wrote:
I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.
I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.
All the errors everyone was getting were similar to this:
"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"
55 Failed sending network data. And other people had 16: 16 HTTP/2 error. A problem was detected in the HTTP2 framing layer. This is somewhat generic and can be one out of several problems, see the error message for details. as reported on the factory mail list. But I do not see the relationship to mirrorbrain. :-? -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
Am 15.03.21 um 09:08 schrieb Carlos E. R.:
On 15/03/2021 01.54, Steven Hess wrote:
I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.
I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.
All the errors everyone was getting were similar to this:
"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"
55 Failed sending network data.
And other people had 16:
16 HTTP/2 error. A problem was detected in the HTTP2 framing layer. This is somewhat generic and can be one out of several problems, see the error message for details.
as reported on the factory mail list.
But I do not see the relationship to mirrorbrain. :-?
I had similar issue with using download.o.o directly. In use is openSUSE Tumbleweed. "zypper dup" wants to download more than 8.000 packages. In the time of "zypper dup" I typed 5 times (r) - retry because after 5 file requests is given up. I assume that the OP (Steven Hess) and myself don't use the same net, but the problems appeared at the same time. The zypper.log for the questionable time: https://paste.opensuse.org/2a5847b6 Log-Time = CET / Germany Internetprovider = Vodafone (former UnityMedia) over fiberglass -- Kind regards, Sebastian - openSUSE Member (Freespacer) https://connect.opensuse.org/pg/profile/Freespacer Important notes on openSUSE Mailing List: https://en.opensuse.org/openSUSE:Mailing_list_netiquette
On 16/03/2021 00.27, Sebastian Siebert wrote:
Am 15.03.21 um 09:08 schrieb Carlos E. R.:
On 15/03/2021 01.54, Steven Hess wrote:
I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.
I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.
All the errors everyone was getting were similar to this:
"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....':
Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"
55 Failed sending network data.
And other people had 16:
16 HTTP/2 error. A problem was detected in the HTTP2 framing layer. This is somewhat generic and can be one out of several problems, see the error message for details.
as reported on the factory mail list.
But I do not see the relationship to mirrorbrain. :-?
I had similar issue with using download.o.o directly. In use is openSUSE Tumbleweed. "zypper dup" wants to download more than 8.000 packages. In the time of "zypper dup" I typed 5 times (r) - retry because after 5 file requests is given up. I assume that the OP (Steven Hess) and myself don't use the same net, but the problems appeared at the same time.
The zypper.log for the questionable time: https://paste.opensuse.org/2a5847b6
It seems "simply" that server was too busy. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
Am 16.03.21 um 02:59 schrieb Carlos E. R.:
It seems "simply" that server was too busy.
I hope the underlying problem is solved now - since March 4th mirrorbrain was not scanning mirrors, so all new files since were served by the same server. That affected obviously Tumbleweed, but also maintenance updates released since. Greetings, Stephan -- Lighten up, just enjoy life, smile more, laugh more, and don't get so worked up about things. Kenneth Branagh
Well that explains it all. And it was MirrorBrain. Thanks for this important information. Pretty much puts this to rest. Again thanks.
Steven Hess wrote:
I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.
I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.
Please simply report mirror problems to admin@o.o. We certainly don't want any unstable mirrors going undetected. -- Per Jessen, Zürich (3.1°C) Member, openSUSE Heroes
Adrien Glauser wrote:
Yeah the word "crisis" was an exaggeration of mine.
Fair enough, okay :-)
I just mean that Steven Hess' concerns about the mirrors are shared by many people, for instance https://www.reddit.com/r/openSUSE/comments/m3vbx2/download_mirrors_confusion....
It's a bit too long for my attention span - I see someone complaining about a mirror being out-of-date, which just ought to be reported (if it is correct). Which MasterPatricko also mentioned in his reply. As for the "master download server" being overloaded - yeah, it is certainly possible. Many of our mirrors will be trying to sync, which will likely use up all available bandwidth. It is a temporary situation though.
As for my remark on zypper, see https://www.reddit.com/r/openSUSE/comments/m3uj78/seriously_please_fix_zyppe....
MasterPatricko seems to be providing some sane answers. All in all, I see business as usual: - a large number of changes are pushed to download.o.o - mirrors will be picking those up, but slowly. - syncing large amounts of data take large amounts of time, exacerbated by multiple mirrors sync'ing concurrently. - users attempting download will suffer, until we (i.e. mirrorbrain) have ascertained mirror status. -- Per Jessen, Zürich (3.0°C) Member, openSUSE Heroes
On Mon, Mar 15, 2021 at 4:06 AM Per Jessen <per@opensuse.org> wrote:
Adrien Glauser wrote:
I don't know about the mirror issues but apparently there was a cascade of issues lately with updating TW, from the rebuild of all packages after updating a C library to saturated servers to zypper errors caused by the curl utility.
A major update of TW will cause some delays in mirrors getting updated.
As someone who maintain 2 mirrors for ID/AS, I can verify this. 15.3 Beta release and the last 2 TW snapshot was so closed. It makes mirrors way too busy to send and receive the traffic. One of my mirror can only get 20 Mbit/s rsync data from staging, usually it can get at least 50 Mbit/s. Best, -- Edwin
On Mon 2021-03-15, medwinz wrote:
A major update of TW will cause some delays in mirrors getting updated. As someone who maintain 2 mirrors for ID/AS, I can verify this. 15.3 Beta release and the last 2 TW snapshot was so closed. It makes mirrors way too busy to send and receive the traffic.
Thank you for sharing this, Edwin! In the weekly release engineering meeting I just suggested whether we could look into spacing Leap Betas/RCs and larger Tumbleweed snapshots farther apart, and Lubos (Leap) kindly indicated to have a chat with Dominique (Tumbleweed). Gerald
On 17/03/2021 11.05, Gerald Pfeifer wrote:
On Mon 2021-03-15, medwinz wrote:
A major update of TW will cause some delays in mirrors getting updated. As someone who maintain 2 mirrors for ID/AS, I can verify this. 15.3 Beta release and the last 2 TW snapshot was so closed. It makes mirrors way too busy to send and receive the traffic.
Thank you for sharing this, Edwin!
In the weekly release engineering meeting I just suggested whether we could look into spacing Leap Betas/RCs and larger Tumbleweed snapshots farther apart, and Lubos (Leap) kindly indicated to have a chat with Dominique (Tumbleweed).
How about some system to signal to zypper to abort the update, not to do it now, and retry in some hours? It could be a flag file at http://download.opensuse.org itself (not synced to mirrors). When it is considered that mirrors had time to sync, that "not_now" file gets removed. The small flag file could contain some information text. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
On Wed, Mar 17, 2021 at 5:05 PM Gerald Pfeifer <gp@suse.com> wrote:
On Mon 2021-03-15, medwinz wrote:
A major update of TW will cause some delays in mirrors getting updated. As someone who maintain 2 mirrors for ID/AS, I can verify this. 15.3 Beta release and the last 2 TW snapshot was so closed. It makes mirrors way too busy to send and receive the traffic.
Thank you for sharing this, Edwin!
In the weekly release engineering meeting I just suggested whether we could look into spacing Leap Betas/RCs and larger Tumbleweed snapshots farther apart, and Lubos (Leap) kindly indicated to have a chat with Dominique (Tumbleweed).
Gerald
Thank you Gerald. My logic is simple, mirrors cannot catch up with the new release if several major updates happen at nearly the same time. They will be late. Many mirrors provide repositories not only for openSUSE but also for other linux distributions. This will overburden the main update repository and probably users will complain about time-out or delay in update and dup process as well. Complaining about the update process is not good for the distribution. I read the meeting minutes, thanks for the hard work of Lubos and Dominique (Is Max also still involved in TW?). Anyway, the green vaccine seems to work quite well on them! -- Edwin
On Thu 2021-03-18, medwinz wrote:
My logic is simple, mirrors cannot catch up with the new release if several major updates happen at nearly the same time.
That, plus the timing of announcements also makes a difference. Not everyone here probably follows the factory@ list closely (I, too, read project@ more regularly), so allow me to share a change Bernhard has recently made which should also help. Thank you, Bernhard! ==== forwarded message ==== From: Bernhard M. Wiedemann <bernhardout@lsmod.de> To: factory@lists.opensuse.org Date: Sun, 28 Mar 2021 15:09:28 +0200 Subject: factory publish delay [was New Tumbleweed snapshot 20210325] 20210325 was the first large snapshot to trigger my new logic for a longer publish delay (6h instead of 2h). The publisher became active when the repo age was 22203 s and the repo rsync log had 32114 lines and 2091370 byte. This time, download.o.o seems to have had a much better time handling the load than with previous full rebuilds. Though this seems to have been only a partial rebuild because there are 62802 entries in factory/repo Medium-term we can do more fine-tuning to balance publish delay vs load of download.o.o Ciao Bernhard M.
On 30/03/2021 11.08, Gerald Pfeifer wrote:
On Thu 2021-03-18, medwinz wrote:
My logic is simple, mirrors cannot catch up with the new release if several major updates happen at nearly the same time.
That, plus the timing of announcements also makes a difference.
Not everyone here probably follows the factory@ list closely (I, too, read project@ more regularly), so allow me to share a change Bernhard has recently made which should also help.
Thank you, Bernhard!
For the record, here is what I found during debugging: After the previous full rebuilds, download.opensuse.org IPv6 address had ~50% packet loss for several hours. My nagios[1] has reported this at 2021-02-16 12:20 UTC (20210212 glibc rebuild published) 2021-03-08 11:25 UTC (20210307 glibc rebuild published) 2021-03-12 18:35 UTC (20210311 KDE update published) 2021-03-13 10:37 UTC (still fallout from 20210311?) 2021-03-13 18:09 UTC (still fallout from 20210311?) On the download.o.o VM, top showed ksoftirqd very busy at 98% CPU and several active rsyncd processes using the CPU. This indicates that syncing to the mirrors was still ongoing and that probably also slowed mirrorbrain mirror scans, so that more traffic was directed towards the downloadcontent.o.o site which happens to be the same machine as mirrorbrain (just another IP). All that syncing, scanning and downloading slowed each other down. Apart from the longer publish delay for large Tumbleweed snapshots, I plan to push the setup of the new widehat.o.o machine, so that it can act as a fallback mirror most of the time and relieve downloadcontent.o.o bandwidth. Ciao Bernhard M. [1] http://nagios.zq1.de/nagios4 => http://nagios.zq1.de/cgi-bin/nagios4/extinfo.cgi?type=1&host=oodownload
==== forwarded message ==== From: Bernhard M. Wiedemann <bernhardout@lsmod.de> To: factory@lists.opensuse.org Date: Sun, 28 Mar 2021 15:09:28 +0200 Subject: factory publish delay [was New Tumbleweed snapshot 20210325]
20210325 was the first large snapshot to trigger my new logic for a longer publish delay (6h instead of 2h).
The publisher became active when the repo age was 22203 s and the repo rsync log had 32114 lines and 2091370 byte.
This time, download.o.o seems to have had a much better time handling the load than with previous full rebuilds. Though this seems to have been only a partial rebuild because there are 62802 entries in factory/repo
Medium-term we can do more fine-tuning to balance publish delay vs load of download.o.o
Ciao Bernhard M.
On Tue, Mar 30, 2021 at 4:48 PM Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote:
On 30/03/2021 11.08, Gerald Pfeifer wrote:
On Thu 2021-03-18, medwinz wrote:
My logic is simple, mirrors cannot catch up with the new release if several major updates happen at nearly the same time.
That, plus the timing of announcements also makes a difference.
Not everyone here probably follows the factory@ list closely (I, too, read project@ more regularly), so allow me to share a change Bernhard has recently made which should also help.
Thank you, Bernhard!
Apart from the longer publish delay for large Tumbleweed snapshots, I plan to push the setup of the new widehat.o.o machine, so that it can act as a fallback mirror most of the time and relieve downloadcontent.o.o bandwidth.
...
This time, download.o.o seems to have had a much better time handling the load than with previous full rebuilds. Though this seems to have been only a partial rebuild because there are 62802 entries in
factory/repo
....
Bernhard M.
Thank you Gerhard and Bernhard. Really appreciate your solutions. -- Edwin
participants (15)
-
Adrien Glauser -
Bernhard M. Wiedemann -
Carlos E. R. -
Carlos E. R. -
Christian Boltz -
Gerald Pfeifer -
medwinz -
medwinz -
Neal Gompa -
Per Jessen -
Sebastian Siebert -
Simon Lees -
Stefan Seyfried -
Stephan Kulow -
Steven Hess