Steven Hess wrote:

And that will accomplish?

Uh, what will accomplish what? Reporting the issues I assume.

It would seem you'd need detailed errors from many people and I do not believe that any such logging is done at least on my machine.

Somewhat detailed from at least one or two would be a good starting point.

We do not even know what mirror MirrorBrain selected for us. I am simply raising awareness of an ongoing and persistent problem for many of us. Wait and try later in not really a solution.

However ongoing and persistent it may be, raising awareness to something largely unknown is not conducive to a solution either. :-) IOW, if I don't know what it is, there is little I can do about it. -- Per Jessen, Zürich (3.6°C) Member, openSUSE Heroes

Trying to get useful information that file now. I located made a copy to my documents folder and changed the permissions so I can manipulate it. It's more lines than the defaults on my editor will handle. It will take me a while to process. It's huge. This won't be quick.

On 3/15/21 7:36 AM, Per Jessen wrote:

Adrien Glauser wrote:

...

I don't know about the mirror issues but apparently there was a cascade of issues lately with updating TW, from the rebuild of all packages after updating a C library to saturated servers to zypper errors caused by the curl utility.

A major update of TW will cause some delays in mirrors getting updated.

...

Are we planning on a post-mortem of the situation, tomake sure that we extract everything there is to know about this crisis?

For starters, I don't even know of any crisis. :-)

Likewise I was unaware of one, but i'll correct a slight misunderstanding, The rebuild of all packages isn't due to an issue its an intentional design decision. A couple of times a year the release managers decide to rebuild the distro to ensure nothing gets missed. Often they time it with an update like glibc because due to the very large number of packages that depend on it most of the distro would have been updated anyway. If there have been serious bugs that have come up, the best approach as always is to figure out if they could have been replicated by the packages unit test suite or openqa and then add appropriate tests so that a similar issue can't be released again. Often this is much easier said then done. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B

On 15/03/2021 00.37, Adrien Glauser wrote:

Yeah the word "crisis" was an exaggeration of mine. I just mean that Steven Hess' concerns about the mirrors are shared by many people, for instance https://www.reddit.com/r/openSUSE/comments/m3vbx2/download_mirrors_confusion.... As for my remark on zypper, see https://www.reddit.com/r/openSUSE/comments/m3uj78/seriously_please_fix_zyppe.... Finally for the idea that TW snapshots are having a tough time, see https://tinyurl.com/yjkb9pgg.

So yeah, perhaps not a crisis, but overall reasons for concerns.

Perhaps those people should speak up the issues here, in the the mail list, rather than elsewhere. That way we all would know about them. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

Hi Steven, Am 15.03.21 um 01:54 schrieb Steven Hess:

All the errors everyone was getting were similar to this:

"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"

Maybe a bug report against libzypp / zypper would be appropriate here. Not to fix the download error (they probably can't fix mirror problems in software easily ;-)), but the error message has much room for improvement: * it should show additionally where the request has been redirected to (and where the error probably really occured) * "Curl error 55" -- in addition to the numerical error, a descriptive string would be nice (the number is good because it has no issues with translations etc, but having to look it up is not so nice) But maybe it's really time to get rid of zypp and switch to dnf (if it only could do the equivalent of "zypper dupp --no-allow-vendor-change"). Best regards -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman

On 15/03/2021 01.54, Steven Hess wrote:

I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.

I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.

All the errors everyone was getting were similar to this:

"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"

55 Failed sending network data. And other people had 16: 16 HTTP/2 error. A problem was detected in the HTTP2 framing layer. This is somewhat generic and can be one out of several problems, see the error message for details. as reported on the factory mail list. But I do not see the relationship to mirrorbrain. :-? -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 16/03/2021 00.27, Sebastian Siebert wrote:

Am 15.03.21 um 09:08 schrieb Carlos E. R.:

...

On 15/03/2021 01.54, Steven Hess wrote:

...

I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.

I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.

All the errors everyone was getting were similar to this:

"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....':

Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"

         55     Failed sending network data.

And other people had 16:

         16     HTTP/2 error. A problem  was  detected  in  the                 HTTP2  framing  layer. This is somewhat generic                 and can be one out of several problems, see the                 error message for details.

as reported on the factory mail list.

But I do not see the relationship to mirrorbrain. :-?

I had similar issue with using download.o.o directly. In use is openSUSE Tumbleweed. "zypper dup" wants to download more than 8.000 packages. In the time of "zypper dup" I typed 5 times (r) - retry because after 5 file requests is given up. I assume that the OP (Steven Hess) and myself don't use the same net, but the problems appeared at the same time.

The zypper.log for the questionable time: https://paste.opensuse.org/2a5847b6

It seems "simply" that server was too busy. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

Well that explains it all. And it was MirrorBrain. Thanks for this important information. Pretty much puts this to rest. Again thanks.

Adrien Glauser wrote:

Yeah the word "crisis" was an exaggeration of mine.

Fair enough, okay :-)

I just mean that Steven Hess' concerns about the mirrors are shared by many people, for instance https://www.reddit.com/r/openSUSE/comments/m3vbx2/download_mirrors_confusion....

It's a bit too long for my attention span - I see someone complaining about a mirror being out-of-date, which just ought to be reported (if it is correct). Which MasterPatricko also mentioned in his reply. As for the "master download server" being overloaded - yeah, it is certainly possible. Many of our mirrors will be trying to sync, which will likely use up all available bandwidth. It is a temporary situation though.

As for my remark on zypper, see https://www.reddit.com/r/openSUSE/comments/m3uj78/seriously_please_fix_zyppe....

MasterPatricko seems to be providing some sane answers. All in all, I see business as usual: - a large number of changes are pushed to download.o.o - mirrors will be picking those up, but slowly. - syncing large amounts of data take large amounts of time, exacerbated by multiple mirrors sync'ing concurrently. - users attempting download will suffer, until we (i.e. mirrorbrain) have ascertained mirror status. -- Per Jessen, Zürich (3.0°C) Member, openSUSE Heroes

On Mon 2021-03-15, medwinz wrote:

...

A major update of TW will cause some delays in mirrors getting updated. As someone who maintain 2 mirrors for ID/AS, I can verify this. 15.3 Beta release and the last 2 TW snapshot was so closed. It makes mirrors way too busy to send and receive the traffic.

Thank you for sharing this, Edwin! In the weekly release engineering meeting I just suggested whether we could look into spacing Leap Betas/RCs and larger Tumbleweed snapshots farther apart, and Lubos (Leap) kindly indicated to have a chat with Dominique (Tumbleweed). Gerald

On Wed, Mar 17, 2021 at 5:05 PM Gerald Pfeifer wrote:

On Mon 2021-03-15, medwinz wrote:

...

...

A major update of TW will cause some delays in mirrors getting updated. As someone who maintain 2 mirrors for ID/AS, I can verify this. 15.3 Beta release and the last 2 TW snapshot was so closed. It makes mirrors way too busy to send and receive the traffic.

Thank you for sharing this, Edwin!

In the weekly release engineering meeting I just suggested whether we could look into spacing Leap Betas/RCs and larger Tumbleweed snapshots farther apart, and Lubos (Leap) kindly indicated to have a chat with Dominique (Tumbleweed).

Gerald

Thank you Gerald. My logic is simple, mirrors cannot catch up with the new release if several major updates happen at nearly the same time. They will be late. Many mirrors provide repositories not only for openSUSE but also for other linux distributions. This will overburden the main update repository and probably users will complain about time-out or delay in update and dup process as well. Complaining about the update process is not good for the distribution. I read the meeting minutes, thanks for the hard work of Lubos and Dominique (Is Max also still involved in TW?). Anyway, the green vaccine seems to work quite well on them! -- Edwin

On 30/03/2021 11.08, Gerald Pfeifer wrote:

On Thu 2021-03-18, medwinz wrote:

...

My logic is simple, mirrors cannot catch up with the new release if several major updates happen at nearly the same time.

That, plus the timing of announcements also makes a difference.

Not everyone here probably follows the factory@ list closely (I, too, read project@ more regularly), so allow me to share a change Bernhard has recently made which should also help.

Thank you, Bernhard!

For the record, here is what I found during debugging: After the previous full rebuilds, download.opensuse.org IPv6 address had ~50% packet loss for several hours. My nagios[1] has reported this at 2021-02-16 12:20 UTC (20210212 glibc rebuild published) 2021-03-08 11:25 UTC (20210307 glibc rebuild published) 2021-03-12 18:35 UTC (20210311 KDE update published) 2021-03-13 10:37 UTC (still fallout from 20210311?) 2021-03-13 18:09 UTC (still fallout from 20210311?) On the download.o.o VM, top showed ksoftirqd very busy at 98% CPU and several active rsyncd processes using the CPU. This indicates that syncing to the mirrors was still ongoing and that probably also slowed mirrorbrain mirror scans, so that more traffic was directed towards the downloadcontent.o.o site which happens to be the same machine as mirrorbrain (just another IP). All that syncing, scanning and downloading slowed each other down. Apart from the longer publish delay for large Tumbleweed snapshots, I plan to push the setup of the new widehat.o.o machine, so that it can act as a fallback mirror most of the time and relieve downloadcontent.o.o bandwidth. Ciao Bernhard M. [1] http://nagios.zq1.de/nagios4 => http://nagios.zq1.de/cgi-bin/nagios4/extinfo.cgi?type=1&host=oodownload

==== forwarded message ==== From: Bernhard M. Wiedemann To: factory@lists.opensuse.org Date: Sun, 28 Mar 2021 15:09:28 +0200 Subject: factory publish delay [was New Tumbleweed snapshot 20210325]

20210325 was the first large snapshot to trigger my new logic for a longer publish delay (6h instead of 2h).

The publisher became active when the repo age was 22203 s and the repo rsync log had 32114 lines and 2091370 byte.

This time, download.o.o seems to have had a much better time handling the load than with previous full rebuilds. Though this seems to have been only a partial rebuild because there are 62802 entries in factory/repo

Medium-term we can do more fine-tuning to balance publish delay vs load of download.o.o

Ciao Bernhard M.