Steven Hess wrote:

It was suggested I repost this here. I originally posted it in Project on the openSUSE Discord Server. "I don't know if this is the proper place for this to get attention but I'll post this here anyway. Firstly I want to praise all the people who contribute to the required work to keep Tumbleweed rolling on. The latest snapshot was a huge undertaking and much work was required to make it a success. So thanks to all those people.

That said however many people had a painful experience obtaining this snapshot. For whatever reason there was much timing out and many had to manually ride herd on the dup. I don't think it is an issue of sufficiency of mirrors. I think it has more to do with MirrorBrain. It's just not routing people to the closest and most robust mirror.

'robustness' was never a selection factor :-)

Here on Discord there was much discussion of the multi day difficulty in getting the multi gigabyte snapshot downloaded. Is there something that can be done about this?"

If anyone is going to look at anything, we'll need some more information. For instance, example of Mirrorbrain not picking the right mirrors for people. Timeouts is not something we can do much about. Some example filenames or URLs would also be helpful.

Please let us continue this on mirror.lists or heroes.list.

I don't know about the mirror issues but apparently there was a cascade of issues lately with updating TW, from the rebuild of all packages after updating a C library to saturated servers to zypper errors caused by the curl utility. Are we planning on a post-mortem of the situation, tomake sure that we extract everything there is to know about this crisis? I feel like the huge drop in stability scores of TW snapshots (seeing from https://review.tumbleweed.boombatower.com/) justifies taking this seriously

Adrien Glauser wrote:

I don't know about the mirror issues but apparently there was a cascade of issues lately with updating TW, from the rebuild of all packages after updating a C library to saturated servers to zypper errors caused by the curl utility.

A major update of TW will cause some delays in mirrors getting updated.

Are we planning on a post-mortem of the situation, tomake sure that we extract everything there is to know about this crisis?

For starters, I don't even know of any crisis. :-)

Yeah the word "crisis" was an exaggeration of mine. I just mean that Steven Hess' concerns about the mirrors are shared by many people, for instance https://www.reddit.com/r/openSUSE/comments/m3vbx2/download_mirrors_confusion.... As for my remark on zypper, see https://www.reddit.com/r/openSUSE/comments/m3uj78/seriously_please_fix_zyppe.... Finally for the idea that TW snapshots are having a tough time, see https://tinyurl.com/yjkb9pgg.

So yeah, perhaps not a crisis, but overall reasons for concerns.

I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.

I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.

All the errors everyone was getting were similar to this:

"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"

On 15/03/2021 01.54, Steven Hess wrote:

I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.

I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.

All the errors everyone was getting were similar to this:

"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"

55 Failed sending network data.

And other people had 16:

16 HTTP/2 error. A problem was detected in the HTTP2 framing layer. This is somewhat generic and can be one out of several problems, see the error message for details.

as reported on the factory mail list.

But I do not see the relationship to mirrorbrain. :-?

Steven Hess wrote:

I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.

I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.

Please simply report mirror problems to admin@o.o. We certainly don't want any unstable mirrors going undetected.

Hello,

Am Sonntag, 14. März 2021, 19:37:50 CET schrieb Steven Hess:

And that will accomplish? It would seem you'd need detailed errors from many people and I do not believe that any such logging is done at least on my machine. We do not even know what mirror MirrorBrain selected for us.

It's a bit hidden, but it's there ;-) - check your /var/log/zypper.log (or the YaST log if you use YaST).

To give you an idea what you have to look for, I'll pick a random example from my zypper.log - in this case, the package was probably just built and not on any mirror yet, therefore I ended up on downloadcontent.opensuse.org:

2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp++] MediaCurl.cc(doGetFileCopyFile):1139 URL: http://download.opensuse.org/repositories/home:/cboltz/openSUSE_Factory/noar... 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp++] MediaMultiCurl.cc(doGetFileCopy):1374 HTTP response: 200 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp] MediaCurl.cc(MediaCurl):192 MediaCurl::MediaCurl(http://downloadcontent.opensuse.org/repositories/home:/cboltz/openSUSE_Facto..., ) 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp++] MediaCurl.cc(setupEasy):472 Proxy: not explicitly set 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp++] MediaCurl.cc(setupEasy):473 Proxy: libcurl may look into the environment 2021-03-09 12:52:19 <1> tux.boltz(13697) [zypp] PathInfo.cc(chmod):1060 assert_file_mode 00600 /var/lib/YaST2/cookieschmod /var/lib/YaST2/cookies 00600 2021-03-09 12:52:20 <2> tux.boltz(13697) [zypp] MediaMultiCurl.cc(run):1150 overall result 2021-03-09 12:52:20 <2> tux.boltz(13697) [zypp] MediaMultiCurl.cc(run):1154 #0: state: 4 received: 418994 url: http://downloadcontent.opensuse.org/repositories/home:/cboltz/openSUSE_Facto...

And yes, I fully agree with Seife that zypper should print a more useful error message so that you see the failing mirror instead of having to read the logfile.

Regards,

Christian Boltz

On 15/03/2021 23.26, Steven Hess wrote:

Trying to get useful information that file now. I located made a copy to my documents folder and changed the permissions so I can manipulate it. It's more lines than the defaults on my editor will handle. It will take me a while to process. It's huge. This won't be quick.

Use "less" to see the file, in a terminal.

Or use "grep" to write entries that contain useful strings to another file. Or to automatically remove lines that do not interest you, producing a smaller file.

Am 15.03.21 um 09:08 schrieb Carlos E. R.:

On 15/03/2021 01.54, Steven Hess wrote:

...

I left the regular openSUSE mail lists because they were ineffective, often unhelpful, and full of noise. You'll only see me post on project and I'll only be readfin it and only reading it and the factory list.

I'm willing to do anything can as an individual on this list to try and help avoid the repeat of these errors and unstable mirrors being offered.

All the errors everyone was getting were similar to this:

"Download (curl) error for 'http://download.opensuse.org/tumbleweed/repo/oss/x86_64/xkbutils-1.0.4-5.20....': Error code: Curl error 55 Error message: Connection died, tried 5 times before giving up"

     55     Failed sending network data.

And other people had 16:

     16     HTTP/2 error. A problem  was  detected  in  the
            HTTP2  framing  layer. This is somewhat generic
            and can be one out of several problems, see the
            error message for details.

as reported on the factory mail list.

But I do not see the relationship to mirrorbrain. :-?

I had similar issue with using download.o.o directly. In use is openSUSE Tumbleweed. "zypper dup" wants to download more than 8.000 packages. In the time of "zypper dup" I typed 5 times (r) - retry because after 5 file requests is given up. I assume that the OP (Steven Hess) and myself don't use the same net, but the problems appeared at the same time.

The zypper.log for the questionable time: https://paste.opensuse.org/2a5847b6

Log-Time = CET / Germany Internetprovider = Vodafone (former UnityMedia) over fiberglass -- Kind regards, Sebastian - openSUSE Member (Freespacer) https://connect.opensuse.org/pg/profile/Freespacer Important notes on openSUSE Mailing List: https://en.opensuse.org/openSUSE:Mailing_list_netiquette

Am 16.03.21 um 02:59 schrieb Carlos E. R.:

It seems "simply" that server was too busy.

I hope the underlying problem is solved now - since March 4th mirrorbrain was not scanning mirrors, so all new files since were served by the same server. That affected obviously Tumbleweed, but also maintenance updates released since.

Greetings, Stephan

On Mon 2021-03-15, medwinz wrote:

...

A major update of TW will cause some delays in mirrors getting updated.

As someone who maintain 2 mirrors for ID/AS, I can verify this. 15.3 Beta release and the last 2 TW snapshot was so closed. It makes mirrors way too busy to send and receive the traffic.

Thank you for sharing this, Edwin!

In the weekly release engineering meeting I just suggested whether we could look into spacing Leap Betas/RCs and larger Tumbleweed snapshots farther apart, and Lubos (Leap) kindly indicated to have a chat with Dominique (Tumbleweed).

Gerald

On Wed, Mar 17, 2021 at 5:05 PM Gerald Pfeifer gp@suse.com wrote:

On Mon 2021-03-15, medwinz wrote:

...

...

A major update of TW will cause some delays in mirrors getting updated.

As someone who maintain 2 mirrors for ID/AS, I can verify this. 15.3 Beta release and the last 2 TW snapshot was so closed. It makes mirrors way too busy to send and receive the traffic.

Thank you for sharing this, Edwin!

In the weekly release engineering meeting I just suggested whether we could look into spacing Leap Betas/RCs and larger Tumbleweed snapshots farther apart, and Lubos (Leap) kindly indicated to have a chat with Dominique (Tumbleweed).

Gerald

Thank you Gerald.

My logic is simple, mirrors cannot catch up with the new release if several major updates happen at nearly the same time. They will be late. Many mirrors provide repositories not only for openSUSE but also for other linux distributions. This will overburden the main update repository and probably users will complain about time-out or delay in update and dup process as well. Complaining about the update process is not good for the distribution.

I read the meeting minutes, thanks for the hard work of Lubos and Dominique (Is Max also still involved in TW?). Anyway, the green vaccine seems to work quite well on them!

-- Edwin

On 30/03/2021 11.08, Gerald Pfeifer wrote:

On Thu 2021-03-18, medwinz wrote:

...

My logic is simple, mirrors cannot catch up with the new release if several major updates happen at nearly the same time.

That, plus the timing of announcements also makes a difference.

Not everyone here probably follows the factory@ list closely (I, too, read project@ more regularly), so allow me to share a change Bernhard has recently made which should also help.

Thank you, Bernhard!

For the record, here is what I found during debugging:

After the previous full rebuilds, download.opensuse.org IPv6 address had ~50% packet loss for several hours. My nagios[1] has reported this at 2021-02-16 12:20 UTC (20210212 glibc rebuild published) 2021-03-08 11:25 UTC (20210307 glibc rebuild published) 2021-03-12 18:35 UTC (20210311 KDE update published) 2021-03-13 10:37 UTC (still fallout from 20210311?) 2021-03-13 18:09 UTC (still fallout from 20210311?)

On the download.o.o VM, top showed ksoftirqd very busy at 98% CPU and several active rsyncd processes using the CPU.

This indicates that syncing to the mirrors was still ongoing and that probably also slowed mirrorbrain mirror scans, so that more traffic was directed towards the downloadcontent.o.o site which happens to be the same machine as mirrorbrain (just another IP). All that syncing, scanning and downloading slowed each other down.

Apart from the longer publish delay for large Tumbleweed snapshots, I plan to push the setup of the new widehat.o.o machine, so that it can act as a fallback mirror most of the time and relieve downloadcontent.o.o bandwidth.

Ciao Bernhard M.

[1] http://nagios.zq1.de/nagios4 => http://nagios.zq1.de/cgi-bin/nagios4/extinfo.cgi?type=1&host=oodownload

==== forwarded message ==== From: Bernhard M. Wiedemann bernhardout@lsmod.de To: factory@lists.opensuse.org Date: Sun, 28 Mar 2021 15:09:28 +0200 Subject: factory publish delay [was New Tumbleweed snapshot 20210325]

20210325 was the first large snapshot to trigger my new logic for a longer publish delay (6h instead of 2h).

The publisher became active when the repo age was 22203 s and the repo rsync log had 32114 lines and 2091370 byte.

This time, download.o.o seems to have had a much better time handling the load than with previous full rebuilds. Though this seems to have been only a partial rebuild because there are 62802 entries in

factory/repo

Medium-term we can do more fine-tuning to balance publish delay vs load of download.o.o

Ciao Bernhard M.