* William Brown [03-29-22 20:34]:

...

On 30 Mar 2022, at 10:27, Ralph Schaffner wrote:

The opensuse.org mirrors seem to be out of sync. Specifically, the provo-mirror.opensuse.org server.

This is extremely common elsewhere too. Specifically in AU/NZ we were noticing every few days it was impossible to do updates due to errors with sync. It's what led to me creating my own mirroring tool. ( https://github.com/Firstyear/opensuse-proxy-cache )

...

I tried to install some of my OBS project repositories today and got an error.

neotinker@localhost:~> sudo zypper refresh Repository 'SLE-Module-Basesystem15-SP3-Pool' is up to date. Repository 'SLE-Module-Basesystem15-SP3-Updates' is up to date. Repository 'SLES15-SP3-15.3-0' is up to date. Repository 'SLE-Product-SLES15-SP3-Pool' is up to date. Repository 'SLE-Product-SLES15-SP3-Updates' is up to date. Repository 'SLE-Module-Server-Applications15-SP3-Pool' is up to date. Repository 'SLE-Module-Server-Applications15-SP3-Updates' is up to date. Retrieving repository 'home:neotinker3:ROS:External_Dependencies (SLE_15_SP3)' metadata .........................................[error] Repository 'home:neotinker3:ROS:External_Dependencies (SLE_15_SP3)' is invalid. [home_neotinker3_ROS_External_Dependencies|https://download.opensuse.org/repositories/home:/neotinker3:/ROS:/External_D...] Valid metadata not found at specified URL History: - File './repodata/66b1b42b9b7789c9ef188777265e266f28093dda83d9970c01ad5d634bc35616-primary.xml.gz' not found on medium 'https://download.opensuse.org/repositories/home:/neotinker3:/ROS:/External_D...' - Can't provide ./repodata/66b1b42b9b7789c9ef188777265e266f28093dda83d9970c01ad5d634bc35616-primary.xml.gz

Please check if the URIs defined for this repository are pointing to a valid repository. Skipping repository 'home:neotinker3:ROS:External_Dependencies (SLE_15_SP3)' because of the above error. Retrieving repository 'Test Builds (SLE_15_SP3)' metadata ......................................................................[error] Repository 'Test Builds (SLE_15_SP3)' is invalid. [home_neotinker3_ROS_External_Dependencies_Test|https://download.opensuseorg/repositories/home:/neotinker3:/ROS:/External_De...] Valid metadata not found at specified URL History: - File './repodata/aa69665681d2905630e52c543665b0dcc99589d0a4d13abb3d42178f0a7ea700-primary.xml.gz' not found on medium 'https://download.opensuse.org/repositories/home:/neotinker3:/ROS:/External_D...' - Can't provide ./repodata/aa69665681d2905630e52c543665b0dcc99589d0a4d13abb3d42178f0a7ea700-primary.xml.gz

Please check if the URIs defined for this repository are pointing to a valid repository. Skipping repository 'Test Builds (SLE_15_SP3)' because of the above error. Repository 'ROS Melodic for SLE 15.3 (SLE_15_SP3)' is up to date. Repository 'ROS Infrastructure (SLE_15_SP3)' is up to date. Some of the repositories have not been refreshed because of an error. neotinker@localhost:~>

When I went to the repository url with my browser, the files look to be present but I can only access the repomd.* files. The primary, other, and filelist xml files return 404 errors.

https://download.opensuse.org/repositories/home:/neotinker3:/ROS:/External_D...

If I click on the "details" link for one of the missing files, I can see different sources to access the file. The downloadcontent.opensuse.org link returns the file but the US mirror (provo-mirror.opensuse.org) returns 404.

https://mirrorcache.opensuse.org/repositories/home:/neotinker3:/ROS:/Externa... https://downloadcontent.opensuse.org/repositories/home:/neotinker3:/ROS:/Ext... https://provo-mirror.opensuse.org/repositories/home:/neotinker3:/ROS:/Extern...

Does anyone know what is going on with the mirrors?

It happens all the time. I don't think anyone really knows.

But this is a good reminder for me to stand up a us instance of my mirror to avoid this.

...

Neotinker

adding several mirrors to repos will mitigate. ex: [Tumbleweed.OSS] name=openSUSE-Tumbleweed OSS enabled=1 autorefresh=0 baseurl=http://ftp.ussg.iu.edu/linux/opensuse/opensuse/tumbleweed/repo/oss/boot/x86_... baseurl=https://download.opensuse.org/tumbleweed/repo/oss/ baseurl=http://download.opensuse.org/tumbleweed/repo/oss/ https://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/ http://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/ http://ftp.ussg.iu.edu/linux/opensuse/tumbleweed/repo/oss/ http://distro.ibiblio.org/opensuse/tumbleweed/repo/oss/ http://suse.mobile-central.org/tumbleweed/repo/oss/ http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/non-oss/ http://ftp5.gwdg.de/pub/opensuse/tumbleweed/repo/oss/ type=rpm-md -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc What sort of day was it? A day like all days, filled with those events that alter and illuminate our times...

On 30 Mar 2022, at 12:54, Ralph Schaffner wrote:

On Tue, Mar 29, 2022 at 8:59 PM Patrick Shanahan wrote: * William Brown [03-29-22 20:34]:

...

...

Does anyone know what is going on with the mirrors?

It happens all the time. I don't think anyone really knows.

But this is a good reminder for me to stand up a us instance of my mirror to avoid this.

...

Neotinker

adding several mirrors to repos will mitigate.

I don't mind adding mirrors manually but in theory I shouldn't have to. They are using mirrorcache. It should be trying at least 2 servers to get the data. A mirror server that is "closest" to me and the "main server" if the mirror doesn't have it. I'd like to think it keeps trying mirrors until it finds it but I guess that doesn't happen.

https://github.com/openSUSE/MirrorCache

I find the fact that the "main server" appears to also not have the files troubling.

It's not that the main servers don't have the files, zypper shows you the "main server url" when it misses the download. It's not showing you what the real url was that you tried to access.

-- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc What sort of day was it? A day like all days, filled with those events that alter and illuminate our times...

-- Sincerely, William Brown Senior Software Engineer, Identity and Access Management SUSE Labs, Australia

JFYI, the mirror admins do not read this list usually. Please use #opensuse-admin / admin@opensuse.org (IRC or mail) instead. good morning adrian On Mittwoch, 30. März 2022, 08:28:47 CEST Ralph Schaffner wrote:

...

It's not that the main servers don't have the files, zypper shows you the "main server url" when it misses the download. It's not showing you what the real url was that you tried to access.

I'm just going by what the README on the mirrorcache github says.

"Cache hit" means that MirrorCache was able to redirect to proper (the

...

closest) mirror. "Cache miss" means that MirrorCache had to redirect request to the Main Server.

Output below domonstrates a cache miss, so the download request will be

...

redirected to the Main Server (in this case download.opensuse.org): Then background jobs will collect info about the hottest misses and scan predefined mirrors for presence of these files. Further requests will be redirected to one of the mirrors that has the file:

So perhaps I'm misunderstanding or the readme is wrong but it sounds like it will pull the file from the "Main Server" if it can't find it in its cache. And it will scan mirrors to find the file for future downloads. If it is scanning the mirrors then it should see which ones have it and use one of them.

On Tue, Mar 29, 2022 at 10:42 PM William Brown wrote:

...

...

On 30 Mar 2022, at 12:54, Ralph Schaffner wrote:

On Tue, Mar 29, 2022 at 8:59 PM Patrick Shanahan wrote: * William Brown [03-29-22 20:34]:

...

...

Does anyone know what is going on with the mirrors?

It happens all the time. I don't think anyone really knows.

But this is a good reminder for me to stand up a us instance of my mirror to avoid this.

...

Neotinker

adding several mirrors to repos will mitigate.

I don't mind adding mirrors manually but in theory I shouldn't have to. They are using mirrorcache. It should be trying at least 2 servers to get the data. A mirror server that is "closest" to me and the "main server" if the mirror doesn't have it. I'd like to think it keeps trying mirrors until it finds it but I guess that doesn't happen.

https://github.com/openSUSE/MirrorCache

I find the fact that the "main server" appears to also not have the files troubling.

It's not that the main servers don't have the files, zypper shows you the "main server url" when it misses the download. It's not showing you what the real url was that you tried to access.

...

-- (paka)Patrick Shanahan Plainfield, Indiana, USA

@ptilopteri

...

http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc What sort of day was it? A day like all days, filled with those events that alter and illuminate our times...

-- Sincerely,

William Brown

Senior Software Engineer, Identity and Access Management SUSE Labs, Australia

-- Adrian Schroeter Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev

But that rescan didn't happen for some reasons, I must have a look and will report back later.

The problem was affecting around 80 folders, which were ignored by mirror scan scheduler (only North America mirrors were affected). It should be fixed now, if you see any problems like this - please report them as mentioned above. The simplest way is to use some the irc chat, but others should work as well.