I agree that there is a problem, but not sure which exact one. E.g. the github issue is about libgcrypt20-32bit-1.8.2-8.39.1.x86_64.rpm and I see plenty of mirrors are having that file: https://download.opensuse.org/update/leap/15.3/sle/sle-sp1/x86_64/libgcrypt2... The backports folder /update/leap/15.3/backports/ is empty in the location from where the mirrors sync, so most probably the files were just deleted, so removed from the mirrors as well. I don't know whether it was on purpose or by mistake. Maybe you can point me to place in documentation or something similar which says that files must be there? Regards, -- Andrii Nikitin <andrii.nikitin@suse.com> DevOPS Automation and Build Service Engineer SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany (HRB 247165, AG München) Managing Director: Felix Imendörffer ________________________________ From: Neal Gompa <ngompa@datto.com> Sent: Wednesday, July 14, 2021 9:32 PM To: Giacomo Comes <comes@naic.edu> Cc: mirror@lists.opensuse.org <mirror@lists.opensuse.org> Subject: Re: mirroring of http://download.opensuse.org/update/leap/15.3/backports still not working On Wed, Jul 14, 2021 at 3:23 PM Giacomo Comes <comes@naic.edu<mailto:comes@naic.edu>> wrote: Hi, about a month ago Marcus Meissner reported to this list that http://download.opensuse.org/update/leap/15.3/backports/ was not mirrored. As of today, the mirroring is still not happening. Example: https://ftp5.gwdg.de/pub/opensuse/update/leap/15.3/backports/ https://mirrors.edge.kernel.org/opensuse/update/leap/15.3/backports/ are empty. This is affecting me and many others as well when trying to build packages for openSUSE Leap 15.3. There's an issue filed in Mock about the empty repo breaking things: https://github.com/rpm-software-management/mock/issues/750 Can someone please look into fixing this? -- Neal Gompa Senior DevOps Engineer Datto, Inc. www.datto.com<https://www.datto.com>

Andrii is there anything I can do? shouldn't we take this to mirror@opensuse.org?

mirror@opensuse.org is in CC, but the problem is not with mirrors, but with the folder on ftp-stage, which is empty. I think the first step is to acknowledge whether /update/leap/15.3/backports/ should have any files, then probably think why they were removed, then probably how to publish them again. But I am no big help in either of these. (I also wrote you in the chat as it should be faster). Regards, -- Andrii Nikitin <andrii.nikitin@suse.com> DevOPS Automation and Build Service Engineer SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany (HRB 247165, AG München) Managing Director: Felix Imendörffer

Neal, On Thu, 15 Jul 2021 05:34:33 -0400 Neal Gompa wrote:

There was a ton of confusion at the last second on the layout for openSUSE Leap 15.3 due to lack of communication by the Build Service team,

---- As I am part of the Build Service team, I can not do more than to completely disagree here. While my team is running the Build Service and develops the needed tools around it, we are - in no case - responsible for the content that is created by our customers. I guess you would not complain against your car company, if you expect a parcel to get delivered and the postman did not find your house, right? ;-) OBS delivered - as usual. If we don't do this, you are free to claim this to me directly - even in public. But my team is too often used as excuse, because other people don't do their jobs. I have to step up here to protect them. ---- I already suggested to get rid of this historic ftp-stage directory and all the overlays and symlinks and scripts and ...(code archaeology anyone?) in the backend of download.opensuse.org. But neither I nor my team is officially responsible for this machine. Instead, to me it looks like nobody (neither ReleaseEng, who did the job before, nor the openSUSE heroes, nor anyone else) wants to touch anything on this machine any longer. Maybe it's time to create a successor download.o.o from scratch to get rid of all the inherited burden from the last 15 years? - I vote for this, but I think we should clarify first who takes over the responsibility of this openSUSE core service in the future. I have two sayings for the current situation: * too many cooks spoil the broth * responsibility has to be clear - and can not be shared with kind regards, Lars

On Thu, Jul 15, 2021 at 08:27:07PM +0200, Per Jessen wrote:

...

Maybe it's time to create a successor download.o.o from scratch to get rid of all the inherited burden from the last 15 years? - I vote for this, but I think we should clarify first who takes over the responsibility of this openSUSE core service in the future.

I vote no. Rebuilding from scratch is overkill and I don't know that we have the resources. To build something more sustainable than the current setup needs a bigger team.

I'd go further and suggest that the current mirrors framework is extremely inefficient and major distros like OpenSUSE need to stop relying on it: 1. distros have grown to millions of files, but we're still rsync'ing things around just like it's 1999. It creates crazy IO churn on both the primary server and the replica. 2. every distro implements their own mechanism to monitor which mirrors are out of date and which aren't. This, too, creates crazy churn and wasted bandwidth. Major commercially backed distros should drive innovation and stop relying on the framework of free volunteer replicas. It made sense 20 years ago when Linux was young and new and distros had no infrastructure budgets. However, these days entities like Red Hat, SUSE and Canonical operate with budgets worth {b,m}illions of dollars and are very much capable of using major CDN providers to set up their own network of efficiently synchronizing replicas. -K