mirroring of http://download.opensuse.org/update/leap/15.3/backports still not working
Hi, about a month ago Marcus Meissner reported to this list that http://download.opensuse.org/update/leap/15.3/backports/ was not mirrored. As of today, the mirroring is still not happening. Example: https://ftp5.gwdg.de/pub/opensuse/update/leap/15.3/backports/ https://mirrors.edge.kernel.org/opensuse/update/leap/15.3/backports/ are empty. Regards, Giacomo
On Wed, Jul 14, 2021 at 3:23 PM Giacomo Comes
Hi, about a month ago Marcus Meissner reported to this list that http://download.opensuse.org/update/leap/15.3/backports/ was not mirrored. As of today, the mirroring is still not happening. Example: https://ftp5.gwdg.de/pub/opensuse/update/leap/15.3/backports/ https://mirrors.edge.kernel.org/opensuse/update/leap/15.3/backports/ are empty.
This is affecting me and many others as well when trying to build packages for openSUSE Leap 15.3. There's an issue filed in Mock about the empty repo breaking things: https://github.com/rpm-software-management/mock/issues/750 Can someone *please* look into fixing this? -- *Neal Gompa* Senior DevOps Engineer Datto, Inc. www.datto.com
On Wed, Jul 14, 2021 at 03:32:15PM -0400, 'Neal Gompa' via Systems Administrators wrote:
Example: https://ftp5.gwdg.de/pub/opensuse/update/leap/15.3/backports/ https://mirrors.edge.kernel.org/opensuse/update/leap/15.3/backports/ are empty.
FYI, at least on the kernel.org side this is what we're getting from upstream: $ rsync --no-motd --list-only rsync://stage.opensuse.org/opensuse-full-with-factory/opensuse/update/leap/15.3/backports/ drwxr-xr-x 10 2021/01/27 10:45:09 . I'm guessing permissions on that dir don't let rsyncd chdir into it. -K
I agree that there is a problem, but not sure which exact one.
E.g. the github issue is about libgcrypt20-32bit-1.8.2-8.39.1.x86_64.rpm and I see plenty of mirrors are having that file:
https://download.opensuse.org/update/leap/15.3/sle/sle-sp1/x86_64/libgcrypt2...
The backports folder /update/leap/15.3/backports/ is empty in the location from where the mirrors sync, so most probably the files were just deleted, so removed from the mirrors as well.
I don't know whether it was on purpose or by mistake. Maybe you can point me to place in documentation or something similar which says that files must be there?
Regards,
--
Andrii Nikitin
Andrii is there anything I can do? shouldn't we take this to mirror@opensuse.org?
Lubos
On Thu, 2021-07-15 at 08:36 +0000, Andrii Nikitin wrote:
I agree that there is a problem, but not sure which exact one.
E.g. the github issue is about libgcrypt20-32bit-1.8.2-8.39.1.x86_64.rpm and I see plenty of mirrors are having that file:
https://download.opensuse.org/update/leap/15.3/sle/sle-sp1/x86_64/libgcrypt2...
The backports folder /update/leap/15.3/backports/ is empty in the location from where the mirrors sync, so most probably the files were just deleted, so removed from the mirrors as well.
I don't know whether it was on purpose or by mistake. Maybe you can point me to place in documentation or something similar which says that files must be there?
Regards,
--
Andrii Nikitin
Andrii is there anything I can do? shouldn't we take this to mirror@opensuse.org?
mirror@opensuse.org is in CC, but the problem is not with mirrors, but with the folder on ftp-stage, which is empty.
I think the first step is to acknowledge whether /update/leap/15.3/backports/ should have any files, then probably think why they were removed, then probably how to publish them again.
But I am no big help in either of these. (I also wrote you in the chat as it should be faster).
Regards,
--
Andrii Nikitin
On Thu, Jul 15, 2021 at 5:12 AM Andrii Nikitin
Andrii is there anything I can do? shouldn't we take this to mirror@opensuse.org?
mirror@opensuse.org is in CC, but the problem is not with mirrors, but with the folder on ftp-stage, which is empty. I think the first step is to acknowledge whether /update/leap/15.3/backports/ should have any files, then probably think why they were removed, then probably how to publish them again. But I am no big help in either of these. (I also wrote you in the chat as it should be faster).
Yes, it's supposed to have content. There was a ton of confusion at the last second on the layout for openSUSE Leap 15.3 due to lack of communication by the Build Service team, the SLE PM team, and the openSUSE Release Engineering teams. One such bug about this: https://bugzilla.opensuse.org/show_bug.cgi?id=1186593 There are a few other similar bugs floating around in the openSUSE Bugzilla. -- *Neal Gompa* Senior DevOps Engineer Datto, Inc. www.datto.com
Neal, On Thu, 15 Jul 2021 05:34:33 -0400 Neal Gompa wrote:
There was a ton of confusion at the last second on the layout for openSUSE Leap 15.3 due to lack of communication by the Build Service team,
---- As I am part of the Build Service team, I can not do more than to completely disagree here. While my team is running the Build Service and develops the needed tools around it, we are - in no case - responsible for the content that is created by our customers. I guess you would not complain against your car company, if you expect a parcel to get delivered and the postman did not find your house, right? ;-) OBS delivered - as usual. If we don't do this, you are free to claim this to me directly - even in public. But my team is too often used as excuse, because other people don't do their jobs. I have to step up here to protect them. ---- I already suggested to get rid of this historic ftp-stage directory and all the overlays and symlinks and scripts and ...(code archaeology anyone?) in the backend of download.opensuse.org. But neither I nor my team is officially responsible for this machine. Instead, to me it looks like nobody (neither ReleaseEng, who did the job before, nor the openSUSE heroes, nor anyone else) wants to touch anything on this machine any longer. Maybe it's time to create a successor download.o.o from scratch to get rid of all the inherited burden from the last 15 years? - I vote for this, but I think we should clarify first who takes over the responsibility of this openSUSE core service in the future. I have two sayings for the current situation: * too many cooks spoil the broth * responsibility has to be clear - and can not be shared with kind regards, Lars
Lars Vogdt wrote:
I already suggested to get rid of this historic ftp-stage directory and all the overlays and symlinks and scripts and ...(code archaeology anyone?) in the backend of download.opensuse.org.
I think I already voted yes to that one, but I am not about to take the lead.
But neither I nor my team is officially responsible for this machine. Instead, to me it looks like nobody (neither ReleaseEng, who did the job before, nor the openSUSE heroes, nor anyone else) wants to touch anything on this machine any longer.
I feel like I am fiddling with various MB sh.. stuff all the time, but it is firefighting at most. I do not have the time nor the inclination to do much more than keeping it running. Which it actually does fairly well, never mind $SUBJ.
Maybe it's time to create a successor download.o.o from scratch to get rid of all the inherited burden from the last 15 years? - I vote for this, but I think we should clarify first who takes over the responsibility of this openSUSE core service in the future.
I vote no. Rebuilding from scratch is overkill and I don't know that we have the resources. To build something more sustainable than the current setup needs a bigger team. Also, personally, I am not keen on re-learning the same things again, just with a different flavour.
I have two sayings for the current situation: * too many cooks spoil the broth * responsibility has to be clear - and can not be shared
+1 -- Per Jessen, Zürich (18.8°C)
Andrii Nikitin wrote:
Andrii is there anything I can do? shouldn't we take this to mirror@opensuse.org?
mirror@opensuse.org is in CC, but the problem is not with mirrors, but with the folder on ftp-stage, which is empty.
Isn't this simply dealt with by hardlinking /srv/ftp/pub/opensuse/update/leap/15.3/backports as /srv/ftp-stage/pub/opensuse/update/leap/15.3/backports However, trying to do that fails with "invalid cross-device link" which I don't really get. -- Per Jessen, Zürich (18.0°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland.
On Thu, Jul 15, 2021 at 08:27:07PM +0200, Per Jessen wrote:
Maybe it's time to create a successor download.o.o from scratch to get rid of all the inherited burden from the last 15 years? - I vote for this, but I think we should clarify first who takes over the responsibility of this openSUSE core service in the future.
I vote no. Rebuilding from scratch is overkill and I don't know that we have the resources. To build something more sustainable than the current setup needs a bigger team.
I'd go further and suggest that the current mirrors framework is extremely inefficient and major distros like OpenSUSE need to stop relying on it: 1. distros have grown to millions of files, but we're still rsync'ing things around just like it's 1999. It creates crazy IO churn on both the primary server and the replica. 2. every distro implements their own mechanism to monitor which mirrors are out of date and which aren't. This, too, creates crazy churn and wasted bandwidth. Major commercially backed distros should drive innovation and stop relying on the framework of free volunteer replicas. It made sense 20 years ago when Linux was young and new and distros had no infrastructure budgets. However, these days entities like Red Hat, SUSE and Canonical operate with budgets worth {b,m}illions of dollars and are very much capable of using major CDN providers to set up their own network of efficiently synchronizing replicas. -K
participants (7)
-
Andrii Nikitin -
Giacomo Comes -
Konstantin Ryabitsev -
Lars Vogdt -
Lubos Kocman -
Neal Gompa -
Per Jessen