mirror problem - previous release already gone, link dead
If you go to http://download.opensuse.org/tumbleweed/repo/oss/repodata in your browser, then see the file: 882df121b85d85b1124d6e88478132807ab2587034b90b5e1591ceb76ab057e3-filelists.xml.gz and click on it, I got a "not found": Object not found! The requested URL was not found on this server. If you entered the URL manually please check your spelling and try again. If you think this is a server error, please contact the webmaster. Error 404 / mirror.linux-ia64.org / Apache ---- Is there some way to retrieve content from D.OS.O and not a mirror when it seems the mirrors are not so reliable? Unfortunately, I seem to be stuck on this bad mirror for the time being -- it will eventually go away, but until my mirror is updated, that file is no longer available. How useful are the history links for previous releases, if a meta file from a few days ago is already gone? There really needs to be some way for d.os.org to verify a mirror having specific content before blindly referring a user off to a dead location. Isn't this possible?
On Mon, Sep 20, 2021 at 2:59 PM L A Walsh <suse@tlinx.org> wrote:
rsync? https://en.opensuse.org/openSUSE:Mirror_infrastructure#Rsync_servers
On 20/09/2021 13.57, L A Walsh wrote:
There really needs to be some way for d.os.org to verify a mirror having specific content before blindly referring a user off to a dead location.
This is done, but not real time. -- Cheers / Saludos, Carlos E. R. (from oS Leap 15.2 x86_64 (Minas Tirith))
On 20/09/2021 13.57, L A Walsh wrote: ...
Use boombatower instead. [1] https://lists.opensuse.org/opensuse-factory/2017-11/msg00591.html [2] http://review.tumbleweed.boombatower.com/ [3] https://github.com/boombatower/tumbleweed-review [4] http://review.tumbleweed.boombatower.com/data.html [5] http://release-tools.opensuse.org/2018/02/09/w05-06.html [1] http://release-tools.opensuse.org/2018/03/09/w09-10.html [2] https://metrics.opensuse.org/d/osrt_release/osrt-release Amazingly, there is no boombatower wiki page. But there is http://boombatower.com/ -- Cheers / Saludos, Carlos E. R. (from oS Leap 15.2 x86_64 (Minas Tirith))
Hi, On Mon, 2021-09-20 at 04:57 -0700, L A Walsh wrote:
Is there some way to retrieve content from D.OS.O and not a mirror when it seems the mirrors are not so reliable?
There is, but you will have to use HTTPS to benefit from it. (http traffic is handled by MirrorBrain, while https by MirrorCache - mirrorcache.opensuse.org) MirrorCache will default to downloadcontent.opensuse.org if there is no available mirror for the requested file. Regards, Eli -- Elisei Roca <eroca@suse.de>
On 2021/09/20 05:33, Elisei Roca wrote:
Ug, that's evil. The reason I stick with http is that its about 20x** faster for an 'up-to-date' check. I.e. not downloading any file, but just checking that your current copy if up-to-date. ** talking about <100ms for HTTP, vs. over 2-3 seconds/request on HTTPS. HTTPS _may_ not be that much slower when you get to content downloading, BUT it's horrible for most of the web, that is based on 20-30 small requests to different servers for tiny files!
Which country are you trying from, could you share example curl output, e.g. time curl -IL https://download.opensuse.org/tumbleweed/repo/oss/x86_64/antimicro-2.23-2.1....
But why do you need that 882df... ? Current repomd.xml doesn't refer it, so `zypper dup` shouldn't try to access that file anymore. It might be temporary problem or maybe you need `zypper ref`? (I've tried `zypper dup` today and it went smoothly, just I use https:// which is handled by different backend behind d.o.o)
In case if you don't want to do `zypper dup` - you may find old snapshot in /history , e.g. http://download.opensuse.org/history/20210916/tumbleweed/repo/oss/ There are some plans to find automatically files from /history when you try to install something without refreshing repos, but nothing particular to promise yet.
On 2021/09/20 05:54, Andrii Nikitin wrote:
I haven't figured out what I want to do with history yet. Working on my own mirroring script that was trying to keep around the past 3-4 downloads. Most immediately I was trying to put together a way for me to tag rpms that belong to one of those 'recent' 3-4 downloads so I can get rid of the the older rpms. Usually I try to d/l the rpm's to local store and install from there.
Do you mean full snapshots or only those rpms which you use? I'd try to keep 3-4 old *-primary.xml files (you can even use http://download.opensuse.org/history/ for that), then all rpms that are mentioned there belong to recent snapshots. (Btw if you plan to parse folders inside history - you can parse json instead of html at https://mirrorcache.opensuse.org/history/?json ). Or even when you get 3-4 entries from history, you can fetch several huge json strings and look inside them for presence of the rpms: https://mirrorcache.opensuse.org/history/20210915/tumbleweed/repo/oss/x86_64...
L A Walsh wrote:
If you go to http://download.opensuse.org/tumbleweed/repo/oss/repodata in your browser, then see the file:
882df121b85d85b1124d6e88478132807ab2587034b90b5e1591ceb76ab057e3-filelists.xml.gz
That file is dated 15 September, and is no longer carried by any mirror.
Unfortunately, I seem to be stuck on this bad mirror for the time being
mirrorbrain lists three mirrors - in Russia, Iran and China. It is possible they were not yet scanned, dunno.
Yup it is and that is what we do too. All mirrors are continually scanned. -- Per Jessen, Zürich (16.4°C) Member, openSUSE Heroes
On 9/20/21 21:27, L A Walsh wrote:
A temporary workaround can be to find a good mirror near you and point to it rather then download.opensuse.org Cheers -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On 2021/09/20 17:16, Simon Lees wrote:
A temporary workaround can be to find a good mirror near you and point to it rather then download.opensuse.org
I can do that manually, but I'm trying to run scripts to cache the latest release. This time, I got stopped on a different file: .*Failure: (url=http://download.opensuse.org/tumbleweed/repo/oss/x86_64/babe-1.2.1.1-1.3.x86...), GET http://download.opensuse.org/tumbleweed/repo/oss/x86_64/babe-1.2.1.1-1.3.x86...: 404 Not Found +1.67 163㎳; 1858 (1.1K/11K) MISS/302 <Ishtar [GET http://download.opensuse.org/tumbleweed/repo/oss/x86_64/babe-1.2.1.1-1.3.x86... - 195.135.221.134 text/html] +0.04 29㎳; 279 (8.3K/9.4K) MISS/301 <Ishtar [GET http://sfo-korg-mirror.kernel.org/opensuse/tumbleweed/repo/oss/x86_64/babe-1... - 149.20.37.36 -] +0.07 62㎳; 5356 (80K/84K) TUNNEL/200 <Ishtar [CONNECT mirrors.edge.kernel.org:443 - 147.75.69.165 -] +0.18 166㎳; 1858 (11K/11K) MISS/302 <Ishtar [GET http://download.opensuse.org/tumbleweed/repo/oss/x86_64/babe-1.2.1.1-1.3.x86... - 195.135.221.134 text/html] +0.03 27㎳; 279 (9.1K/10K) MISS/301 <Ishtar [GET http://sfo-korg-mirror.kernel.org/opensuse/tumbleweed/repo/oss/x86_64/babe-1... - 149.20.37.36 -] [0923_164110.00] 61㎳; 5356 (82K/86K) TUNNEL/200 <Ishtar [CONNECT mirrors.edge.kernel.org:443 - 147.75.69.165 -] +0.22 168㎳; 1858 (8.6K/11K) MISS/302 <Ishtar [GET http://download.opensuse.org/tumbleweed/repo/oss/x86_64/babe-1.2.1.1-1.3.x86... - 195.135.221.134 text/html] +0.03 26㎳; 279 (9.1K/10K) MISS/301 <Ishtar [GET http://sfo-korg-mirror.kernel.org/opensuse/tumbleweed/repo/oss/x86_64/babe-1... - 149.20.37.36 -] It looks like it was referred to 5-6 caches before it gave up as unable to be found. I found a possible temporary workaround -- if regular http fetch fails, maybe try https. Not guaranteed to work, but at least another programatic thing I can try. Problem with this is that it changes often, but generally causes a new download to fail because the script writer believes that If they are told the file is on a mirror -- then it should be there. Otherwise, it should try to refetch from the original. Someone said that they though that's what happened but only with an http fetch. Is there a reason why something like that would be limited to https?
On Mon, Sep 20, 2021 at 2:59 PM L A Walsh <suse@tlinx.org> wrote:
rsync? https://en.opensuse.org/openSUSE:Mirror_infrastructure#Rsync_servers
On 20/09/2021 13.57, L A Walsh wrote:
There really needs to be some way for d.os.org to verify a mirror having specific content before blindly referring a user off to a dead location.
This is done, but not real time. -- Cheers / Saludos, Carlos E. R. (from oS Leap 15.2 x86_64 (Minas Tirith))
On 20/09/2021 13.57, L A Walsh wrote: ...
Use boombatower instead. [1] https://lists.opensuse.org/opensuse-factory/2017-11/msg00591.html [2] http://review.tumbleweed.boombatower.com/ [3] https://github.com/boombatower/tumbleweed-review [4] http://review.tumbleweed.boombatower.com/data.html [5] http://release-tools.opensuse.org/2018/02/09/w05-06.html [1] http://release-tools.opensuse.org/2018/03/09/w09-10.html [2] https://metrics.opensuse.org/d/osrt_release/osrt-release Amazingly, there is no boombatower wiki page. But there is http://boombatower.com/ -- Cheers / Saludos, Carlos E. R. (from oS Leap 15.2 x86_64 (Minas Tirith))
Hi, On Mon, 2021-09-20 at 04:57 -0700, L A Walsh wrote:
Is there some way to retrieve content from D.OS.O and not a mirror when it seems the mirrors are not so reliable?
There is, but you will have to use HTTPS to benefit from it. (http traffic is handled by MirrorBrain, while https by MirrorCache - mirrorcache.opensuse.org) MirrorCache will default to downloadcontent.opensuse.org if there is no available mirror for the requested file. Regards, Eli -- Elisei Roca <eroca@suse.de>
On 2021/09/20 05:33, Elisei Roca wrote:
Ug, that's evil. The reason I stick with http is that its about 20x** faster for an 'up-to-date' check. I.e. not downloading any file, but just checking that your current copy if up-to-date. ** talking about <100ms for HTTP, vs. over 2-3 seconds/request on HTTPS. HTTPS _may_ not be that much slower when you get to content downloading, BUT it's horrible for most of the web, that is based on 20-30 small requests to different servers for tiny files!
Which country are you trying from, could you share example curl output, e.g. time curl -IL https://download.opensuse.org/tumbleweed/repo/oss/x86_64/antimicro-2.23-2.1....
participants (7)
-
Andrei Borzenkov -
Andrii Nikitin -
Carlos E. R. -
Elisei Roca -
L A Walsh -
Per Jessen -
Simon Lees