[opensuse-factory] mirror system not checking for existence
I've had to start rewriting an old-style fetch script for TW, -- grabbing the repo/<type>/<arch> index.htm and fetching the rpms. Trouble is, alot of rpm's listed on download.OS.org aren't on some mirrors. I seem to have alot of problems with this mirror seeming to always return 'not found'. http://ftp.rz.uni-wuerzburg.de/ -- anytime I get 302'd to this server, I always get a 'not found' -- like for all of libreoffice. When was the last time download.opensuse.org actually checked that any of the files were on the redirected site? I have gotten files from other sites (and some misses), but the above site seems to always get a miss. Is there a direct download address off of dl.os.org that I can fall-back to when a mirror doesn't have the binary and/or a way to blacklist a mirror, since once dl.os.org locks me onto that server, I'm screwed. Worse, I know there are mirrors within 50 miles of me that are much faster -- I really don't want to try to fetch things from an overseas site -- especially one that is so unreliable. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
L A Walsh wrote:
I've had to start rewriting an old-style fetch script for TW, -- grabbing the repo/<type>/<arch> index.htm and fetching the rpms.
Trouble is, alot of rpm's listed on download.OS.org aren't on some mirrors.
I seem to have alot of problems with this mirror seeming to always return 'not found'. http://ftp.rz.uni-wuerzburg.de/ -- anytime I get 302'd to this server, I always get a 'not found' -- like for all of libreoffice.
In general, mirror issues are probably best reported to admin@o.o. Which package were you looking for above? which url?
When was the last time download.opensuse.org actually checked that any of the files were on the redirected site?
In principle, several times a day.
Is there a direct download address off of dl.os.org that I can fall-back to when a mirror doesn't have the binary and/or a way to blacklist a mirror, since once dl.os.org locks me onto that server, I'm screwed.
Provided your country has more than one mirror for the desired package, you are never "locked" to any mirror.
Worse, I know there are mirrors within 50 miles of me that are much faster. I really don't want to try to fetch things from an overseas site -- especially one that is so unreliable.
It does seem odd that you should be given a mirror in Germany - there are plenty of TW mirrors in the US, I see at least 16 (based on the IP address of the tlinx mailserver). -- Per Jessen, Zürich (13.6°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2020/10/23 02:29, Per Jessen wrote:
In general, mirror issues are probably best reported to admin@o.o.
If that information was on every page, I might see and remember it, but the only way I've ever heard of that email address was by posting a problem here first.
Which package were you looking for above? which url?
---- All of them, but I didn't want to list all the not-found packages as it would be several pages? Not that I wanted them, but all of the libre office rpms were not found when I tried d/l'ing from d.os.org which redirected me to the mirror server, but when I went to the mirror with my browser and tried downloading a random libre-office file from there, it worked no problem! In one case, the redirect on d.os.org had a different version than the mirror did causing it to 404. More surprising was when I was downloading several files that didn't goto a mirror and got 404's -- checked them out, and they were wrong versions as well -- meaning the version I'd downloaded of the file-listing page less than an hour before became out of date before I could download the rpms on the page.
When was the last time download.opensuse.org actually checked that any of the files were on the redirected site?
In principle, several times a day.
---- It checks individual versions of each file several times a day? or just whether or not the mirror is up and has maybe some repodata files, as these mirrors are up, and contain a dir structure like they are a mirror, but the version numbers may be different -- that's specifically what I'm referring to -- how often does the master check that the specific versions it is directing people to are actually on that master? I.e. the version list I was downloading from seems to have been out of sync.
Is there a direct download address off of dl.os.org that I can fall-back to when a mirror doesn't have the binary and/or a way to blacklist a mirror, since once dl.os.org locks me onto that server, I'm screwed.
Provided your country has more than one mirror for the desired package, you are never "locked" to any mirror.
---- I was always going to the failing server in germany for various (but not random -- same packages). No matter how many times I stopped and restarted, when it got to the libre packages, it went to the ftp-uni site.
Worse, I know there are mirrors within 50 miles of me that are much faster. I really don't want to try to fetch things from an overseas site -- especially one that is so unreliable.
It does seem odd that you should be given a mirror in Germany - there are plenty of TW mirrors in the US, I see at least 16 (based on the IP address of the tlinx mailserver).
---- I know!!! -- I live <30 miles from silicon valley, several servers are there last time I did d/l's I used the server in provo which was very consistent. At that time I was going from the file lists in the repodata dir and they were pretty reliably there. Might be because some sites (like the provo site) seemed to keep more than one version of the release lists -- like maybe the last 5-7 (as grouped by date), so it *seemed* they also kept more than 1 of updated rpm's that might correspond to the 5-7 groups of repodata files. Maybe sites that can only afford to keep 1 version are more likely to experience problems with mirror indices possibly being updated faster than contents, though not sure how one would deal with that. Right now, I've broken my script again (its suffering growing pains), and haven't quite figured out why (just discovered while writing this email). So far, I'm leaning to the directory listing files on dl.os.org being out of sync -- holding more recent versions than what might be in the content directory of mirrors. I'll keep trying to narrow down the problem... -linda
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 24/10/2020 09.49, L A Walsh wrote:
On 2020/10/23 02:29, Per Jessen wrote:
In general, mirror issues are probably best reported to admin@o.o.
If that information was on every page, I might see and remember it, but the only way I've ever heard of that email address was by posting a problem here first.
There was a page with all the addresses to use for each kind of problem. <https://en.opensuse.org/openSUSE:Communication_channels> Contact Info To report a problem with this site or openSUSE infrastructure, contact admin@opensuse.org. Please note, this is not a tech support email address. Use one of the options shown above if you need tech support.
In one case, the redirect on d.os.org had a different version than the mirror did causing it to 404. More surprising was when I was downloading several files that didn't goto a mirror and got 404's -- checked them out, and they were wrong versions as well -- meaning the version I'd downloaded of the file-listing page less than an hour before became out of date before I could download the rpms on the page.
So, the metadata at the download redirector was different than the actual data at the mirror, which was probably in mid update. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
L A Walsh wrote:
On 2020/10/23 02:29, Per Jessen wrote:
In general, mirror issues are probably best reported to admin@o.o.
If that information was on every page, I might see and remember it, but the only way I've ever heard of that email address was by posting a problem here first.
admin@o.o is for general openSUSE infrastructure issues, an email will automatically open a ticket at https://progress.opensuse.org.
Which package were you looking for above? which url?
---- All of them, but I didn't want to list all the not-found packages as it would be several pages?
Just one would suffice. :-) Otherwise I have no idea what to look for nor where.
Not that I wanted them, but all of the libre office rpms were not found when I tried d/l'ing from d.os.org which redirected me to the mirror server, but when I went to the mirror with my browser and tried downloading a random libre-office file from there, it worked no problem!
If actually the same mirror, doesn't that suggest it might not be a mirror issue?
In principle, several times a day.
---- It checks individual versions of each file several times a day?
Yes, the scanner looks at all our mirrors continually, trying to figure out if they are in sync.
Provided your country has more than one mirror for the desired package, you are never "locked" to any mirror.
---- I was always going to the failing server in germany for various (but not random -- same packages). No matter how many times I stopped and restarted, when it got to the libre packages, it went to the ftp-uni site.
That's definitely not right.
It does seem odd that you should be given a mirror in Germany - there are plenty of TW mirrors in the US, I see at least 16 (based on the IP address of the tlinx mailserver).
---- I know!!! -- I live <30 miles from silicon valley, several servers are there last time I did d/l's I used the server in provo which was very consistent.
That is why it would be useful to know at least one of the packages you were trying to download. -- Per Jessen, Zürich (17.3°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 22/10/2020 20.50, L A Walsh wrote: ...
Is there a direct download address off of dl.os.org that I can fall-back to when a mirror doesn't have the binary and/or a way to blacklist a mirror, since once dl.os.org locks me onto that server, I'm screwed. Worse, I know there are mirrors within 50 miles of me that are much faster -- I really don't want to try to fetch things from an overseas site -- especially one that is so unreliable.
You could try an experiment to block that "bad" server. If you use dnsmasq: <https://lists.opensuse.org/opensuse/2020-10/msg00210.html> DH> [...] use dnsmasq where one entry suffices for a DH> whole domain and supports delivering NXDOMAIN instead of 127/8, DH> e.g.: DH> DH> address=/ftp.rz.uni-wuerzburg.de/ DH> DH> in a seperate file in /etc/dnsmasq.d/, which is included DH> from /etc/dnsmasq.conf Or, you can directly create your own list of servers instead the official one: /etc/zypp/repos.d/Example.repo [Example] name=Example Repository ... baseurl=http://.... http://.... http://.... -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On 2020/10/23 04:08, Carlos E. R. wrote:
You could try an experiment to block that "bad" server. If you use dnsmasq:....
--- That's an interesting idea...easiest to put the change in my /etc/hosts file, but same effect.
Or, you can directly create your own list of servers instead the official one:
/etc/zypp/repos.d/Example.repo
---- I have done that with other mirrored services, since a few I knew of several mirrors "over the hill" (Silicon V.)... Will have to take a look at that -- if it is already populated, I can truncate the list to local servers... Thanks for the ideas! -l -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 24/10/2020 09.54, L A Walsh wrote:
On 2020/10/23 04:08, Carlos E. R. wrote:
You could try an experiment to block that "bad" server. If you use dnsmasq:....
That's an interesting idea...easiest to put the change in my /etc/hosts file, but same effect.
As long as you use dnsmasq, because you get an error (NXDOMAIN). If you use the hosts file, the syntax is different, and you have to redirect to 127.0.0.1, which will produce a download error later.
Or, you can directly create your own list of servers instead the official one:
/etc/zypp/repos.d/Example.repo
I have done that with other mirrored services, since a few I knew of several mirrors "over the hill" (Silicon V.)... Will have to take a look at that -- if it is already populated, I can truncate the list to local servers...
You have to go to: <http://download.opensuse.org/tumbleweed/repo/oss/x86_64/> and then click on "mirrors" to get the mirror list for your site. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
participants (3)
-
Carlos E. R. -
L A Walsh -
Per Jessen