[opensuse-project] a broad appeal to fix opensuse's package-management's repository/redirector failure & recovery mechasnism
In my experience, package management's routine failure to find/use a 'healthy' repository is a long-standing production problem. I'm requesting a fix. Here's a summary: I run Opensuse 13.1 on numerous machines lsb_release -rd Description: openSUSE 13.1 (Bottle) (x86_64) Release: 13.1 Current count is ~200. The machines are installed at multiple locations around the globe. They're connected to the 'net via a variety of different networks providers. Some of the machines are directly connected to the 'net, some are behind LAN routers, switches & firewalls. Package management for all of the machines is handled exclusively via zypper cli. Each machine has a common core of repositories defined in /etc/zypp/repos.d, and frequently has a number of additional @openSUSE dev (!'home') repos defined. In ALL cases, the default install of repos sets have been installed with the meta-director as baseurl, baseurl=http://download.opensuse.org/... Regular package maintenance consists of zypper clean --all zypper (d)up The maintenance frequency is nominally 1/wk, often 1/dy, and in devs' cases, often more frequent. In virtually ALL cases, the update process regularly fails @ retrieving/refreshing the repos' (meta)data. For example, a typical result is: ... Checking whether to refresh metadata for KDE4-Extra-Unstable Retrieving: repomd.xml .......................................................................................[error] File '/repodata/repomd.xml' not found on medium 'http://download.opensuse.org/repositories/KDE:/Unstable:/Extra/KDE_Current_o...' Abort, retry, ignore? [a/r/i/? shows all options] (a): ... This occurs occassionally for any/all repos, whether the standard distribution repos (security, update, etc), core DM (e.g. KDE*) additional repos, or the more 'esoteric' !home OBS-hosted repos (e.g., security:netfilter). The failure rate for overall update/upgrade process attempts is, very roughly, ~15%. The error is NON-recoverable. 'Abort' & 'retry' *never* work. Chats @ IRC re: the issue typically result in the same '(non)responses' : "wait", "works for me", "prove it", etc. The ONLY solution(s) that work are: (1) wait some random amount of time -- typically hours, occassionally days -- until the system magically heals itself, (2) visit the download.opensuse.org link for the repo, click 'details' for a target page, identify a specific working/available repo for the package(s) of interest, and manually edit baseurl= for the problematic repo. Neither is tenable for a reliable operating environment. It is simply unmanageable in either a single, local or widely-distributed environment. (2) is further confounded by the fact that, at any given time, a previously-working, manually-selected repo may, itself, fail, requiring -- yet again -- another manual intervention. Within the scope of our environment, no other distro's package management system has anywhere near the failure rate demonstrated here. (We've ~600+ other machines running a mix of Centos, Fedora, Debian & Ubuntu). This has been occurring for literally years, across multiple openSUSE versions, and remains unaddressed. I know, without any doubt, that others experience similar/frequent failures -- it's been a frequent discussion with our partners, as well as in openSUSE* IRC channels. This needs a fix. As to what, specifically, that fix can/should be -- I'm unclear. If a solution already exists, I'm unaware. One idea -- a fallback mechanism *within* a repos' definition would be useful For example, allow in a given repo's def'n, having multiple, numbered baseurls baseurl1=http://direct/url/to/specific/site/1/... baseurl2=http://direct/url/to/specific/site/2/... baseurl3=http://download.opensuse.org/... ... baseurlN=http://direct/url/to/specific/site/3/... and add fuction to zypper so that for each repo, the baseurls would be tried in order for any given failure. By adding, e.g., a failcount2abort=X to either/both a given repo's defn, or /etc/zypp(er).conf, the overall process could be terminated if there were "X" # of subsequent fails, indicating a likely systemic problem requiring further intervention. I'd appreciate hearing from "those responsible for keeping the redirector & repos working" re: * acknowledgement, or refusal thereof, of the failure issue * clarification as to why it occurs in the first place * ideas/suggestions as to what can/should be done to fix it Thanks. Grant -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On 2014-10-01 16:10, grantksupport@operamail.com wrote:
The failure rate for overall update/upgrade process attempts is, very roughly, ~15%.
Well, I certainly have far less machines to update, but almost never have such problems as you do. You could try "export ZYPP_ARIA2C=1". Might do the trick for you. I have not used it recently, though.
Chats @ IRC re: the issue typically result in the same '(non)responses' : "wait", "works for me", "prove it", etc.
Why don't you try the forums and mail lists? We are a different crowd. For example, I never use IRC. I would like to see logs from your failed attempts, but not in this mail list (project). It might explain what happens.
One idea -- a fallback mechanism *within* a repos' definition would be useful
I did suggest that idea this week, on the factory mail list. Got no response. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Elessar)) -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On Wed, Oct 1, 2014, at 08:42 AM, Carlos E. R. wrote:
You could try "export ZYPP_ARIA2C=1". Might do the trick for you. I have not used it recently, though.
All machines already have env | grep ARIA ZYPP_ARIA2C=1
Why don't you try the forums and mail lists? We are a different crowd. For example, I never use IRC.
This email WAS sent to opensuse@opensuse.org, opensuse-buildservice@opensuse.org, opensuse-project@opensuse.org, zypp-devel@opensuse.org, & opensuse-softwaremgmt@opensuse.org. In general, the reason for this "broad appeal" email to multiple lists is that there already are too many channels, and it's completely unclear who is actually responsible for such a topic/fix. Especially when one asks in IRC and gets "It's not my job" and "I don't know whose it its" from @suse personnel. It's practically impossible to randomly hit the right target with any precision; hence the shotgun approach. Specfically, my experiece @ Forums is that *suse devs/ops/admins do not frequent it, and that once an issue gets 'difficult' or 'advanced' the typical advice is to try IRC or a specific mailing list.
I would like to see logs from your failed attempts, but not in this mail list (project). It might explain what happens.
Certainly. Which specific logs, which debug level is helpful, and which list is appropriate?
One idea -- a fallback mechanism *within* a repos' definition would be useful
I did suggest that idea this week, on the factory mail list. Got no response.
Not surprising. As I said, I've been dancing with this issue for years -- obviously, still unresolved. -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On 2014-10-01 17:57, grantksupport@operamail.com wrote:
On Wed, Oct 1, 2014, at 08:42 AM, Carlos E. R. wrote:
You could try "export ZYPP_ARIA2C=1". Might do the trick for you. I have not used it recently, though.
All machines already have
env | grep ARIA ZYPP_ARIA2C=1
Well, then try removing it :-) It is not a default setting. It was an experimental one some time ago.
Why don't you try the forums and mail lists? We are a different crowd. For example, I never use IRC.
This email WAS sent to opensuse@opensuse.org, opensuse-buildservice@opensuse.org, opensuse-project@opensuse.org, zypp-devel@opensuse.org, & opensuse-softwaremgmt@opensuse.org.
I know. But before this, you did not; apparently you have been trying on IRC for a long time.
Specfically, my experiece @ Forums is that *suse devs/ops/admins do not frequent it, and that once an issue gets 'difficult' or 'advanced' the typical advice is to try IRC or a specific mailing list.
However, the openSUSE forums are populated by a very helpful crowd. Employees and devs are scarce, but admins types, yes, many. And the first issue is to find whether it is a problem with your systems, or generalized. And as far as I know, it is not generalized, which is why devs will not consider changes yet.
I would like to see logs from your failed attempts, but not in this mail list (project). It might explain what happens.
Certainly. Which specific logs, which debug level is helpful, and which list is appropriate?
opensuse@list The log of interest is "/var/log/YaST2/y2log". The region with failed zypper sessions. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Elessar)) -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On Wednesday 01 October 2014 20.21:04 Carlos E. R. wrote:
On 2014-10-01 17:57, grantksupport@operamail.com wrote:
On Wed, Oct 1, 2014, at 08:42 AM, Carlos E. R. wrote:
You could try "export ZYPP_ARIA2C=1". Might do the trick for you. I have not used it recently, though.
All machines already have
env | grep ARIA ZYPP_ARIA2C=1
Well, then try removing it :-)
It is not a default setting. It was an experimental one some time ago.
Why don't you try the forums and mail lists? We are a different crowd. For example, I never use IRC.
This email WAS sent to opensuse@opensuse.org, opensuse-buildservice@opensuse.org, opensuse-project@opensuse.org, zypp-devel@opensuse.org, & opensuse-softwaremgmt@opensuse.org.
I know. But before this, you did not; apparently you have been trying on IRC for a long time.
Specfically, my experiece @ Forums is that *suse devs/ops/admins do not frequent it, and that once an issue gets 'difficult' or 'advanced' the typical advice is to try IRC or a specific mailing list.
However, the openSUSE forums are populated by a very helpful crowd. Employees and devs are scarce, but admins types, yes, many.
And the first issue is to find whether it is a problem with your systems, or generalized. And as far as I know, it is not generalized, which is why devs will not consider changes yet.
I would like to see logs from your failed attempts, but not in this mail list (project). It might explain what happens.
Certainly. Which specific logs, which debug level is helpful, and which list is appropriate?
opensuse@list
The log of interest is "/var/log/YaST2/y2log". The region with failed zypper sessions.
Can't it be reported in a bug, filled with full of details as explained in the wiki how to report a bug give then here a bug report number. If it works for everybody else, then I would also dig, why your network has such kind of trouble too. -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch openSUSE Member & Board, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On 2014-10-01 20:52, Bruno Friedmann wrote:
On Wednesday 01 October 2014 20.21:04 Carlos E. R. wrote:
Can't it be reported in a bug, filled with full of details as explained in the wiki how to report a bug give then here a bug report number.
Yes. That's the official way to report something to the people that "do it" :-) But before that:
If it works for everybody else, then I would also dig, why your network has such kind of trouble too.
You have to verify that it is a problem with the servers or the distribution, and not a problem with your network, hardware, or how you configure it all. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Elessar)) -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On Wednesday 01 October 2014 16:10:32 grantksupport@operamail.com wrote:
One idea -- a fallback mechanism *within* a repos' definition would be useful
For example, allow in a given repo's def'n, having multiple, numbered baseurls
baseurl1=http://direct/url/to/specific/site/1/... baseurl2=http://direct/url/to/specific/site/2/... baseurl3=http://download.opensuse.org/... ... baseurlN=http://direct/url/to/specific/site/3/...
and add fuction to zypper so that for each repo, the baseurls would be tried in order for any given failure.
By adding, e.g., a
failcount2abort=X
to either/both a given repo's defn, or /etc/zypp(er).conf, the overall process could be terminated if there were "X" # of subsequent fails, indicating a likely systemic problem requiring further intervention.
This should not be too hard to achieve. Since libzypp-8.8.0 (openSUSE 11.4) using a 'mirrorlist' url instead of 'baseurl' within the .repo file is already supported: #baseurl=http://direct/url/to/specific/site mirrorlist=url://server/path/to/mirrorlist.file Defining multiple URLs for a repo this way is possible. I can't remember any feedback related to mirrorlist, so this feature either works or isn't used. Probably the later, as a quick check reveals that a local file can't be used as mirrorlist (file:/localpath/to/mirrorlist.file) and zypper does not switch non-interactively between the URLs on error. I filed a bugreport to track this. [https://bugzilla.suse.com/show_bug.cgi?id=899510] -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres SUSE LINUX Products GmbH, Development, ma@suse.de GF:Jeff Hawn,Jennifer Guild,Felix Imendörffer, HRB16746(AG Nürnberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2014-10-02 14:38, Michael Andres wrote:
Since libzypp-8.8.0 (openSUSE 11.4) using a 'mirrorlist' url instead of 'baseurl' within the .repo file is already supported:
#baseurl=http://direct/url/to/specific/site mirrorlist=url://server/path/to/mirrorlist.file
Defining multiple URLs for a repo this way is possible. I can't remember any feedback related to mirrorlist, so this feature either works or isn't used.
Or we simply don't know about it :-) Do you know of a link or file documenting this and other features? - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlQtnKQACgkQtTMYHG2NR9VBhACdHuOBueAVd8NflMLAL/DR2FOu 3ywAoJJ6bHS4IN/9BKNA7znMcfRQddJg =OHEb -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2014-10-02 20:42, Carlos E. R. wrote:
On 2014-10-02 14:38, Michael Andres wrote:
Since libzypp-8.8.0 (openSUSE 11.4) using a 'mirrorlist' url instead of 'baseurl' within the .repo file is already supported:
#baseurl=http://direct/url/to/specific/site mirrorlist=url://server/path/to/mirrorlist.file
Defining multiple URLs for a repo this way is possible. I can't remember any feedback related to mirrorlist, so this feature either works or isn't used.
Or we simply don't know about it :-)
Do you know of a link or file documenting this and other features?
I tried Google: "mirrorlist=url" site:opensuse.org does not find it. Nor sites suse.com, suse.de, novell.com cer@Telcontar:~> apropos mirrorlist mirrorlist: nothing appropriate. cer@Telcontar:~> apropos libzypp locks (5) - libzypp locking file zypper (8) - Command-line interface to ZYpp system management library (libzypp) cer@Telcontar:~> cer@Telcontar:~> man zypper | grep -i mirrorlist cer@Telcontar:~> So that's why nobody uses it... undocumented. Or not easy to find. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlQtoSAACgkQtTMYHG2NR9VNwQCaAwJNp5kppNN+eqKKzxXP/XVU Ix4An1RJSA00Os8PJVb7wSePWCEUKVMF =4LHP -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On Wed, 01 Oct 2014 07:10:32 -0700, grantksupport-Z02xGKUfPWYS+FvcfC7Uqw wrote:
Current count is ~200.
I thought I'd replied to this, but it seems that maybe it didn't go. Sorry if this is a duplicate reply... With that many systems (assuming they're all on a common network), it seems to me the best approach would be to mirror (with rsync) the necessary repositories locally. Heck, I do that here at home, and I only have 5 systems that I keep updated - and I do that just to save on the bandwidth (and improve update speed). Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On Thu, Oct 2, 2014, at 01:49 PM, Jim Henderson wrote:
With that many systems (assuming they're all on a common network), it seems to me the best approach would be to mirror (with rsync) the necessary repositories locally.
That's an option and a choice -- and a workaround of the issue at hand. Whether 1 or 1000 systems are involved, zypper's failure/fallback behavior should be robust. -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On Thu, 02 Oct 2014 13:53:41 -0700, grantksupport-Z02xGKUfPWYS+FvcfC7Uqw wrote:
On Thu, Oct 2, 2014, at 01:49 PM, Jim Henderson wrote:
With that many systems (assuming they're all on a common network), it seems to me the best approach would be to mirror (with rsync) the necessary repositories locally.
That's an option and a choice -- and a workaround of the issue at hand.
Whether 1 or 1000 systems are involved, zypper's failure/fallback behavior should be robust.
Sure, it should be, but at the same time, if the problem is being caused by something upstream from you, having a local repo solves the problem, and is probably the more common solution that those managing large numbers of systems use. I remember from my own days working in IT that we'd test patches from a vendor rather than let an automatic update push them out (or pull them in, depending on the platform and technology used), just in case a vendor- supplied patch broke some third party app in an unexpected way. The only way to effectively manage that was to have our own patch distribution system/system management tool separate from the OS' own tool. Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
No in principle disagreement. Just not the issue at hand. Re: upstream, hard-pressed to believe that the same problem occurs in multiple locations / providers around the globe, and that it's simply my network end of things. In the longer-term I have alternatives tbat render this issue moot, for me. Zypper STILL needs robust fail/fallback behavior. -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
On Thu, 02 Oct 2014 15:07:28 -0700, grantksupport-Z02xGKUfPWYS+FvcfC7Uqw wrote:
No in principle disagreement. Just not the issue at hand.
Re: upstream, hard-pressed to believe that the same problem occurs in multiple locations / providers around the globe, and that it's simply my network end of things.
Given what you say the responses are that you've received when you've raised this in the past, though, it certainly seems like a localized issue to your setup.
In the longer-term I have alternatives tbat render this issue moot, for me.
That's good.
Zypper STILL needs robust fail/fallback behavior.
I don't disagree. Just trying to help you with your specific issue. Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org To contact the owner, email: opensuse-project+owner@opensuse.org
participants (5)
-
Bruno Friedmann -
Carlos E. R. -
grantksupport＠operamail.com -
Jim Henderson -
Michael Andres