Mailinglist Archive: opensuse-features (893 mails)
| < Previous | Next > |
[openFATE 305320] Show accessed mirror(s)
- From: fate_noreply@xxxxxxx
- Date: Tue, 18 Aug 2009 17:30:42 +0200 (CEST)
- Message-id: <feature-305320-32@xxxxxxxxxxxxxx>
Feature changed by: Ján Kupec (jkupec)
Feature #305320, revision 32
Title: Show accessed mirror(s)
openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov)
reject date: 2008-10-01 15:49:39
reject reason: Too late, but let's consider for the future ones.
Priority
Requester: Mandatory
openSUSE-11.2: Done
Priority
Requester: Mandatory
Projectmanager: Mandatory
Requested by: Peter Poeml (poeml)
Description:
This concerns both YaST and zypper.
It would help a lot if they would show the mirror which is accessed
when downloading a package.
(Technically, the download client is following redirects and the
actually URL needs to be shown, not just the first URL that was
contacted but that redirected to another URL)
It is a long-standing problem that this information is lacking and it
makes debugging and optimizing much harder than necessary, because this
information is only available deeply buried in some logfile.
We need this information for two reasons:
1) Debugging - to see which URL was accessed when something goes wrong,
like a mirror timing out. For example, Whenever a mirror returns
garbage, it takes a lot of additional time to find out which one,
because all reports are lacking the critical detail, and it is usually
hard to find it out later.
2) Optimizing the load sharing between different mirrors - to
distinguish slow mirrors from fast ones. For example, I am very much in
need of information about the situation in the US, but even though
people in the US run installations for me and are willing to send me
logs, I can't get this information.
In reply to a question from Stano:
> Do you want the information in the log or to be shown in e.g. zypper
-vvv ?
Shown, in normal output. It is already in some log I believe, I don't
know which one exactly anymore though.
Thing is, users won't be able to tell us anything useful if they
weren't shown the information. And later it is not reproducible.
At least in a failure situation the URLs must be shown. That alone
wouldn't give us the chance to find out about slow mirrors, but at
least it would allow to debug problems.
Real life example #1:
If a mirror is behind a firewall that lets every 10th connection stall
(so the transfer hangs or the connection is reset), it usually takes
weeks (!) to find out about it. But it hits virtually every user from
mirrors country during the time. We have had this 4 times during the
last 18 months.
Real life example #2:
I got feedback from someone who did a network install of openSUSE 11.0
from download.opensuse.org, located in the US. He had to let it running
overnight because it took a long time. Next morning, the install hadn't
even completed. YaST was hanging and waiting for somebody to hit retry.
Problem is, it took us a lot of work to find out which mirror was at
fault. We actually had to repeat the install, I had to follow the log
and live watching his machine accessing download.opensuse.org, to see
where it was redirected, and where it spent most time (while
downloading some package). We found a mirror which was really slow, and
I could disable it. I have a suspicion though, that we have more such
mirrors, because I have got some similar reports from recently. But how
to find out? People only complain about slowness, without the ability
to give any useful detail. See
http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments
(http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments)
for examples of this.
References:
https://bugzilla.novell.com/show_bug.cgi?id=522260 - patch adding
logging of redirections for curl backend
+ Test Case:
+ For zypp on code 11 (openSUSE 11.1, SLE11):
+ Add any repository below download.opensuse.org via zypper or YaST and
+ install a package from it (the package should be served from a mirror).
+ Zypper/YaST log should contain lines with "Location:" string (grep
+ 'Location: ' /var/log/zypper.log).
+ For zypp on code 12, the redirections are not logged, but are part of
+ the error message details in case download error. See the testcase in
+ Fate #302923 for info on how to get broken mirrors from the redirector
+ in order to test this.
Discussion:
#1: Susanne Oberhauser (froh) (2008-10-01 15:16:30)
What if we focussed on something like torrent for distributing packages
instead?
AFAICT these p2p network technologies have mechanisms to recover from
missing fragments, i.e. failing servers.
In that world, todays mirrors would be strong and almost always present
peers in the network.
#2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1)
Torrents are not efficient for files of this small size.
Another concern is that, on large scale, the reasonability of P2P
traffic with regard to network topology is questionable; confer to
what's called the Application-Layer Traffic Optimization (ALTO)
problem. See http://alto.tilab.com/ (http://alto.tilab.com/) .
There are working examples where e.g. university have rolled out a P2P
update distribution campus-wide; in a controlled limited environment
this is feasible, but not across the Internet.
Further, note that libzypp in 11.1 has preliminary support for using
metalinks for downlaods. Metalink clients recover nicely from missing
fragments. See http://en.opensuse.org/Libzypp/Failover
(http://en.opensuse.org/Libzypp/Failover) .
#3: Mark Goldstein (markgo) (2008-10-04 17:20:42)
Other usage of this feature would be possibility of "blacklisting"
servers with persistent problems. (See thread in opensuse list "Is it
possible to "blacklist" specific download server in Yast?" ).
As Carlos E. R. suggested other useful option could be for YaST/zypper
to fail gracefully requesting another server to the redirector on
failure on the one it gave use previously.
#4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3)
Gracefully falling back to other mirrors is already being implemented
(currently in prototype stage) by way of
http://en.opensuse.org/Libzypp/Failover
(http://en.opensuse.org/Libzypp/Failover)
.
The feature of blacklisting a specific mirrors is included in that
proposal, and I have added it explicitely.
#5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25)
Looking at the discussion on opensuse-factory and my own experience, I
also see this as high priority for 11.2.
#6: Peter Poeml (poeml) (2009-01-27 20:39:59)
Just for the record, the way to enable logging that makes mirrors show
up in the log seems to be:
$ ZYPP_MEDIA_CURL_DEBUG=1 zypper ...
Cf. http://en.opensuse.org/Debugging_YaST
#7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6)
Yes, it is, but that's quite excessive log that we don't want to be
enabled by default. Instead we need to pick up the redirects and log
them one line per each.
Michal Marek already gave me a hint how to do this: "Use
CURLOPT_HEADERFUNCTION callback, which gets each http header and then
test for 'Location:' and log that." I'll try to do it ASAP.
#8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7)
Let's try to display this information at least with zypper. However, we
will need some investigation, since we switched to aria2c.
#9: Peter Poeml (poeml) (2009-06-24 15:04:21)
Another example of the implications. What users get right now is this
(http://www.3111skyline.com/download/openSUSE_bugs/kde4/screenshots/YastsZypperStuck-soundtouch.jpeg)
(screenshot
that shows a rather meaningless URL, but not the accessed mirror). See
this thread (http://lists.opensuse.org/opensuse/2009-06/msg01206.html)
on the English opensuse mailing list.
#10: Ján Kupec (jkupec) (2009-07-14 17:54:07) (reply to #9)
OK, this is all about the CURL backend, so we're developing an
improvment to be released as an online update for code 10 products
(openSUSE 11.1 and SLE11), right?
Peter, is this needed also for the aria2c backend? If so, can you give
me some hints how to do it with aria (i have zero knowlege of aria)?
#11: Ján Kupec (jkupec) (2009-07-14 18:53:11) (reply to #10)
above, i meant code 11, of course
#13: Peter Poeml (poeml) (2009-07-15 16:05:29) (reply to #10)
I should mention that when I initially requested this feature, we were
still developing 11.1, and the fail-safe aria2c backend wasn't
implemented yet. Now that the new fail-safe backend is becoming the
default download backend in 11.2, the need for this feature request is
a lot smaller.
It would still be beneficial for us if users get direct feedback (via
the user interface) about mirror performance, because we will still
need to maintain an overall healthy mirror system, and tune here and
there. The fail-safe backend will compensate for many or most problems.
The fail-safe backend collects its own mirror performance stats, and we
could actually think about submit those stats to a central place to
gather a high-level view on mirror performance. We could actually
correlate the performance to geographical or network topological client
location. This would allow for better mirror selection in the first
place, which will allow for a better client experience overall.
openSUSE 11.1 and 11.0 would benefit a lot from the improvement
requested here, of course. Furthermore, there might be edge cases where
the aria2c backend isn't used - I'm not sure if we do (or can) use it
from the install initial ramdisk when installing with a "mini iso", for
instance.
The curl backend of 11.2 should definitely have good logging of all
server accesses - not only in a debug mode. The information must be
easily available. Best would be if its shown directly to the user, so
they know what could be a possible problem, and know what to report.
aria2c logs exceptions in a way that we can use to enhance our logging
(and UI). Tatsuhiro added the patch for us mentioned below:
I created a patch to embed URI(redirected one) to exception message.
Since there are many exception messages, I wrapped exception with new
one whose message contains URI. For example, the exception message for
404 error is now:
Exception: URI=http://localhost/abc
-> Resource not found
#14: Ján Kupec (jkupec) (2009-07-28 23:23:06) (reply to #13)
Thanx Peter for the comment. So we have the curl logging for code 11 (
bug 522260 (https://bugzilla.novell.com/show_bug.cgi?id=522260) ) and
code 12 (and not only in debug mode) ready.
As for 11.2, looking at the MediaAria2c.cc code, the exceptions are
already parsed from the output of aria2c and passed to callbacks, thus
shown to the user, so nothing to do here (Duncan, please correct me if
i'm wrong).
Regarding curl in 11.2, i guess having the redirects logged is enough,
since aria2c is now the default backend (even though it may not be
suitable for some uses, as Peter suggests).
All in all, i'd consider this feature done. Please reopen if i missed
something.
#12: Ján Kupec (jkupec) (2009-07-15 14:16:05)
Alright, logging is easy. If a Location: line will be found in reponse
header, it will be logged like: "MediaCurl.cc(log_redirects_curl):120
redirecting to Location: URL" in zypper.log. I have the patch ready and
it is pretty simple and can be released as update for openSUSE 11.x and
SLE11.
With showing the redirects in the UIs it is quite complicated since it
requires MediaException API changes and adding the list of redirects to
each MediaException thrown. Nothing for the released products i'd say.
Can we agree on this? Let me empasize that this is for the cURL backend
used in code11 products. For openSUSE 11.2 and later (using aria2c
backend) i'm still waiting for info asked in c#10.
--
openSUSE Feature:
https://features.opensuse.org/305320
Feature #305320, revision 32
Title: Show accessed mirror(s)
openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov)
reject date: 2008-10-01 15:49:39
reject reason: Too late, but let's consider for the future ones.
Priority
Requester: Mandatory
openSUSE-11.2: Done
Priority
Requester: Mandatory
Projectmanager: Mandatory
Requested by: Peter Poeml (poeml)
Description:
This concerns both YaST and zypper.
It would help a lot if they would show the mirror which is accessed
when downloading a package.
(Technically, the download client is following redirects and the
actually URL needs to be shown, not just the first URL that was
contacted but that redirected to another URL)
It is a long-standing problem that this information is lacking and it
makes debugging and optimizing much harder than necessary, because this
information is only available deeply buried in some logfile.
We need this information for two reasons:
1) Debugging - to see which URL was accessed when something goes wrong,
like a mirror timing out. For example, Whenever a mirror returns
garbage, it takes a lot of additional time to find out which one,
because all reports are lacking the critical detail, and it is usually
hard to find it out later.
2) Optimizing the load sharing between different mirrors - to
distinguish slow mirrors from fast ones. For example, I am very much in
need of information about the situation in the US, but even though
people in the US run installations for me and are willing to send me
logs, I can't get this information.
In reply to a question from Stano:
> Do you want the information in the log or to be shown in e.g. zypper
-vvv ?
Shown, in normal output. It is already in some log I believe, I don't
know which one exactly anymore though.
Thing is, users won't be able to tell us anything useful if they
weren't shown the information. And later it is not reproducible.
At least in a failure situation the URLs must be shown. That alone
wouldn't give us the chance to find out about slow mirrors, but at
least it would allow to debug problems.
Real life example #1:
If a mirror is behind a firewall that lets every 10th connection stall
(so the transfer hangs or the connection is reset), it usually takes
weeks (!) to find out about it. But it hits virtually every user from
mirrors country during the time. We have had this 4 times during the
last 18 months.
Real life example #2:
I got feedback from someone who did a network install of openSUSE 11.0
from download.opensuse.org, located in the US. He had to let it running
overnight because it took a long time. Next morning, the install hadn't
even completed. YaST was hanging and waiting for somebody to hit retry.
Problem is, it took us a lot of work to find out which mirror was at
fault. We actually had to repeat the install, I had to follow the log
and live watching his machine accessing download.opensuse.org, to see
where it was redirected, and where it spent most time (while
downloading some package). We found a mirror which was really slow, and
I could disable it. I have a suspicion though, that we have more such
mirrors, because I have got some similar reports from recently. But how
to find out? People only complain about slowness, without the ability
to give any useful detail. See
http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments
(http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments)
for examples of this.
References:
https://bugzilla.novell.com/show_bug.cgi?id=522260 - patch adding
logging of redirections for curl backend
+ Test Case:
+ For zypp on code 11 (openSUSE 11.1, SLE11):
+ Add any repository below download.opensuse.org via zypper or YaST and
+ install a package from it (the package should be served from a mirror).
+ Zypper/YaST log should contain lines with "Location:" string (grep
+ 'Location: ' /var/log/zypper.log).
+ For zypp on code 12, the redirections are not logged, but are part of
+ the error message details in case download error. See the testcase in
+ Fate #302923 for info on how to get broken mirrors from the redirector
+ in order to test this.
Discussion:
#1: Susanne Oberhauser (froh) (2008-10-01 15:16:30)
What if we focussed on something like torrent for distributing packages
instead?
AFAICT these p2p network technologies have mechanisms to recover from
missing fragments, i.e. failing servers.
In that world, todays mirrors would be strong and almost always present
peers in the network.
#2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1)
Torrents are not efficient for files of this small size.
Another concern is that, on large scale, the reasonability of P2P
traffic with regard to network topology is questionable; confer to
what's called the Application-Layer Traffic Optimization (ALTO)
problem. See http://alto.tilab.com/ (http://alto.tilab.com/) .
There are working examples where e.g. university have rolled out a P2P
update distribution campus-wide; in a controlled limited environment
this is feasible, but not across the Internet.
Further, note that libzypp in 11.1 has preliminary support for using
metalinks for downlaods. Metalink clients recover nicely from missing
fragments. See http://en.opensuse.org/Libzypp/Failover
(http://en.opensuse.org/Libzypp/Failover) .
#3: Mark Goldstein (markgo) (2008-10-04 17:20:42)
Other usage of this feature would be possibility of "blacklisting"
servers with persistent problems. (See thread in opensuse list "Is it
possible to "blacklist" specific download server in Yast?" ).
As Carlos E. R. suggested other useful option could be for YaST/zypper
to fail gracefully requesting another server to the redirector on
failure on the one it gave use previously.
#4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3)
Gracefully falling back to other mirrors is already being implemented
(currently in prototype stage) by way of
http://en.opensuse.org/Libzypp/Failover
(http://en.opensuse.org/Libzypp/Failover)
.
The feature of blacklisting a specific mirrors is included in that
proposal, and I have added it explicitely.
#5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25)
Looking at the discussion on opensuse-factory and my own experience, I
also see this as high priority for 11.2.
#6: Peter Poeml (poeml) (2009-01-27 20:39:59)
Just for the record, the way to enable logging that makes mirrors show
up in the log seems to be:
$ ZYPP_MEDIA_CURL_DEBUG=1 zypper ...
Cf. http://en.opensuse.org/Debugging_YaST
#7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6)
Yes, it is, but that's quite excessive log that we don't want to be
enabled by default. Instead we need to pick up the redirects and log
them one line per each.
Michal Marek already gave me a hint how to do this: "Use
CURLOPT_HEADERFUNCTION callback, which gets each http header and then
test for 'Location:' and log that." I'll try to do it ASAP.
#8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7)
Let's try to display this information at least with zypper. However, we
will need some investigation, since we switched to aria2c.
#9: Peter Poeml (poeml) (2009-06-24 15:04:21)
Another example of the implications. What users get right now is this
(http://www.3111skyline.com/download/openSUSE_bugs/kde4/screenshots/YastsZypperStuck-soundtouch.jpeg)
(screenshot
that shows a rather meaningless URL, but not the accessed mirror). See
this thread (http://lists.opensuse.org/opensuse/2009-06/msg01206.html)
on the English opensuse mailing list.
#10: Ján Kupec (jkupec) (2009-07-14 17:54:07) (reply to #9)
OK, this is all about the CURL backend, so we're developing an
improvment to be released as an online update for code 10 products
(openSUSE 11.1 and SLE11), right?
Peter, is this needed also for the aria2c backend? If so, can you give
me some hints how to do it with aria (i have zero knowlege of aria)?
#11: Ján Kupec (jkupec) (2009-07-14 18:53:11) (reply to #10)
above, i meant code 11, of course
#13: Peter Poeml (poeml) (2009-07-15 16:05:29) (reply to #10)
I should mention that when I initially requested this feature, we were
still developing 11.1, and the fail-safe aria2c backend wasn't
implemented yet. Now that the new fail-safe backend is becoming the
default download backend in 11.2, the need for this feature request is
a lot smaller.
It would still be beneficial for us if users get direct feedback (via
the user interface) about mirror performance, because we will still
need to maintain an overall healthy mirror system, and tune here and
there. The fail-safe backend will compensate for many or most problems.
The fail-safe backend collects its own mirror performance stats, and we
could actually think about submit those stats to a central place to
gather a high-level view on mirror performance. We could actually
correlate the performance to geographical or network topological client
location. This would allow for better mirror selection in the first
place, which will allow for a better client experience overall.
openSUSE 11.1 and 11.0 would benefit a lot from the improvement
requested here, of course. Furthermore, there might be edge cases where
the aria2c backend isn't used - I'm not sure if we do (or can) use it
from the install initial ramdisk when installing with a "mini iso", for
instance.
The curl backend of 11.2 should definitely have good logging of all
server accesses - not only in a debug mode. The information must be
easily available. Best would be if its shown directly to the user, so
they know what could be a possible problem, and know what to report.
aria2c logs exceptions in a way that we can use to enhance our logging
(and UI). Tatsuhiro added the patch for us mentioned below:
I created a patch to embed URI(redirected one) to exception message.
Since there are many exception messages, I wrapped exception with new
one whose message contains URI. For example, the exception message for
404 error is now:
Exception: URI=http://localhost/abc
-> Resource not found
#14: Ján Kupec (jkupec) (2009-07-28 23:23:06) (reply to #13)
Thanx Peter for the comment. So we have the curl logging for code 11 (
bug 522260 (https://bugzilla.novell.com/show_bug.cgi?id=522260) ) and
code 12 (and not only in debug mode) ready.
As for 11.2, looking at the MediaAria2c.cc code, the exceptions are
already parsed from the output of aria2c and passed to callbacks, thus
shown to the user, so nothing to do here (Duncan, please correct me if
i'm wrong).
Regarding curl in 11.2, i guess having the redirects logged is enough,
since aria2c is now the default backend (even though it may not be
suitable for some uses, as Peter suggests).
All in all, i'd consider this feature done. Please reopen if i missed
something.
#12: Ján Kupec (jkupec) (2009-07-15 14:16:05)
Alright, logging is easy. If a Location: line will be found in reponse
header, it will be logged like: "MediaCurl.cc(log_redirects_curl):120
redirecting to Location: URL" in zypper.log. I have the patch ready and
it is pretty simple and can be released as update for openSUSE 11.x and
SLE11.
With showing the redirects in the UIs it is quite complicated since it
requires MediaException API changes and adding the list of redirects to
each MediaException thrown. Nothing for the released products i'd say.
Can we agree on this? Let me empasize that this is for the cURL backend
used in code11 products. For openSUSE 11.2 and later (using aria2c
backend) i'm still waiting for info asked in c#10.
--
openSUSE Feature:
https://features.opensuse.org/305320
| < Previous | Next > |