[openFate 305320] Show accessed mirror(s)
Feature added by: Peter Poeml <poeml@novell.com> Feature #305320, revision 1, last change by Title: Show accessed mirror(s) openSUSE-11.1: New Priority Requester: Mandatory Requested by: Peter Poeml <poeml@novell.com> Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though.
Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this. -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Susanne Oberhauser <froh@novell.com> Feature #305320, revision 3 Title: Show accessed mirror(s) openSUSE-11.1: New Priority Requester: Mandatory Requested by: Peter Poeml <poeml@novell.com> Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
+ Discussion: + #1: Susanne Oberhauser <froh@novell.com> (2008-10-01 15:16:30) + What if we focussed on something like torrent for distributing packages + instead? + AFAICT these p2p network technologies have mechanisms to recover from + missing fragments, i.e. failing servers. + In that world, todays mirrors would be strong and almost always present + peers in the network. -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Peter Poeml <poeml@novell.com> Feature #305320, revision 4 Title: Show accessed mirror(s) openSUSE-11.1: New Priority Requester: Mandatory Requested by: Peter Poeml <poeml@novell.com> Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser <froh@novell.com> (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. + #2: Peter Poeml <poeml@novell.com> (2008-10-01 15:30:56) (reply to #1) + Torrents are not efficient for files of this small size. + Another concern is that, on large scale, the reasonability of P2P + traffic with regard to network topology is questionable; confer to + what's called the Application-Layer Traffic Optimization (ALTO) + problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . + There are working examples where e.g. university have rolled out a P2P + update distribution campus-wide; in a controlled limited environment + this is feasible, but not across the Internet. + Further, note that libzypp in 11.1 has preliminary support for using + metalinks for downlaods. Metalink clients recover nicely from missing + fragments. See http://en.opensuse.org/Libzypp/Failover + (http://en.opensuse.org/Libzypp/Failover) . -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Stanislav Visnovsky <visnov@novell.com> Feature #305320, revision 5 Title: Show accessed mirror(s) - openSUSE-11.1: New + openSUSE-11.1: Rejected by Stanislav Visnovsky <visnov@novell.com> + reject date: 2008-10-01 15:49:39 + reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory + openSUSE-11.2: New + Priority + Requester: Mandatory Requested by: Peter Poeml <poeml@novell.com> Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser <froh@novell.com> (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml <poeml@novell.com> (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Mark Goldstein <goldstein.mark@gmail.com> Feature #305320, revision 6 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky <visnov@novell.com> reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: New Priority Requester: Mandatory Requested by: Peter Poeml <poeml@novell.com> Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser <froh@novell.com> (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml <poeml@novell.com> (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . + #3: Mark Goldstein <goldstein.mark@gmail.com> (2008-10-04 17:20:42) + Other usage of this feature would be possibility of "blacklisting" + servers with persistent problems. (See thread in opensuse list "Is it + possible to "blacklist" specific download server in Yast?" ). + As Carlos E. R. suggested other useful option could be for YaST/zypper + to fail gracefully requesting another server to the redirector on + failure on the one it gave use previously. -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Peter Poeml <poeml@novell.com> Feature #305320, revision 7 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky <visnov@novell.com> reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: New Priority Requester: Mandatory Requested by: Peter Poeml <poeml@novell.com> Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser <froh@novell.com> (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml <poeml@novell.com> (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein <goldstein.mark@gmail.com> (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. + #4: Peter Poeml <poeml@novell.com> (2008-10-04 17:55:14) (reply to #3) + Gracefully falling back to other mirrors is already being implemented + (currently in prototype stage) by way of + http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) + . + The feature of blacklisting a specific mirrors is included in that + proposal, and I have added it explicitely. -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Michael Loeffler <michl@novell.com> Feature #305320, revision 8 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky <visnov@novell.com> reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory - openSUSE-11.2: New + openSUSE-11.2: Evaluation Priority Requester: Mandatory Requested by: Peter Poeml <poeml@novell.com> Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser <froh@novell.com> (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml <poeml@novell.com> (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein <goldstein.mark@gmail.com> (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml <poeml@novell.com> (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Gerald Pfeifer <gp@novell.com> Feature #305320, revision 11 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky <visnov@novell.com> reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Evaluation Priority Requester: Mandatory Requested by: Peter Poeml <poeml@novell.com> Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser <froh@novell.com> (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml <poeml@novell.com> (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein <goldstein.mark@gmail.com> (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml <poeml@novell.com> (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. + #5: Gerald Pfeifer <gp@novell.com> (2008-10-30 17:33:25) + Looking at the discussion on opensuse-factory and my own experience, I + also see this as high priority for 11.2. -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Peter Poeml (poeml) Feature #305320, revision 14 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky <visnov@novell.com> reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Evaluation Priority Requester: Mandatory Requested by: Peter Poeml (poeml) Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. + #6: Peter Poeml (poeml) (2009-01-27 20:39:59) + Just for the record, the way to enable logging that makes mirrors show + up in the log seems to be: + $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... + Cf. http://en.opensuse.org/Debugging_YaST -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Ján Kupec (jkupec) Feature #305320, revision 15 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky <visnov@novell.com> reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Evaluation Priority Requester: Mandatory Requested by: Peter Poeml (poeml) Partner organization: openSUSE.org Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST + #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) + Yes, it is, but that's quite excessive log that we don't want to be + enabled by default. Instead we need to pick up the redirects and log + them one line per each. + Michal Marek already gave me a hint how to do this: "Use + CURLOPT_HEADERFUNCTION callback, which gets each http header and then + test for 'Location:' and log that." I'll try to do it ASAP. -- openSUSE Feature: https://features.opensuse.org/?rm=feature_show&id=305320
Feature changed by: Christoph Thiel (cthiel1) Feature #305320, revision 20 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov) reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Evaluation Priority Requester: Mandatory + Projectmanager: Mandatory Requested by: Peter Poeml (poeml) Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each. Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP. -- openSUSE Feature: https://features.opensuse.org/305320
Feature changed by: Jiri Srain (jsrain) Feature #305320, revision 21 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov) reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory - openSUSE-11.2: Evaluation + openSUSE-11.2: Candidate Priority Requester: Mandatory Projectmanager: Mandatory Requested by: Peter Poeml (poeml) Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each. Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP. + #8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7) + Let's try to display this information at least with zypper. However, we + will need some investigation, since we switched to aria2c. -- openSUSE Feature: https://features.opensuse.org/305320
Feature changed by: Peter Poeml (poeml) Feature #305320, revision 23 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov) reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Candidate Priority Requester: Mandatory Projectmanager: Mandatory Requested by: Peter Poeml (poeml) Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each. Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP. #8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7) Let's try to display this information at least with zypper. However, we will need some investigation, since we switched to aria2c. + #9: Peter Poeml (poeml) (2009-06-24 15:04:21) + Another example of the implications. What users get right now is this (http://www.3111skyline.com/download/openSUSE_bugs/kde4/screenshots/YastsZypp...) (screenshot + that shows a rather meaningless URL, but not the accessed mirror). See + this thread (http://lists.opensuse.org/opensuse/2009-06/msg01206.html) + on the English opensuse mailing list. -- openSUSE Feature: https://features.opensuse.org/305320
Feature changed by: Ján Kupec (jkupec) Feature #305320, revision 26 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov) reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Candidate Priority Requester: Mandatory Projectmanager: Mandatory Requested by: Peter Poeml (poeml) Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each. Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP. #8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7) Let's try to display this information at least with zypper. However, we will need some investigation, since we switched to aria2c. #9: Peter Poeml (poeml) (2009-06-24 15:04:21) Another example of the implications. What users get right now is this (http://www.3111skyline.com/download/openSUSE_bugs/kde4/screenshots/YastsZypp...) (screenshot that shows a rather meaningless URL, but not the accessed mirror). See this thread (http://lists.opensuse.org/opensuse/2009-06/msg01206.html) on the English opensuse mailing list. + #10: Ján Kupec (jkupec) (2009-07-14 17:54:07) (reply to #9) + OK, this is all about the CURL backend, so we're developing an + improvment to be released as an online update for code 10 products + (openSUSE 11.1 and SLE11), right? + Peter, is this needed also for the aria2c backend? If so, can you give + me some hints how to do it with aria (i have zero knowlege of aria)? -- openSUSE Feature: https://features.opensuse.org/305320
Feature changed by: Ján Kupec (jkupec) Feature #305320, revision 27 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov) reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Candidate Priority Requester: Mandatory Projectmanager: Mandatory Requested by: Peter Poeml (poeml) Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each. Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP. #8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7) Let's try to display this information at least with zypper. However, we will need some investigation, since we switched to aria2c. #9: Peter Poeml (poeml) (2009-06-24 15:04:21) Another example of the implications. What users get right now is this (http://www.3111skyline.com/download/openSUSE_bugs/kde4/screenshots/YastsZypp...) (screenshot that shows a rather meaningless URL, but not the accessed mirror). See this thread (http://lists.opensuse.org/opensuse/2009-06/msg01206.html) on the English opensuse mailing list. #10: Ján Kupec (jkupec) (2009-07-14 17:54:07) (reply to #9) OK, this is all about the CURL backend, so we're developing an improvment to be released as an online update for code 10 products (openSUSE 11.1 and SLE11), right? Peter, is this needed also for the aria2c backend? If so, can you give me some hints how to do it with aria (i have zero knowlege of aria)? + #11: Ján Kupec (jkupec) (2009-07-14 18:53:11) (reply to #10) + above, i meant code 11, of course -- openSUSE Feature: https://features.opensuse.org/305320
Feature changed by: Ján Kupec (jkupec) Feature #305320, revision 28 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov) reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Candidate Priority Requester: Mandatory Projectmanager: Mandatory Requested by: Peter Poeml (poeml) Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each. Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP. #8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7) Let's try to display this information at least with zypper. However, we will need some investigation, since we switched to aria2c. #9: Peter Poeml (poeml) (2009-06-24 15:04:21) Another example of the implications. What users get right now is this (http://www.3111skyline.com/download/openSUSE_bugs/kde4/screenshots/YastsZypp...) (screenshot that shows a rather meaningless URL, but not the accessed mirror). See this thread (http://lists.opensuse.org/opensuse/2009-06/msg01206.html) on the English opensuse mailing list. #10: Ján Kupec (jkupec) (2009-07-14 17:54:07) (reply to #9) OK, this is all about the CURL backend, so we're developing an improvment to be released as an online update for code 10 products (openSUSE 11.1 and SLE11), right? Peter, is this needed also for the aria2c backend? If so, can you give me some hints how to do it with aria (i have zero knowlege of aria)? #11: Ján Kupec (jkupec) (2009-07-14 18:53:11) (reply to #10) above, i meant code 11, of course + #12: Ján Kupec (jkupec) (2009-07-15 14:16:05) + Alright, logging is easy. If a Location: line will be found in reponse + header, it will be logged like: "MediaCurl.cc(log_redirects_curl):120 + redirecting to Location: URL" in zypper.log. I have the patch ready and + it is pretty simple and can be released as update for openSUSE 11.x and + SLE11. + With showing the redirects in the UIs it is quite complicated since it + requires MediaException API changes and adding the list of redirects to + each MediaException thrown. Nothing for the released products i'd say. + Can we agree on this? Let me empasize that this is for the cURL backend + used in code11 products. For openSUSE 11.2 and later (using aria2c + backend) i'm still waiting for info asked in c#10. -- openSUSE Feature: https://features.opensuse.org/305320
Feature changed by: Peter Poeml (poeml) Feature #305320, revision 29 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov) reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Candidate Priority Requester: Mandatory Projectmanager: Mandatory Requested by: Peter Poeml (poeml) Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each. Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP. #8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7) Let's try to display this information at least with zypper. However, we will need some investigation, since we switched to aria2c. #9: Peter Poeml (poeml) (2009-06-24 15:04:21) Another example of the implications. What users get right now is this (http://www.3111skyline.com/download/openSUSE_bugs/kde4/screenshots/YastsZypp...) (screenshot that shows a rather meaningless URL, but not the accessed mirror). See this thread (http://lists.opensuse.org/opensuse/2009-06/msg01206.html) on the English opensuse mailing list. #10: Ján Kupec (jkupec) (2009-07-14 17:54:07) (reply to #9) OK, this is all about the CURL backend, so we're developing an improvment to be released as an online update for code 10 products (openSUSE 11.1 and SLE11), right? Peter, is this needed also for the aria2c backend? If so, can you give me some hints how to do it with aria (i have zero knowlege of aria)? #11: Ján Kupec (jkupec) (2009-07-14 18:53:11) (reply to #10) above, i meant code 11, of course + #13: Peter Poeml (poeml) (2009-07-15 16:05:29) (reply to #10) + I should mention that when I initially requested this feature, we were + still developing 11.1, and the fail-safe aria2c backend wasn't + implemented yet. Now that the new fail-safe backend is becoming the + default download backend in 11.2, the need for this feature request is + a lot smaller. + It would still be beneficial for us if users get direct feedback (via + the user interface) about mirror performance, because we will still + need to maintain an overall healthy mirror system, and tune here and + there. The fail-safe backend will compensate for many or most problems. + The fail-safe backend collects its own mirror performance stats, and we + could actually think about submit those stats to a central place to + gather a high-level view on mirror performance. We could actually + correlate the performance to geographical or network topological client + location. This would allow for better mirror selection in the first + place, which will allow for a better client experience overall. + openSUSE 11.1 and 11.0 would benefit a lot from the improvement + requested here, of course. Furthermore, there might be edge cases where + the aria2c backend isn't used - I'm not sure if we do (or can) use it + from the install initial ramdisk when installing with a "mini iso", for + instance. + The curl backend of 11.2 should definitely have good logging of all + server accesses - not only in a debug mode. The information must be + easily available. Best would be if its shown directly to the user, so + they know what could be a possible problem, and know what to report. + aria2c logs exceptions in a way that we can use to enhance our logging + (and UI). Tatsuhiro added the patch for us mentioned below: + I created a patch to embed URI(redirected one) to exception message. + Since there are many exception messages, I wrapped exception with new + one whose message contains URI. For example, the exception message for + 404 error is now: + Exception: URI=http://localhost/abc + -> Resource not found #12: Ján Kupec (jkupec) (2009-07-15 14:16:05) Alright, logging is easy. If a Location: line will be found in reponse header, it will be logged like: "MediaCurl.cc(log_redirects_curl):120 redirecting to Location: URL" in zypper.log. I have the patch ready and it is pretty simple and can be released as update for openSUSE 11.x and SLE11. With showing the redirects in the UIs it is quite complicated since it requires MediaException API changes and adding the list of redirects to each MediaException thrown. Nothing for the released products i'd say. Can we agree on this? Let me empasize that this is for the cURL backend used in code11 products. For openSUSE 11.2 and later (using aria2c backend) i'm still waiting for info asked in c#10. -- openSUSE Feature: https://features.opensuse.org/305320
Feature changed by: Ján Kupec (jkupec) Feature #305320, revision 30 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov) reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory - openSUSE-11.2: Candidate + openSUSE-11.2: Done Priority Requester: Mandatory Projectmanager: Mandatory Requested by: Peter Poeml (poeml) Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
+ References: + https://bugzilla.novell.com/show_bug.cgi?id=522260 - patch adding + logging of redirections for curl backend Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each. Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP. #8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7) Let's try to display this information at least with zypper. However, we will need some investigation, since we switched to aria2c. #9: Peter Poeml (poeml) (2009-06-24 15:04:21) Another example of the implications. What users get right now is this (http://www.3111skyline.com/download/openSUSE_bugs/kde4/screenshots/YastsZypp...) (screenshot that shows a rather meaningless URL, but not the accessed mirror). See this thread (http://lists.opensuse.org/opensuse/2009-06/msg01206.html) on the English opensuse mailing list. #10: Ján Kupec (jkupec) (2009-07-14 17:54:07) (reply to #9) OK, this is all about the CURL backend, so we're developing an improvment to be released as an online update for code 10 products (openSUSE 11.1 and SLE11), right? Peter, is this needed also for the aria2c backend? If so, can you give me some hints how to do it with aria (i have zero knowlege of aria)? #11: Ján Kupec (jkupec) (2009-07-14 18:53:11) (reply to #10) above, i meant code 11, of course #13: Peter Poeml (poeml) (2009-07-15 16:05:29) (reply to #10) I should mention that when I initially requested this feature, we were still developing 11.1, and the fail-safe aria2c backend wasn't implemented yet. Now that the new fail-safe backend is becoming the default download backend in 11.2, the need for this feature request is a lot smaller. It would still be beneficial for us if users get direct feedback (via the user interface) about mirror performance, because we will still need to maintain an overall healthy mirror system, and tune here and there. The fail-safe backend will compensate for many or most problems. The fail-safe backend collects its own mirror performance stats, and we could actually think about submit those stats to a central place to gather a high-level view on mirror performance. We could actually correlate the performance to geographical or network topological client location. This would allow for better mirror selection in the first place, which will allow for a better client experience overall. openSUSE 11.1 and 11.0 would benefit a lot from the improvement requested here, of course. Furthermore, there might be edge cases where the aria2c backend isn't used - I'm not sure if we do (or can) use it from the install initial ramdisk when installing with a "mini iso", for instance. The curl backend of 11.2 should definitely have good logging of all server accesses - not only in a debug mode. The information must be easily available. Best would be if its shown directly to the user, so they know what could be a possible problem, and know what to report. aria2c logs exceptions in a way that we can use to enhance our logging (and UI). Tatsuhiro added the patch for us mentioned below: I created a patch to embed URI(redirected one) to exception message. Since there are many exception messages, I wrapped exception with new one whose message contains URI. For example, the exception message for 404 error is now: Exception: URI=http://localhost/abc -> Resource not found + #14: Ján Kupec (jkupec) (2009-07-28 23:23:06) (reply to #13) + Thanx Peter for the comment. So we have the curl logging for code 11 ( + bug 522260 (https://bugzilla.novell.com/show_bug.cgi?id=522260) ) and + code 12 (and not only in debug mode) ready. + As for 11.2, looking at the MediaAria2c.cc code, the exceptions are + already parsed from the output of aria2c and passed to callbacks, thus + shown to the user, so nothing to do here (Duncan, please correct me if + i'm wrong). + Regarding curl in 11.2, i guess having the redirects logged is enough, + since aria2c is now the default backend (even though it may not be + suitable for some uses, as Peter suggests). + All in all, i'd consider this feature done. Please reopen if i missed + something. #12: Ján Kupec (jkupec) (2009-07-15 14:16:05) Alright, logging is easy. If a Location: line will be found in reponse header, it will be logged like: "MediaCurl.cc(log_redirects_curl):120 redirecting to Location: URL" in zypper.log. I have the patch ready and it is pretty simple and can be released as update for openSUSE 11.x and SLE11. With showing the redirects in the UIs it is quite complicated since it requires MediaException API changes and adding the list of redirects to each MediaException thrown. Nothing for the released products i'd say. Can we agree on this? Let me empasize that this is for the cURL backend used in code11 products. For openSUSE 11.2 and later (using aria2c backend) i'm still waiting for info asked in c#10. -- openSUSE Feature: https://features.opensuse.org/305320
Feature changed by: Ján Kupec (jkupec) Feature #305320, revision 32 Title: Show accessed mirror(s) openSUSE-11.1: Rejected by Stanislav Visnovsky (visnov) reject date: 2008-10-01 15:49:39 reject reason: Too late, but let's consider for the future ones. Priority Requester: Mandatory openSUSE-11.2: Done Priority Requester: Mandatory Projectmanager: Mandatory Requested by: Peter Poeml (poeml) Description: This concerns both YaST and zypper. It would help a lot if they would show the mirror which is accessed when downloading a package. (Technically, the download client is following redirects and the actually URL needs to be shown, not just the first URL that was contacted but that redirected to another URL) It is a long-standing problem that this information is lacking and it makes debugging and optimizing much harder than necessary, because this information is only available deeply buried in some logfile. We need this information for two reasons: 1) Debugging - to see which URL was accessed when something goes wrong, like a mirror timing out. For example, Whenever a mirror returns garbage, it takes a lot of additional time to find out which one, because all reports are lacking the critical detail, and it is usually hard to find it out later. 2) Optimizing the load sharing between different mirrors - to distinguish slow mirrors from fast ones. For example, I am very much in need of information about the situation in the US, but even though people in the US run installations for me and are willing to send me logs, I can't get this information. In reply to a question from Stano:
Do you want the information in the log or to be shown in e.g. zypper -vvv ? Shown, in normal output. It is already in some log I believe, I don't know which one exactly anymore though. Thing is, users won't be able to tell us anything useful if they weren't shown the information. And later it is not reproducible. At least in a failure situation the URLs must be shown. That alone wouldn't give us the chance to find out about slow mirrors, but at least it would allow to debug problems. Real life example #1: If a mirror is behind a firewall that lets every 10th connection stall (so the transfer hangs or the connection is reset), it usually takes weeks (!) to find out about it. But it hits virtually every user from mirrors country during the time. We have had this 4 times during the last 18 months. Real life example #2: I got feedback from someone who did a network install of openSUSE 11.0 from download.opensuse.org, located in the US. He had to let it running overnight because it took a long time. Next morning, the install hadn't even completed. YaST was hanging and waiting for somebody to hit retry. Problem is, it took us a lot of work to find out which mirror was at fault. We actually had to repeat the install, I had to follow the log and live watching his machine accessing download.opensuse.org, to see where it was redirected, and where it spent most time (while downloading some package). We found a mirror which was really slow, and I could disable it. I have a suspicion though, that we have more such mirrors, because I have got some similar reports from recently. But how to find out? People only complain about slowness, without the ability to give any useful detail. See http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments (http://news.opensuse.org/2008/09/23/upcoming-factory-changes/#comments) for examples of this.
References: https://bugzilla.novell.com/show_bug.cgi?id=522260 - patch adding logging of redirections for curl backend + Test Case: + For zypp on code 11 (openSUSE 11.1, SLE11): + Add any repository below download.opensuse.org via zypper or YaST and + install a package from it (the package should be served from a mirror). + Zypper/YaST log should contain lines with "Location:" string (grep + 'Location: ' /var/log/zypper.log). + For zypp on code 12, the redirections are not logged, but are part of + the error message details in case download error. See the testcase in + Fate #302923 for info on how to get broken mirrors from the redirector + in order to test this. Discussion: #1: Susanne Oberhauser (froh) (2008-10-01 15:16:30) What if we focussed on something like torrent for distributing packages instead? AFAICT these p2p network technologies have mechanisms to recover from missing fragments, i.e. failing servers. In that world, todays mirrors would be strong and almost always present peers in the network. #2: Peter Poeml (poeml) (2008-10-01 15:30:56) (reply to #1) Torrents are not efficient for files of this small size. Another concern is that, on large scale, the reasonability of P2P traffic with regard to network topology is questionable; confer to what's called the Application-Layer Traffic Optimization (ALTO) problem. See http://alto.tilab.com/ (http://alto.tilab.com/) . There are working examples where e.g. university have rolled out a P2P update distribution campus-wide; in a controlled limited environment this is feasible, but not across the Internet. Further, note that libzypp in 11.1 has preliminary support for using metalinks for downlaods. Metalink clients recover nicely from missing fragments. See http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . #3: Mark Goldstein (markgo) (2008-10-04 17:20:42) Other usage of this feature would be possibility of "blacklisting" servers with persistent problems. (See thread in opensuse list "Is it possible to "blacklist" specific download server in Yast?" ). As Carlos E. R. suggested other useful option could be for YaST/zypper to fail gracefully requesting another server to the redirector on failure on the one it gave use previously. #4: Peter Poeml (poeml) (2008-10-04 17:55:14) (reply to #3) Gracefully falling back to other mirrors is already being implemented (currently in prototype stage) by way of http://en.opensuse.org/Libzypp/Failover (http://en.opensuse.org/Libzypp/Failover) . The feature of blacklisting a specific mirrors is included in that proposal, and I have added it explicitely. #5: Gerald Pfeifer (geraldpfeifer) (2008-10-30 17:33:25) Looking at the discussion on opensuse-factory and my own experience, I also see this as high priority for 11.2. #6: Peter Poeml (poeml) (2009-01-27 20:39:59) Just for the record, the way to enable logging that makes mirrors show up in the log seems to be: $ ZYPP_MEDIA_CURL_DEBUG=1 zypper ... Cf. http://en.opensuse.org/Debugging_YaST #7: Ján Kupec (jkupec) (2009-01-28 12:25:39) (reply to #6) Yes, it is, but that's quite excessive log that we don't want to be enabled by default. Instead we need to pick up the redirects and log them one line per each. Michal Marek already gave me a hint how to do this: "Use CURLOPT_HEADERFUNCTION callback, which gets each http header and then test for 'Location:' and log that." I'll try to do it ASAP. #8: Jiri Srain (jsrain) (2009-06-02 11:02:23) (reply to #7) Let's try to display this information at least with zypper. However, we will need some investigation, since we switched to aria2c. #9: Peter Poeml (poeml) (2009-06-24 15:04:21) Another example of the implications. What users get right now is this (http://www.3111skyline.com/download/openSUSE_bugs/kde4/screenshots/YastsZypp...) (screenshot that shows a rather meaningless URL, but not the accessed mirror). See this thread (http://lists.opensuse.org/opensuse/2009-06/msg01206.html) on the English opensuse mailing list. #10: Ján Kupec (jkupec) (2009-07-14 17:54:07) (reply to #9) OK, this is all about the CURL backend, so we're developing an improvment to be released as an online update for code 10 products (openSUSE 11.1 and SLE11), right? Peter, is this needed also for the aria2c backend? If so, can you give me some hints how to do it with aria (i have zero knowlege of aria)? #11: Ján Kupec (jkupec) (2009-07-14 18:53:11) (reply to #10) above, i meant code 11, of course #13: Peter Poeml (poeml) (2009-07-15 16:05:29) (reply to #10) I should mention that when I initially requested this feature, we were still developing 11.1, and the fail-safe aria2c backend wasn't implemented yet. Now that the new fail-safe backend is becoming the default download backend in 11.2, the need for this feature request is a lot smaller. It would still be beneficial for us if users get direct feedback (via the user interface) about mirror performance, because we will still need to maintain an overall healthy mirror system, and tune here and there. The fail-safe backend will compensate for many or most problems. The fail-safe backend collects its own mirror performance stats, and we could actually think about submit those stats to a central place to gather a high-level view on mirror performance. We could actually correlate the performance to geographical or network topological client location. This would allow for better mirror selection in the first place, which will allow for a better client experience overall. openSUSE 11.1 and 11.0 would benefit a lot from the improvement requested here, of course. Furthermore, there might be edge cases where the aria2c backend isn't used - I'm not sure if we do (or can) use it from the install initial ramdisk when installing with a "mini iso", for instance. The curl backend of 11.2 should definitely have good logging of all server accesses - not only in a debug mode. The information must be easily available. Best would be if its shown directly to the user, so they know what could be a possible problem, and know what to report. aria2c logs exceptions in a way that we can use to enhance our logging (and UI). Tatsuhiro added the patch for us mentioned below: I created a patch to embed URI(redirected one) to exception message. Since there are many exception messages, I wrapped exception with new one whose message contains URI. For example, the exception message for 404 error is now: Exception: URI=http://localhost/abc -> Resource not found #14: Ján Kupec (jkupec) (2009-07-28 23:23:06) (reply to #13) Thanx Peter for the comment. So we have the curl logging for code 11 ( bug 522260 (https://bugzilla.novell.com/show_bug.cgi?id=522260) ) and code 12 (and not only in debug mode) ready. As for 11.2, looking at the MediaAria2c.cc code, the exceptions are already parsed from the output of aria2c and passed to callbacks, thus shown to the user, so nothing to do here (Duncan, please correct me if i'm wrong). Regarding curl in 11.2, i guess having the redirects logged is enough, since aria2c is now the default backend (even though it may not be suitable for some uses, as Peter suggests). All in all, i'd consider this feature done. Please reopen if i missed something. #12: Ján Kupec (jkupec) (2009-07-15 14:16:05) Alright, logging is easy. If a Location: line will be found in reponse header, it will be logged like: "MediaCurl.cc(log_redirects_curl):120 redirecting to Location: URL" in zypper.log. I have the patch ready and it is pretty simple and can be released as update for openSUSE 11.x and SLE11. With showing the redirects in the UIs it is quite complicated since it requires MediaException API changes and adding the list of redirects to each MediaException thrown. Nothing for the released products i'd say. Can we agree on this? Let me empasize that this is for the cURL backend used in code11 products. For openSUSE 11.2 and later (using aria2c backend) i'm still waiting for info asked in c#10. -- openSUSE Feature: https://features.opensuse.org/305320
participants (1)
-
fate_noreply＠suse.de