Re: [opensuse] One repo in the pool killing updates - probably utexas.edu - can it be fixed?
On 09/21/2010 04:24 AM, Peter Pöml wrote:
David,
Le 21.09.2010 à 00:18, David C. Rankin a écrit :
Guys,
It is literally like Russian-Roulette updating openSuSE with the mirror pool you get with download.opensuse.org. There is at least 1 maybe more mirrors that give no more than 1-4K/s download speed. (utexas.edu is a know problem repo). This is getting ridiculous. I have had to kill 3-4 updates in the past 2-3 days when I hit one of these mirrors. It would take more than an hour to download an 800k file.
I have searched zypper.log and can't find the problem mirror this time. Where to look? Also, this has been a problem for more than a year, why aren't these mirrors being dropped or fixed? I end up having to hard-code the mirror I want just to get reasonable consistent updates. Something should be done to correct this.
Two questions:
Have you reported it to those who can fix it? (I don't consider this mailing list as "those who can fix it". So I'm not sure what you expect by posting here. Does it help others?)
If you didn't: *do* you know where to report it?
If you did (which is what I assume), and it's not fixed after such a long time (a year!), I think that makes the openSUSE project look pretty bad. :-(
Peter
Brother Peter, Yes on all 3. Question: With yast, I could look in the yast log and find the offending repo IP and hostname, can I find that same info if I use zypper? If so where? I've dug through zypper.log but struck out.... and grepped the remainder of /var/log for the package name I was updating when the update hung on the bad repo. Is there anywhere else to look to help narrow it down? -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday, September 22, 2010 02:00:35 David C. Rankin wrote:
Is there anywhere else to look to help narrow it down?
/var/log/zypp/history -- phani. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hi David, Le 21.09.2010 à 22:30, David C. Rankin a écrit :
On 09/21/2010 04:24 AM, Peter Pöml wrote:
David,
Le 21.09.2010 à 00:18, David C. Rankin a écrit :
Guys,
It is literally like Russian-Roulette updating openSuSE with the mirror pool you get with download.opensuse.org. There is at least 1 maybe more mirrors that give no more than 1-4K/s download speed. (utexas.edu is a know problem repo). This is getting ridiculous. I have had to kill 3-4 updates in the past 2-3 days when I hit one of these mirrors. It would take more than an hour to download an 800k file.
I have searched zypper.log and can't find the problem mirror this time. Where to look? Also, this has been a problem for more than a year, why aren't these mirrors being dropped or fixed? I end up having to hard-code the mirror I want just to get reasonable consistent updates. Something should be done to correct this.
Two questions:
Have you reported it to those who can fix it? (I don't consider this mailing list as "those who can fix it". So I'm not sure what you expect by posting here. Does it help others?)
If you didn't: *do* you know where to report it?
If you did (which is what I assume), and it's not fixed after such a long time (a year!), I think that makes the openSUSE project look pretty bad. :-(
Peter
Brother Peter,
Yes on all 3. Question: With yast, I could look in the yast log and find the offending repo IP and hostname, can I find that same info if I use zypper? If so where? I've dug through zypper.log but struck out.... and grepped the remainder of /var/log for the package name I was updating when the update hung on the bad repo. Is there anywhere else to look to help narrow it down?
Hm, makes me sad to hear that this problem is not being taken care of, and I can well understand frustration following that. It's probably packages from the build service which are the problem. There are two "classes" of packages in the build service: those that are more popular and are mirrored by 4 mirrors in the US, and those that are a little less popular and are mirrored only by 2 mirrors. This includes all packages in the "home:" namespace, but also many others. Recently, one US mirror stopped mirroring the build service, so the latter packages are not properly mirrored anymore in the US. The remaining two mirrors in the US are: University of Texas, and a mirror operated by susestudio.com. From what I hear, both are very slow. I'd think that this is the #1 reason that makes downloads very slow for you. All other mirrors are in Europe (4xde, gb, pt, nl, ro). I don't know if the US mirrors are slow as such, or if they are just flooded with too many requests because they are the only ones. Anyway, the situation would improve if somebody in the US would volunteer and contribute a mirror. However, nobody there seems to be interested enough to step up and contribute spare time and resources for this purpose. There may still be possibilities. I have found that chances in finding new mirrors can be considerably improved by actively approaching existing mirrors and ask them for help. Beyond that, if one of the US mirrors is not performing well, it should be taken out of the rotation. But maybe they do perform well but are just overloaded, due to lack of other mirrors. I can't tell. (I am assuming that utexas.edu is indeed the recurrent problem for you.) Having stated all that, maybe you can confirm some observations? I guess you are using a more recent openSUSE version, like 11.3? As you know, since 11.2, zypper/YaST use a download backend which should work around about exactly the above issues, by contacting several mirrors in parallel. Maybe that is not configured well. Take repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm as an example for a package which is only on few mirrors. Here are they, as seen from the US: % curl -s 'http://download.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64...' | grep location <url type="http" location="us" preference="100">http://ftp.utexas.edu/opensuse/repositories/home:/-miska-/openSUSE_11.3/x86_...</url> <url type="http" location="us" preference="99">http://130.57.19.201/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-...</url> <url type="http" location="de" preference="98">http://ftp.halifax.rwth-aachen.de/opensuse/repositories/home:/-miska-/openSU...</url> <url type="http" location="de" preference="97">http://ftp5.gwdg.de/pub/opensuse/repositories/home:/-miska-/openSUSE_11.3/x8...</url> <url type="http" location="de" preference="96">http://widehat.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64/...</url> <url type="http" location="ro" preference="95">http://ftp.roedu.net/mirrors/opensuse.org/repositories/home:/-miska-/openSUS...</url> Using aria2c directly, we can see what it does with that: % aria2c --log=- --log-level=debug '--user-agent=ZYpp 7.7.5 (aria2 version 1.9.1) openSUSE-11.3-x86_64' --summary-interval=1 --follow-metalink=mem --check-integrity=true --file-allocation=none --server-stat-of=/home/poeml/.aria2/aria2.stats --server-stat-if=/home/poeml/.aria2/aria2.stats --uri-selector=adaptive --use-head=false --max-tries=5 --max-concurrent-downloads=2 --header=Pragma: --connect-timeout=180 'http://download.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64...' | grep Host Host: download.opensuse.org Host: widehat.opensuse.org Host: ftp.halifax.rwth-aachen.de Host: ftp5.gwdg.de Host: ftp.roedu.net Host: 130.57.19.201 It contacts 5 servers, which is the built-in default for concurrency. The job finishes after 1 or 2 seconds for me, which is about the same time to download the package from a single source in this case. Maybe if you can try that out, do you get a working download with that? If you see a long hang with that, it might be due to this bug: https://bugzilla.novell.com/show_bug.cgi?id=555803 Sigh. When testing the same as above from the US, aria2c does not contact 5 mirrors; it contacts only one, and if that is utexas.edu, the download takes a minute instead of 2-4 seconds. The log shows that only one server is contacted, and it is reproducible: % aria2c --log=- --log-level=debug '--user-agent=ZYpp 7.7.5 (aria2 version 1.9.1) openSUSE-11.3-x86_64' --summary-interval=1 --follow-metalink=mem --check-integrity=true --file-allocation=none --server-stat-of=/home/poeml/.aria2/aria2.stats --server-stat-if=/home/poeml/.aria2/aria2.stats --uri-selector=adaptive --use-head=false --max-tries=5 --max-concurrent-downloads=2 --header=Pragma: --connect-timeout=180 'http://download.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64...' | grep Host Host: download.opensuse.org Host: 130.57.19.201 The worst is, I don't know why this differs. Apart from the aria2c version everything should be exactly the same. In the latter case, I used the newest version, 1.10.3. Which aria2 package version do you have installed? Peter-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hi again, if the problem I just described is indeed affecting you, I might have interesting new info: Le 22.09.2010 à 11:23, Peter Pöml a écrit :
When testing the same as above from the US, aria2c does not contact 5 mirrors; it contacts only one, and if that is utexas.edu, the download takes a minute instead of 2-4 seconds. The log shows that only one server is contacted, and it is reproducible:
% aria2c --log=- --log-level=debug '--user-agent=ZYpp 7.7.5 (aria2 version 1.9.1) openSUSE-11.3-x86_64' --summary-interval=1 --follow-metalink=mem --check-integrity=true --file-allocation=none --server-stat-of=/home/poeml/.aria2/aria2.stats --server-stat-if=/home/poeml/.aria2/aria2.stats --uri-selector=adaptive --use-head=false --max-tries=5 --max-concurrent-downloads=2 --header=Pragma: --connect-timeout=180 'http://download.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64...' | grep Host Host: download.opensuse.org Host: 130.57.19.201
The worst is, I don't know why this differs. Apart from the aria2c version everything should be exactly the same. In the latter case, I used the newest version, 1.10.3.
Which aria2 package version do you have installed?
I found http://sourceforge.net/tracker/?func=detail&aid=3043968&group_id=159897&atid=813673 which mentions a changed behaviour in aria2c 1.10.x, compared to 1.9.x. Indeed, when I try the mentioned options "--max-connection-per-server=4 --min-split-size=1M" aria2c behaves again as I would expect it: poeml@download:~/Images> time aria2c --log=- --log-level=debug --summary-interval=1 --follow-metalink=mem --check-integrity=true --file-allocation=none --use-head=false --max-tries=5 --connect-timeout=180 --max-connection-per-server=1 --min-split-size=1M 'http://download.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64...' | grep Host Host: download.opensuse.org 2010-09-22 11:30:19.744906 DEBUG - UsedHost=1, ftp.utexas.edu 2010-09-22 11:30:19.745048 DEBUG - UsedHost=1, 130.57.19.201 2010-09-22 11:30:19.745057 DEBUG - UsedHost=1, ftp.utexas.edu 2010-09-22 11:30:19.745179 DEBUG - UsedHost=1, 130.57.19.201 2010-09-22 11:30:19.745189 DEBUG - UsedHost=1, ftp.utexas.edu 2010-09-22 11:30:19.745198 DEBUG - UsedHost=1, ftp5.gwdg.de 2010-09-22 11:30:19.745318 DEBUG - UsedHost=1, 130.57.19.201 2010-09-22 11:30:19.745328 DEBUG - UsedHost=1, ftp.utexas.edu 2010-09-22 11:30:19.745337 DEBUG - UsedHost=1, ftp5.gwdg.de 2010-09-22 11:30:19.745345 DEBUG - UsedHost=1, widehat.opensuse.org Host: 130.57.19.201 Host: ftp.halifax.rwth-aachen.de Host: widehat.opensuse.org Host: ftp5.gwdg.de Host: ftp.utexas.edu Host: 130.57.19.201 aria2c --log=- --log-level=debug --summary-interval=1 --follow-metalink=mem 0.00s user 0.24s system 5% cpu 4.089 total grep Host 0.00s user 0.05s system 1% cpu 4.053 total As the log excerpt shows, aria2c contacts not only one server, and it finishes the download in about 4 seconds (instead of one minute) even though the slow utexas.edu mirror was used (among others). That's how it should be. So, if you see the same happening, and if you are indeed using a new aria2c (1.10.x), then one should figure out how zypper/yast can use the right options (or if this is a bug). Peter-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 09/22/2010 04:54 AM, Peter Pöml wrote:
As the log excerpt shows, aria2c contacts not only one server, and it finishes the download in about 4 seconds (instead of one minute) even though the slow utexas.edu mirror was used (among others).
That's how it should be.
So, if you see the same happening, and if you are indeed using a new aria2c (1.10.x), then one should figure out how zypper/yast can use the right options (or if this is a bug).
Peter, Thanks for your follow-up, it will take me a bit to digest all of this info since I'm not that familiar with aria2 (yet). I'll try and test in the next night or two and report back. I'll be the utexas.edu US mirror is just saturated to the point of being worthless. When I hit it, I'll get 2-6K/s then it will drop to 0.0K/s for 5-10 secs and then give a second or two of download and then drop back to 0. I don't know if this is aria2 or just getting attached to a bad server. However, I can tell you that once I am attached to that server -- it will never change and will never find another until I ctrl+c the update and start it over and grab another mirror. Thanks again. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday, 2010-09-22 at 11:54 +0200, Peter Pöml wrote:
I found http://sourceforge.net/tracker/?func=detail&aid=3043968&group_id=159897&atid=813673 which mentions a changed behaviour in aria2c 1.10.x, compared to 1.9.x. Indeed, when I try the mentioned options "--max-connection-per-server=4 --min-split-size=1M" aria2c behaves again as I would expect it:
... Then I think that aria2c should be upgraded (via update repo, recomended update) on all openSUSE distros. That would reduce the problem of missing mirrors in the US. -- Cheers, Carlos E. R.
On 09/22/2010 04:17 PM, Carlos E. R. wrote:
Then I think that aria2c should be upgraded (via update repo, recomended update) on all openSUSE distros. That would reduce the problem of missing mirrors in the US.
+1 -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday, 2010-09-22 at 16:51 -0500, David C. Rankin wrote:
On 09/22/2010 04:17 PM, Carlos E. R. wrote:
Then I think that aria2c should be upgraded (via update repo, recomended update) on all openSUSE distros. That would reduce the problem of missing mirrors in the US.
+1
Who writes the Bugzilla? ;-) -- Cheers, Carlos E. R. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
* Carlos E. R.
On Wednesday, 2010-09-22 at 16:51 -0500, David C. Rankin wrote:
On 09/22/2010 04:17 PM, Carlos E. R. wrote:
Then I think that aria2c should be upgraded (via update repo, recomended update) on all openSUSE distros. That would reduce the problem of missing mirrors in the US.
+1
Who writes the Bugzilla? ;-)
http://download.opensuse.org/repositories/network:/utilities/openSUSE_11.2/ aria2-1.10.3-30.1.x86_64 -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday, 2010-09-22 at 19:36 -0400, Patrick Shanahan wrote:
On 09/22/2010 04:17 PM, Carlos E. R. wrote:
Then I think that aria2c should be upgraded (via update repo, recomended update) on all openSUSE distros. That would reduce the problem of missing mirrors in the US.
+1
Who writes the Bugzilla? ;-)
http://download.opensuse.org/repositories/network:/utilities/openSUSE_11.2/ aria2-1.10.3-30.1.x86_64
That's not the point. I know how to upgrade my system. The point is that everybody get their system automatically updated. -- Cheers, Carlos E. R. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 09/22/2010 06:17 PM, Carlos E. R. wrote:
Who writes the Bugzilla? ;-)
I started the post so I'll file the bug -- even though I don't know what the hell Peter said :p -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 09/22/2010 08:15 PM, David C. Rankin wrote:
On 09/22/2010 06:17 PM, Carlos E. R. wrote:
Who writes the Bugzilla? ;-)
I started the post so I'll file the bug -- even though I don't know what the hell Peter said :p
Done: Also providing full attribution for the smart people that figured out what the issue was: https://bugzilla.novell.com/show_bug.cgi?id=641257 -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hi, Le 22.09.2010 à 23:17, Carlos E. R. a écrit :
On Wednesday, 2010-09-22 at 11:54 +0200, Peter Pöml wrote:
I found http://sourceforge.net/tracker/?func=detail&aid=3043968&group_id=159897&atid=813673 which mentions a changed behaviour in aria2c 1.10.x, compared to 1.9.x. Indeed, when I try the mentioned options "--max-connection-per-server=4 --min-split-size=1M" aria2c behaves again as I would expect it:
...
Then I think that aria2c should be upgraded (via update repo, recomended update) on all openSUSE distros. That would reduce the problem of missing mirrors in the US.
I do not come to this conclusion. The new version behaves just like what David is seeing, unless used with additional options. The 1.9.5 I have however does the right thing. The question is why David's 1.9.x doesn't the right thing as well. It's not guaranteed that some update will make it work for him either. See my comment in the bug that David opened: https://bugzilla.novell.com/show_bug.cgi?id=641257#c2 Peter-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 09/23/2010 03:40 AM, Peter Pöml wrote:
Hi,
Le 22.09.2010 à 23:17, Carlos E. R. a écrit :
On Wednesday, 2010-09-22 at 11:54 +0200, Peter Pöml wrote:
I found http://sourceforge.net/tracker/?func=detail&aid=3043968&group_id=159897&atid=813673 which mentions a changed behaviour in aria2c 1.10.x, compared to 1.9.x. Indeed, when I try the mentioned options "--max-connection-per-server=4 --min-split-size=1M" aria2c behaves again as I would expect it:
...
Then I think that aria2c should be upgraded (via update repo, recomended update) on all openSUSE distros. That would reduce the problem of missing mirrors in the US.
I do not come to this conclusion. The new version behaves just like what David is seeing, unless used with additional options. The 1.9.5 I have however does the right thing. The question is why David's 1.9.x doesn't the right thing as well. It's not guaranteed that some update will make it work for him either. See my comment in the bug that David opened: https://bugzilla.novell.com/show_bug.cgi?id=641257#c2
Peter-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Peter, Thanks for helping fill in the bug report. As I stated, I wasn't up to speed on all the inner-workings of aria2c, but felt obligated to open the bug report. I'm still trying to get me head around the multiple server contact scheme. I'll have some additional time this weekend to look into it further. Thanks for your clarifications. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 09/22/2010 04:23 AM, Peter Pöml wrote:
The worst is, I don't know why this differs. Apart from the aria2c version everything should be exactly the same. In the latter case, I used the newest version, 1.10.3.
Which aria2 package version do you have installed?
Peter, Thank you, that was an awesome explanation. Currently I have: aria2-1.9.1-1.10.i586 -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (5)
-
Carlos E. R. -
David C. Rankin -
Patrick Shanahan -
Peter Pöml -
phanisvara das