Mailinglist Archive: opensuse (1839 mails)

< Previous Next >
Re: [opensuse] One repo in the pool killing updates - probably utexas.edu - can it be fixed?
  • From: Peter Pöml <peter@xxxxxxxx>
  • Date: Wed, 22 Sep 2010 11:23:15 +0200
  • Message-id: <DA9287A4-0966-47BF-908F-BD2474A5C00B@xxxxxxxx>
Hi David,

Le 21.09.2010 à 22:30, David C. Rankin a écrit :

On 09/21/2010 04:24 AM, Peter Pöml wrote:
David,

Le 21.09.2010 à 00:18, David C. Rankin a écrit :
Guys,

It is literally like Russian-Roulette updating openSuSE with the mirror
pool you get with download.opensuse.org. There is at least 1 maybe more
mirrors that give no more than 1-4K/s download speed. (utexas.edu is a know
problem repo). This is getting ridiculous. I have had to kill 3-4 updates
in the past 2-3 days when I hit one of these mirrors. It would take more
than an hour to download an 800k file.

I have searched zypper.log and can't find the problem mirror this time.
Where to look? Also, this has been a problem for more than a year, why
aren't these mirrors being dropped or fixed? I end up having to hard-code
the mirror I want just to get reasonable consistent updates. Something
should be done to correct this.

Two questions:

Have you reported it to those who can fix it?
(I don't consider this mailing list as "those who can fix it". So I'm not
sure what you expect by posting here. Does it help others?)

If you didn't: *do* you know where to report it?

If you did (which is what I assume), and it's not fixed after such a long
time (a year!), I think that makes the openSUSE project look pretty bad. :-(

Peter

Brother Peter,

Yes on all 3. Question: With yast, I could look in the yast log and
find the
offending repo IP and hostname, can I find that same info if I use zypper? If
so
where? I've dug through zypper.log but struck out.... and grepped the
remainder
of /var/log for the package name I was updating when the update hung on the
bad
repo. Is there anywhere else to look to help narrow it down?

Hm, makes me sad to hear that this problem is not being taken care of, and I
can well understand frustration following that.

It's probably packages from the build service which are the problem. There are
two "classes" of packages in the build service: those that are more popular and
are mirrored by 4 mirrors in the US, and those that are a little less popular
and are mirrored only by 2 mirrors. This includes all packages in the "home:"
namespace, but also many others.

Recently, one US mirror stopped mirroring the build service, so the latter
packages are not properly mirrored anymore in the US. The remaining two mirrors
in the US are: University of Texas, and a mirror operated by susestudio.com.
From what I hear, both are very slow. I'd think that this is the #1 reason that
makes downloads very slow for you.

All other mirrors are in Europe (4xde, gb, pt, nl, ro). I don't know if the US
mirrors are slow as such, or if they are just flooded with too many requests
because they are the only ones.

Anyway, the situation would improve if somebody in the US would volunteer and
contribute a mirror. However, nobody there seems to be interested enough to
step up and contribute spare time and resources for this purpose. There may
still be possibilities. I have found that chances in finding new mirrors can be
considerably improved by actively approaching existing mirrors and ask them for
help.

Beyond that, if one of the US mirrors is not performing well, it should be
taken out of the rotation. But maybe they do perform well but are just
overloaded, due to lack of other mirrors. I can't tell. (I am assuming that
utexas.edu is indeed the recurrent problem for you.)


Having stated all that, maybe you can confirm some observations? I guess you
are using a more recent openSUSE version, like 11.3? As you know, since 11.2,
zypper/YaST use a download backend which should work around about exactly the
above issues, by contacting several mirrors in parallel. Maybe that is not
configured well.

Take
repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm as
an example for a package which is only on few mirrors. Here are they, as seen
from the US:

% curl -s
'http://download.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm.metalink?country=us'
| grep location
<url type="http" location="us"
preference="100">http://ftp.utexas.edu/opensuse/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm</url>
<url type="http" location="us"
preference="99">http://130.57.19.201/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm</url>
<url type="http" location="de"
preference="98">http://ftp.halifax.rwth-aachen.de/opensuse/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm</url>
<url type="http" location="de"
preference="97">http://ftp5.gwdg.de/pub/opensuse/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm</url>
<url type="http" location="de"
preference="96">http://widehat.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm</url>
<url type="http" location="ro"
preference="95">http://ftp.roedu.net/mirrors/opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm</url>


Using aria2c directly, we can see what it does with that:

% aria2c --log=- --log-level=debug '--user-agent=ZYpp 7.7.5 (aria2 version
1.9.1) openSUSE-11.3-x86_64' --summary-interval=1 --follow-metalink=mem
--check-integrity=true --file-allocation=none
--server-stat-of=/home/poeml/.aria2/aria2.stats
--server-stat-if=/home/poeml/.aria2/aria2.stats --uri-selector=adaptive
--use-head=false --max-tries=5 --max-concurrent-downloads=2 --header=Pragma:
--connect-timeout=180
'http://download.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm.metalink?country=us'
| grep Host
Host: download.opensuse.org
Host: widehat.opensuse.org
Host: ftp.halifax.rwth-aachen.de
Host: ftp5.gwdg.de
Host: ftp.roedu.net
Host: 130.57.19.201

It contacts 5 servers, which is the built-in default for concurrency. The job
finishes after 1 or 2 seconds for me, which is about the same time to download
the package from a single source in this case.

Maybe if you can try that out, do you get a working download with that? If you
see a long hang with that, it might be due to this bug:
https://bugzilla.novell.com/show_bug.cgi?id=555803


Sigh. When testing the same as above from the US, aria2c does not contact 5
mirrors; it contacts only one, and if that is utexas.edu, the download takes a
minute instead of 2-4 seconds. The log shows that only one server is contacted,
and it is reproducible:

% aria2c --log=- --log-level=debug '--user-agent=ZYpp 7.7.5 (aria2 version
1.9.1) openSUSE-11.3-x86_64' --summary-interval=1 --follow-metalink=mem
--check-integrity=true --file-allocation=none
--server-stat-of=/home/poeml/.aria2/aria2.stats
--server-stat-if=/home/poeml/.aria2/aria2.stats --uri-selector=adaptive
--use-head=false --max-tries=5 --max-concurrent-downloads=2 --header=Pragma:
--connect-timeout=180
'http://download.opensuse.org/repositories/home:/-miska-/openSUSE_11.3/x86_64/mapnik-0.7.1-1.1.x86_64.rpm.metalink?country=us'
| grep Host
Host: download.opensuse.org
Host: 130.57.19.201

The worst is, I don't know why this differs. Apart from the aria2c version
everything should be exactly the same. In the latter case, I used the newest
version, 1.10.3.

Which aria2 package version do you have installed?

Peter--
To unsubscribe, e-mail: opensuse+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: opensuse+help@xxxxxxxxxxxx

< Previous Next >