[opensuse-buildservice] Problematic mirrors causing hangs while downloading packages with osc build

Hi, recently, we see hangs when osc downlaods packages during osc build. You might have seen it as well. Ctrl-C and restarting helps in those cases. I just debugged this a bit, and it seems that some mirrors are faulty. -H will show which mirrors osc contacts, but it doesn't show the individual following of redirects, so I had to dig a little deeper. I saw a hang with this server for example: python 1645 poeml 28u IPv4 43107270 TCP batavia510.suse.de:45944->babbage.hrz.tu-chemnitz.de:http (ESTABLISHED) With more debug output (in /usr/lib*/python2.4/urllib2.py), showing the redirects, I could immediately reproduce a hang with another server: % ftp http://ftp.tu-chemnitz.de/pub/linux/opensuse/distribution/10.2/repo/oss/suse... Requesting http://ftp.tu-chemnitz.de/pub/linux/opensuse/distribution/10.2/repo/oss/suse... 76% |************************************************************************************************************************************************* | 1194 KB 79.63 KB/s - stalled - Stalled since several minutes now. (No web cache in between in my case.) I fear, that we will need to do routine random download tests on mirrors in the future, to test if they are really functional....... Although it's not even clear if that would help, because once I tried the above another time, it worked right away :( We definitely need good debug output for this, to spot failing mirrors better. A hack is --- /usr/lib64/python2.4/urllib2.py 2007-05-16 18:47:58.000000000 +0200 +++ /dev/shm/urllib2.py 2007-05-16 18:47:44.987975358 +0200 @@ -526,6 +526,7 @@ else: return newurl = urlparse.urljoin(req.get_full_url(), newurl) + print newurl # XXX Probably want to forget about the state of the current # request, although that might interact poorly with other Peter -- Allen Gewalten zum Trutz sich erhalten. SUSE LINUX Products GmbH Research & Development

On Wed, May 16, 2007 at 06:49:09PM +0200, Dr. Peter Poeml wrote:
I saw a hang with this server for example: python 1645 poeml 28u IPv4 43107270 TCP batavia510.suse.de:45944->babbage.hrz.tu-chemnitz.de:http (ESTABLISHED)
With more debug output (in /usr/lib*/python2.4/urllib2.py), showing the redirects, I could immediately reproduce a hang with another server:
% ftp http://ftp.tu-chemnitz.de/pub/linux/opensuse/distribution/10.2/repo/oss/suse... Requesting http://ftp.tu-chemnitz.de/pub/linux/opensuse/distribution/10.2/repo/oss/suse... 76% |************************************************************************************************************************************************* | 1194 KB 79.63 KB/s - stalled -
Not another server, as I just noticed -- the same server. I also got a number of connection resets from it; I disabled it now. Peter -- Allen Gewalten zum Trutz sich erhalten. SUSE LINUX Products GmbH Research & Development

On Wed, May 16, 2007 at 07:15:14PM +0200, Dr. Peter Poeml wrote:
% ftp http://ftp.tu-chemnitz.de/pub/linux/opensuse/distribution/10.2/repo/oss/suse... Requesting http://ftp.tu-chemnitz.de/pub/linux/opensuse/distribution/10.2/repo/oss/suse... 76% |************************************************************************************************************************************************* | 1194 KB 79.63 KB/s - stalled -
After some debugging it turned out that a firewall at the university of Chemnitz was causing the problem. It is fixed now (and I'll reenable the mirror later). Peter -- Allen Gewalten zum Trutz sich erhalten. SUSE LINUX Products GmbH Research & Development
participants (1)
-
Dr. Peter Poeml