[opensuse] More SLOOOWW redirectors...
Marcus, et. al. We have another trickling update using the download.opensuse.org redirector system. I've looked in y2log, but don't know what I'm reading. It looks like the server is one of these. (if this actually tells you anything) 2010-03-03 01:49:15 <1> alchemy(10373) [zypp] RpmDb.cc(doInstallPackage):1974 RpmDb::installPackage(/var/cache/zypp/packages/gnome1/x86_64/imlib-32bit-1.9.14-408.1.x86_64.rpm,12) 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #0: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #1: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #2: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #3: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #4: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #5: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #6: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #7: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #8: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #9: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #10: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #11: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #12: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #13: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #14: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #15: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #16: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #17: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #18: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #19: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #20: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #21: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #22: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #23: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #24: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #25: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #26: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #27: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #28: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #29: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #30: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #31: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #32: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #33: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #34: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #35: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #36: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #37: [16410624] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #38: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #39: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #40: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #41: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #42: [] 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:1046 src #43: [] 2010-03-03 01:49:17 <1> alchemy(10373) [zypp] PackageProvider.cc(providePackage):95 provide Package (82788)imlib-1.9.14-408.1.x86_64(gnome1) 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:739 Package 'imlib' is remote 2010-03-03 01:49:17 <1> alchemy(10373) [zypp] RepoProvideFile.cc(provideFile):204 [1]./x86_64/imlib-1.9.14-408.1.x86_64.rpm{133.0 K|sha1-a41ce39b6377dbf428994a7e911f4e4187d2b03d} 2010-03-03 01:49:17 <1> alchemy(10373) [zypp] RepoProvideFile.cc(provideFile):229 Added cache path /var/cache/zypp/packages/gnome1 If this isn't the right info to identify to slow server, tell me what you need to see and I'll get it for you. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 03/03/2010 08:53 AM, David C. Rankin wrote:
Marcus, et. al.
We have another trickling update using the download.opensuse.org redirector system. I've looked in y2log, but don't know what I'm reading. It looks like the server is one of these. (if this actually tells you anything)
2010-03-03 01:49:15 <1> alchemy(10373) [zypp] RpmDb.cc(doInstallPackage):1974 RpmDb::installPackage(/var/cache/zypp/packages/gnome1/x86_64/imlib-32bit-1.9.14-408.1.x86_64.rpm,12) cut useless lines 2010-03-03 01:49:17 <1> alchemy(10373) [zypp] PackageProvider.cc(providePackage):95 provide Package (82788)imlib-1.9.14-408.1.x86_64(gnome1) 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:739 Package 'imlib' is remote 2010-03-03 01:49:17 <1> alchemy(10373) [zypp] RepoProvideFile.cc(provideFile):204 [1]./x86_64/imlib-1.9.14-408.1.x86_64.rpm{133.0 K|sha1-a41ce39b6377dbf428994a7e911f4e4187d2b03d} 2010-03-03 01:49:17 <1> alchemy(10373) [zypp] RepoProvideFile.cc(provideFile):229 Added cache path /var/cache/zypp/packages/gnome1
If this isn't the right info to identify to slow server, tell me what you need to see and I'll get it for you.
Well, it isn't. The log is cut at the point where the download just starts and does not show where it ends. And it all happens in the very same second. Maybe you could do $ time curl --max-redirs 0 <urlofsomerpm> ------------------------------------------------------------------- $ time curl --max-redirs 0 http://download.opensuse.org/distribution/11.2/repo/oss/suse/i586/PackageKit... <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>302 Found</title> </head><body> <h1>Found</h1> <p>The document has moved <a href="http://ftp.linux.cz/pub/linux/opensuse/distribution/11.2/repo/oss/suse/i586/PackageKit-0.5.1-3.7.i586.rpm">here</a>.</p> <hr> <address>Apache/2.2.14 (Linux/SUSE) Server at download.opensuse.org Port 80</address> </body></html> real 0m0.155s user 0m0.004s sys 0m0.000s ------------------------------------------------------------------- second try ------------------------------------------------------------------- $ time curl --max-redirs 0 http://download.opensuse.org/distribution/11.2/repo/oss/suse/i586/PackageKit... ... real 0m4.282s user 0m0.008s sys 0m0.008s ------------------------------------------------------------------- Hmm. that was pretty slow. Let's try verbose output: ------------------------------------------------------------------- $ time curl -v --max-redirs 0 http://download.opensuse.org/distribution/11.2/repo/oss/suse/i586/PackageKit... * About to connect() to download.opensuse.org port 80 (#0) * Trying 195.135.221.130... connected * Connected to download.opensuse.org (195.135.221.130) port 80 (#0)
GET /distribution/11.2/repo/oss/suse/i586/PackageKit-0.5.1-3.7.i586.rpm HTTP/1.1 User-Agent: curl/7.19.6 (x86_64-unknown-linux-gnu) libcurl/7.19.6 OpenSSL/0.9.8k zlib/1.2.3 libidn/1.10 Host: download.opensuse.org
Accept: */*
< HTTP/1.1 302 Found < Date: Thu, 04 Mar 2010 07:30:06 GMT < Server: Apache/2.2.14 (Linux/SUSE) < X-Prefix: 195.47.0.0/17 < X-AS: 2819 < X-MirrorBrain-Mirror: ftp.linux.cz < X-MirrorBrain-Realm: country < Location: http://ftp.linux.cz/pub/linux/opensuse/distribution/11.2/repo/oss/suse/i586/... < Content-Length: 380 < Content-Type: text/html; charset=iso-8859-1 < <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>302 Found</title> </head><body> <h1>Found</h1> <p>The document has moved <a href="http://ftp.linux.cz/pub/linux/opensuse/distribution/11.2/repo/oss/suse/i586/PackageKit-0.5.1-3.7.i586.rpm">here</a>.</p> <hr> <address>Apache/2.2.14 (Linux/SUSE) Server at download.opensuse.org Port 80</address> </body></html> * Connection #0 to host download.opensuse.org left intact * Closing connection #0 real 0m0.482s user 0m0.008s sys 0m0.004s ------------------------------------------------------------------- Looks like these 0m0.482s really was waiting for the response from the redirector. Most of the time the response is quick, but some requests get as much as few seconds response time (like in the second example). It might be worth to check this also with aria2c. -- cheers, jano Ján Kupec YaST team ---------------------------------------------------------(PGP)--- Key ID: 637EE901 Fingerprint: 93B9 C79B 2D20 51C3 800B E09B 8048 46A6 637E E901 ---------------------------------------------------------(IRC)--- Server: irc.freenode.net Nick: jniq Channels: #zypp #yast #suse #susecz ---------------------------------------------------------(EOF)---
Hi, Am 04.03.2010 um 08:35 schrieb Jano Kupec:
On 03/03/2010 08:53 AM, David C. Rankin wrote:
Marcus, et. al.
We have another trickling update using the download.opensuse.org redirector system. I've looked in y2log, but don't know what I'm reading. It looks like the server is one of these. (if this actually tells you anything)
tcpdump would help. I'm not sure whether the yast log would be useful. (Might be.)
2010-03-03 01:49:15 <1> alchemy(10373) [zypp] RpmDb.cc(doInstallPackage):1974 RpmDb::installPackage(/var/cache/zypp/packages/gnome1/x86_64/imlib-32bit-1.9.14-408.1.x86_64.rpm,12) cut useless lines 2010-03-03 01:49:17 <1> alchemy(10373) [zypp] PackageProvider.cc(providePackage):95 provide Package (82788)imlib-1.9.14-408.1.x86_64(gnome1) 2010-03-03 01:49:17 <1> alchemy(10373) [YCP] SlideShow.ycp:739 Package 'imlib' is remote 2010-03-03 01:49:17 <1> alchemy(10373) [zypp] RepoProvideFile.cc(provideFile):204 [1]./x86_64/imlib-1.9.14-408.1.x86_64.rpm{133.0 K|sha1-a41ce39b6377dbf428994a7e911f4e4187d2b03d} 2010-03-03 01:49:17 <1> alchemy(10373) [zypp] RepoProvideFile.cc(provideFile):229 Added cache path /var/cache/zypp/packages/gnome1
If this isn't the right info to identify to slow server, tell me what you need to see and I'll get it for you.
Well, it isn't. The log is cut at the point where the download just starts and does not show where it ends. And it all happens in the very same second.
Maybe you could do $ time curl --max-redirs 0 <urlofsomerpm>
------------------------------------------------------------------- $ time curl --max-redirs 0 http://download.opensuse.org/distribution/11.2/repo/oss/suse/i586/PackageKit... <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>302 Found</title> </head><body> <h1>Found</h1> <p>The document has moved <a href="http://ftp.linux.cz/pub/linux/opensuse/distribution/11.2/repo/oss/suse/i586/PackageKit-0.5.1-3.7.i586.rpm">here</a>.</p> <hr> <address>Apache/2.2.14 (Linux/SUSE) Server at download.opensuse.org Port 80</address> </body></html>
real 0m0.155s user 0m0.004s sys 0m0.000s -------------------------------------------------------------------
second try
------------------------------------------------------------------- $ time curl --max-redirs 0 http://download.opensuse.org/distribution/11.2/repo/oss/suse/i586/PackageKit... ... real 0m4.282s user 0m0.008s sys 0m0.008s -------------------------------------------------------------------
Hmm. that was pretty slow. Let's try verbose output:
------------------------------------------------------------------- $ time curl -v --max-redirs 0 http://download.opensuse.org/distribution/11.2/repo/oss/suse/i586/PackageKit...
* About to connect() to download.opensuse.org port 80 (#0)
* Trying 195.135.221.130... connected
* Connected to download.opensuse.org (195.135.221.130) port 80 (#0)
GET /distribution/11.2/repo/oss/suse/i586/PackageKit-0.5.1-3.7.i586.rpm HTTP/1.1 User-Agent: curl/7.19.6 (x86_64-unknown-linux-gnu) libcurl/7.19.6 OpenSSL/0.9.8k zlib/1.2.3 libidn/1.10 Host: download.opensuse.org
Accept: */*
< HTTP/1.1 302 Found
< Date: Thu, 04 Mar 2010 07:30:06 GMT
< Server: Apache/2.2.14 (Linux/SUSE)
< X-Prefix: 195.47.0.0/17
< X-AS: 2819
< X-MirrorBrain-Mirror: ftp.linux.cz
< X-MirrorBrain-Realm: country
< Location: http://ftp.linux.cz/pub/linux/opensuse/distribution/11.2/repo/oss/suse/i586/...
< Content-Length: 380
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://ftp.linux.cz/pub/linux/opensuse/distribution/11.2/repo/oss/suse/i586/PackageKit-0.5.1-3.7.i586.rpm">here</a>.</p>
<hr>
<address>Apache/2.2.14 (Linux/SUSE) Server at download.opensuse.org Port 80</address> </body></html>
* Connection #0 to host download.opensuse.org left intact
* Closing connection #0
real 0m0.482s user 0m0.008s sys 0m0.004s -------------------------------------------------------------------
Looks like these 0m0.482s really was waiting for the response from the redirector. Most of the time the response is quick, but some requests get as much as few seconds response time (like in the second example).
That looks to me as if SUSE has changed the database server / kernel configuration of the database behind the redirector. I know the symptom well, because I tuned it in the beginning and it ran perfectly for a long enough time. That said, it's also entirely possible that there's another reason for this; however, Apache doesn't have to do anything else than waiting for the database, and database stalls with PostgreSQL are a well-known misconfiguration issue. (Tuning PostgreSQL (resp. the operating system it runs on) for high concurrency (as needed with web apps) isn't as easy as with MySQL.) If you see most requests being handled in milliseconds, but some (every 5th or 10th request maybe?) takes seconds instead, that's most certainly the reason. If so, it'd be best to contact admin at opensuse.org to fix it. (Cc'ed.)
It might be worth to check this also with aria2c.
You don't need to, because for the server it doesn't make a difference whether it sends a redirect or a metalink. It's the same. curl -sI <url> is enough to reproduce it. Locally (talking to the database), a simple SELECT statement on the mirror list would show the same delays. The redirector (Apache) is just a consumer on the functionality, so if the database blocks, Apache needs to wait for it. Obviously, with a web application, a blocking database is deadly to the user experience. It may be possible that the stalls don't occur at any time of the day, but only during certain periods -- then they are most likely related to write activity due to mirror scanning. Peter-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (3)
-
David C. Rankin -
Jano Kupec -
Peter Pöml