[Bug 590606] New: osc/fetch.py does not support authenticated URLs
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c0 Summary: osc/fetch.py does not support authenticated URLs Classification: openSUSE Product: openSUSE.org Version: unspecified Platform: Other OS/Version: Other Status: NEW Severity: Major Priority: P5 - None Component: BuildService AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: michael_e_brown@dell.com QAContact: adrian@novell.com Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.8) Gecko/20100216 Fedora/3.5.8-1.fc12 Firefox/3.5.8 "osc build" fails to pull down packages to do a local build. In my local obs instance, I have this URL: http://omb-obs-frontend.us.dell.com/build/Ubuntu:10.04/standard/x86_64/_repo... It requires authentication with a frontend user/pass combo (similar to _buildconfig). Downloading from that URL gets a 401 Unauthorized response, and osc does not retry with authentication. $ osc -vv -t -d -H build ub1004 x86_64 makeurl: http://omb-obs-frontend.us.dell.com ['source', 'home:michael_e_brown:ubuntu', '_meta'] [] -- GET http://omb-obs-frontend.us.dell.com/source/home:michael_e_brown:ubuntu/_meta GET http://omb-obs-frontend.us.dell.com/source/home:michael_e_brown:ubuntu/_meta send: 'GET /source/home:michael_e_brown:ubuntu/_meta HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: omb-obs-frontend.us.dell.com\r\nConnection: close\r\nUser-Agent: osc/0.125.5\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: text/xml; charset=utf-8 header: ETag: "1afe1103546acc0c4f7f0988acf24d02" header: X-Opensuse-APIVersion: 0.1.1 header: X-Runtime: 0.00153 header: Content-Length: 338 header: Cache-Control: private, max-age=0, must-revalidate header: Connection: close header: Date: Tue, 23 Mar 2010 19:16:30 GMT header: Server: lighttpd/1.4.25 Building srvadmin-omilcore.dsc for ub1004/x86_64 Getting buildinfo from server and store to /home/michael_e_brown/suse/home:michael_e_brown:ubuntu/srvadmin-omilcore/.osc/_buildinfo-ub1004-x86_64.xml makeurl: http://omb-obs-frontend.us.dell.com ['build', 'home:michael_e_brown:ubuntu', 'ub1004', 'x86_64', 'srvadmin-omilcore', '_buildinfo'] [] -- POST http://omb-obs-frontend.us.dell.com/build/home:michael_e_brown:ubuntu/ub1004... POST http://omb-obs-frontend.us.dell.com/build/home:michael_e_brown:ubuntu/ub1004... send: 'POST /build/home:michael_e_brown:ubuntu/ub1004/x86_64/srvadmin-omilcore/_buildinfo HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 271\r\nHost: omb-obs-frontend.us.dell.com\r\nContent-Type: application/octet-stream\r\nConnection: close\r\nUser-Agent: osc/0.125.5\r\n\r\n' send: 'Format: 1.0\nSource: srvadmin-omilcore\nBinary: srvadmin-omilcore, srvadmin-omilcore-dev\nArchitecture: all\nVersion: 6.3.0-1\nMaintainer: Brian Thomason <brian.thomason@canonical.com>\nHomepage: http://support.dell.com\nStandards-Version: 3.8.4\nBuild-Depends: debhelper (>> 7)\n' reply: 'HTTP/1.1 401 Unauthorized\r\n' header: Content-Type: application/xml; charset=utf-8 header: WWW-Authenticate: basic realm="Frontend login" header: X-Opensuse-Errorcode: unknown header: X-Runtime: 0.00155 header: Content-Length: 140 header: Cache-Control: no-cache header: Connection: close header: Date: Tue, 23 Mar 2010 19:16:31 GMT header: Server: lighttpd/1.4.25 send: 'POST /build/home:michael_e_brown:ubuntu/ub1004/x86_64/srvadmin-omilcore/_buildinfo HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 271\r\nHost: omb-obs-frontend.us.dell.com\r\nUser-Agent: osc/0.125.5\r\nConnection: close\r\nContent-Type: application/octet-stream\r\nAuthorization: Basic XXXXXXXXXX\r\n\r\n' send: 'Format: 1.0\nSource: srvadmin-omilcore\nBinary: srvadmin-omilcore, srvadmin-omilcore-dev\nArchitecture: all\nVersion: 6.3.0-1\nMaintainer: Brian Thomason <brian.thomason@canonical.com>\nHomepage: http://support.dell.com\nStandards-Version: 3.8.4\nBuild-Depends: debhelper (>> 7)\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: text/xml header: ETag: "de1a647386caac9541cfdb6eb93e98b7" header: X-Opensuse-APIVersion: 0.1.1 header: Content-Transfer-Encoding: binary header: Content-Disposition: inline header: X-Runtime: 0.54785 header: Content-Length: 25604 header: Cache-Control: private header: Connection: close header: Date: Tue, 23 Mar 2010 19:16:31 GMT header: Server: lighttpd/1.4.25 Getting buildconfig from server and store to /home/michael_e_brown/suse/home:michael_e_brown:ubuntu/srvadmin-omilcore/.osc/_buildconfig-ub1004-x86_64 makeurl: http://omb-obs-frontend.us.dell.com ['build', 'home:michael_e_brown:ubuntu', 'ub1004', '_buildconfig'] [] -- GET http://omb-obs-frontend.us.dell.com/build/home:michael_e_brown:ubuntu/ub1004... GET http://omb-obs-frontend.us.dell.com/build/home:michael_e_brown:ubuntu/ub1004... send: 'GET /build/home:michael_e_brown:ubuntu/ub1004/_buildconfig HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: omb-obs-frontend.us.dell.com\r\nConnection: close\r\nUser-Agent: osc/0.125.5\r\n\r\n' reply: 'HTTP/1.1 401 Unauthorized\r\n' header: Content-Type: application/xml; charset=utf-8 header: WWW-Authenticate: basic realm="Frontend login" header: X-Opensuse-Errorcode: unknown header: X-Runtime: 0.00157 header: Content-Length: 140 header: Cache-Control: no-cache header: Connection: close header: Date: Tue, 23 Mar 2010 19:16:31 GMT header: Server: lighttpd/1.4.25 send: 'GET /build/home:michael_e_brown:ubuntu/ub1004/_buildconfig HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: omb-obs-frontend.us.dell.com\r\nConnection: close\r\nAuthorization: Basic XXXXXXXXXXXXXXXXXX\r\nUser-Agent: osc/0.125.5\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: text/plain header: ETag: "472fa6b98fb15718569b343406694e6c" header: X-Opensuse-APIVersion: 0.1.1 header: Content-Transfer-Encoding: binary header: Content-Disposition: inline header: X-Runtime: 0.02781 header: Content-Length: 9233 header: Cache-Control: private header: Connection: close header: Date: Tue, 23 Mar 2010 19:16:31 GMT header: Server: lighttpd/1.4.25 Updating cache of required packages 100.0% cache miss. 0/195 dependencies cached. URLs to try for package 'bash': file:///var/tmp/osbuild-packagecache/Ubuntu:10.04/standard/amd64/bash-4.1-1ubuntu2.amd64.deb/ http://omb-obs-repo.us.dell.com//Ubuntu:/10.04/standard/amd64/bash-4.1-1ubun... http://omb-obs-frontend.us.dell.com/build/Ubuntu:10.04/standard/x86_64/_repo... Trying openSUSE Build Service server for bash (Ubuntu:10.04), since it is not on omb-obs-repo.us.dell.com. Trying openSUSE Build Service server for bash (Ubuntu:10.04), since it is not on omb-obs-frontend.us.dell.com. Error: No more mirrors to try. Failed to retrieve bash-4.1-1ubuntu2.amd64.deb from the following locations (in order): file:///var/tmp/osbuild-packagecache/Ubuntu:10.04/standard/amd64/bash-4.1-1ubuntu2.amd64.deb/ http://omb-obs-repo.us.dell.com//Ubuntu:/10.04/standard/amd64/bash-4.1-1ubun... http://omb-obs-frontend.us.dell.com/build/Ubuntu:10.04/standard/x86_64/_repo... Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/osc/babysitter.py", line 46, in run return prg.main() File "/usr/lib/python2.6/site-packages/osc/cmdln.py", line 308, in main return self.cmd(args) File "/usr/lib/python2.6/site-packages/osc/cmdln.py", line 331, in cmd retval = self.onecmd(argv) File "/usr/lib/python2.6/site-packages/osc/cmdln.py", line 447, in onecmd return self._dispatch_cmd(handler, argv) File "/usr/lib/python2.6/site-packages/osc/cmdln.py", line 1177, in _dispatch_cmd return handler(argv[0], opts, *args) File "/usr/lib/python2.6/site-packages/osc/commandline.py", line 3227, in do_build return osc.build.main(opts, args) File "/usr/lib/python2.6/site-packages/osc/build.py", line 532, in main fetcher.run(bi) File "/usr/lib/python2.6/site-packages/osc/fetch.py", line 143, in run self.fetch(i) File "/usr/lib/python2.6/site-packages/osc/fetch.py", line 91, in fetch sys.exit(1) SystemExit: 1 Reproducible: Always Steps to Reproduce: 1. 2. 3. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c Michael Brown <michael_e_brown@dell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team-screening@forge.pr |suse-tux@gmx.de |ovo.novell.com | -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c1 Marcus Hüwe <suse-tux@gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |suse-tux@gmx.de --- Comment #1 from Marcus Hüwe <suse-tux@gmx.de> 2010-03-24 10:50:31 UTC --- (In reply to comment #0)
It requires authentication with a frontend user/pass combo (similar to _buildconfig). Downloading from that URL gets a 401 Unauthorized response, and osc does not retry with authentication.
It should retry after it encounters a 401.
Trying openSUSE Build Service server for bash (Ubuntu:10.04), since it is not on omb-obs-repo.us.dell.com.
Don't you get any debug messages here or did you forgot to copy and paste them? Normally there should be 2 requests: the first request should fail with status code 401 and the second request should succeed. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c2 --- Comment #2 from Michael Brown <michael_e_brown@dell.com> 2010-03-24 14:41:17 UTC --- It only retries _buildinfo after 401. In osc/fetch.py, if it gets a 401 (at least on my system), it never retries. Also, note that _buildinfo (osc/core.py) and the RPM/DEB fetching (osc/fetch.py) use completely different code paths. I pasted the entire output. Also: $ rpm -q osc osc-0.125.5-2.2.x86_64 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
It only retries _buildinfo after 401. In osc/fetch.py, if it gets a 401 (at least on my system), it never retries. But where do you "see" that it gets a 401? The pasted output doesn't indicate
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c3 --- Comment #3 from Marcus Hüwe <suse-tux@gmx.de> 2010-03-24 16:55:04 UTC --- (In reply to comment #2) that there was 401 response or a request at all. For instance I get the following: URLs to try for package 'libc6-dev': file:///var/tmp/osbuild-packagecache/Debian:Etch/standard/i386/libc6-dev_2.3.6.ds1-13etch9+b1_i386.deb/ http://linux.linux-network/repositories/Debian:/Etch/standard/i386/libc6-dev... http://0.0.0.0:3000/build/Debian:Etch/standard/i586/_repository/libc6-dev Trying openSUSE Build Service server for libc6-dev (Debian:Etch), not found at linux.linux-network. send: 'GET /build/Debian%3AEtch/standard/i586/_repository/libc6-dev HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 0.0.0.0:3000\r\nConnection: close\r\nUser-Agent: osbuild/0.1\r\n\r\n' reply: 'HTTP/1.1 401 Unauthorized \r\n' header: Connection: close header: Content-Type: application/xml; charset=utf-8 header: X-Opensuse-Apiversion: 1.8 header: WWW-Authenticate: basic realm="API login" header: Date: Wed, 24 Mar 2010 16:51:30 GMT header: Server: WEBrick/1.3.1 (Ruby/1.8.7/2008-08-11) header: X-Runtime: 2 header: Content-Length: 140 header: X-Opensuse-Errorcode: unknown header: Cache-Control: no-cache send: 'GET /build/Debian%3AEtch/standard/i586/_repository/libc6-dev HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 0.0.0.0:3000\r\nConnection: close\r\nAuthorization: Basic <FOO>\r\nUser-Agent: osbuild/0.1\r\n\r\n' reply: 'HTTP/1.1 200 OK \r\n' header: Etag: "6a7a0aa76356e646127dbe82a7f147b2" header: Connection: close header: Content-Type: application/x-debian-package header: Content-Transfer-Encoding: binary header: X-Opensuse-Apiversion: 1.8 header: Date: Wed, 24 Mar 2010 16:51:30 GMT header: Server: WEBrick/1.3.1 (Ruby/1.8.7/2008-08-11) header: X-Runtime: 248 header: Content-Length: 2717858 header: Content-Disposition: inline header: Cache-Control: private (Debian:Etch) libc6-dev_2.3.6.ds1-13etch9+b1_i386.deb 100% |===============================================| 2.6 MB 00:00 The strange thing is that such a log is missing in your paste.
Also, note that _buildinfo (osc/core.py) and the RPM/DEB fetching (osc/fetch.py) use completely different code paths.
Yes fetch.py uses urlgrabber to fetch the binaries and urlgrabber uses the urllib2 openers. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c4 --- Comment #4 from Michael Brown <michael_e_brown@dell.com> 2010-03-24 17:02:50 UTC --- I see the 401 response in a couple places: -- The server logs. -- strace output which was not attached. Osc doesnt give any debug http output for the failed deb download on my system. I pasted the full output from osc. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c5 --- Comment #5 from Marcus Hüwe <suse-tux@gmx.de> 2010-03-26 09:17:03 UTC --- Which version of python-base and python-urlgrabber do you have installed? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c6 --- Comment #6 from Michael Brown <michael_e_brown@dell.com> 2010-03-26 14:51:14 UTC --- Fedora 12 $ rpm -q python python-urlgrabber python-2.6.2-4.fc12.x86_64 python-urlgrabber-3.9.1-4.fc12.noarch -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c7 --- Comment #7 from Marcus Hüwe <suse-tux@gmx.de> 2010-03-26 15:38:18 UTC --- Ok it's a problem with your python-urlgrabber version. I'll have a look at it. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c8 Sven Herzberg <herzi@gnome-de.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |herzi@gnome-de.org --- Comment #8 from Sven Herzberg <herzi@gnome-de.org> 2010-04-05 14:53:12 UTC --- Apparently, I'm hitting this, too… -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c9 --- Comment #9 from Marcus Hüwe <suse-tux@gmx.de> 2010-04-07 01:22:24 UTC --- I just fixed it in git master and submitted a new package to the openSUSE:Tools:Unstable repo (unfortunately we don't have a Fedora12 build target in this project yet). Does the fix work for you? -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c10 --- Comment #10 from Sven Herzberg <herzi@gnome-de.org> 2010-04-07 13:02:17 UTC --- (In reply to comment #9)
I just fixed it in git master and submitted a new package to the openSUSE:Tools:Unstable repo (unfortunately we don't have a Fedora12 build target in this project yet). Does the fix work for you?
Yes:
$ osc build Ubuntu_9.10 gutachter.dsc Building gutachter.dsc for Ubuntu_9.10/i586 Getting buildinfo from server and store to /home/herzi/Hacking/Packaging/home:herzi/gutachter/.osc/_buildinfo-Ubuntu_9.10-i586.xml Getting buildconfig from server and store to /home/herzi/Hacking/Packaging/home:herzi/gutachter/.osc/_buildconfig-Ubuntu_9.10-i586 Updating cache of required packages 100.0% cache miss. 0/142 dependencies cached.
Trying openSUSE Build Service server for bash (Ubuntu:9.10), not found at download.opensuse.org. (Ubuntu:9.10) bash_4.0-5ubuntu2_i386.deb 100% |===========================| 642 kB 00:01 …
Thank you very much. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=590606 http://bugzilla.novell.com/show_bug.cgi?id=590606#c11 Marcus Hüwe <suse-tux@gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution| |FIXED --- Comment #11 from Marcus Hüwe <suse-tux@gmx.de> 2010-04-07 15:15:59 UTC --- Ok - thanks for testing! I think we can close this bug as FIXED (if not please reopen). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply@novell.com