[Bug 214983] New: https proxy support broken in python 2.5's urllib2?
https://bugzilla.novell.com/show_bug.cgi?id=214983 Summary: https proxy support broken in python 2.5's urllib2? Product: openSUSE 10.2 Version: Beta 1 Platform: Other OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem AssignedTo: jmatejek@novell.com ReportedBy: poeml@novell.com QAContact: qa@suse.de Using urllib2 via http proxy works fine, but not via https proxy: poeml@shannon ~ % python -c "import urllib2; urllib2.urlopen('http://www.sf.net').read()" poeml@shannon ~ % python -c "import urllib2; urllib2.urlopen('https://www.sf.net').read()" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.5/urllib2.py", line 121, in urlopen return _opener.open(url, data) File "/usr/lib/python2.5/urllib2.py", line 374, in open response = self._open(req, data) File "/usr/lib/python2.5/urllib2.py", line 392, in _open '_open', req) File "/usr/lib/python2.5/urllib2.py", line 353, in _call_chain result = func(*args) File "/usr/lib/python2.5/urllib2.py", line 668, in <lambda> meth(r, proxy, type)) File "/usr/lib/python2.5/urllib2.py", line 691, in proxy_open return self.parent.open(req) File "/usr/lib/python2.5/urllib2.py", line 380, in open response = meth(req, response) File "/usr/lib/python2.5/urllib2.py", line 491, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.5/urllib2.py", line 418, in error return self._call_chain(*args) File "/usr/lib/python2.5/urllib2.py", line 353, in _call_chain result = func(*args) File "/usr/lib/python2.5/urllib2.py", line 499, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 400: Bad Request [1] 31530 exit 1 python -c "import urllib2; urllib2.urlopen('https://www.sf.net').read()" It could be either the proxy, or python, which is doing something wrong. But on 10.1 it works. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #1 from mhopf@novell.com 2006-10-25 10:20 MST ------- Happens on 10.1 also for me. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #2 from mhopf@novell.com 2006-10-25 10:23 MST ------- Forgot to mention my python version on ivanova: # rpm -q python python-2.4.2-18.3 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 sndirsch@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mhopf@novell.com ------- Comment #3 from sndirsch@novell.com 2006-10-25 11:19 MST ------- So this is at least *no* regression. $https_proxy is set to "http://proxy.suse.de:3128". I'm even not sure if squid (on proxy.suse.de) is configured/capable as https proxy or I'm trying to use the wrong port. Maybe this setting did never work and I didn't notice it, because I never did use it before the buildservice switched to https and osc tries to use it now. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 jmatejek@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED ------- Comment #4 from jmatejek@novell.com 2006-10-25 11:32 MST ------- ok, can anyone with a functional https proxy confirm this? i don't have one at hand (or maybe i do, but i don't know about it) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #5 from poeml@novell.com 2006-10-26 02:51 MST ------- Oh, I indeed overlooked that I didn't check with HTTPS proxy on 10.0. The HTTPS proxy "proxy.suse.de:3128" works, as can be seen here (149.44.160.33 is its IP): strace -econnect w3m -dump 'https://imap.suse.de' > /dev/null connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0 connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(3128), sin_addr=inet_addr("149.44.160.33")}, 16) = 0 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.10.0.1")}, 28) = 0 Process 13023 detached Running this command: https_proxy=http://proxy.suse.de:3128 python -c "import urllib2; urllib2.urlopen('https://imap.suse.de').read()" and using ssldump, one can see the traffic: # ssldump -X -d -A -i eth0 host proxy.suse.de and port 3128 New TCP connection #1: batavia510.suse.de(55683) <-> siddhartha.suse.de(3128) 0.0007 (0.0007) C>S --------------------------------------------------------------- GET https://imap.suse.de HTTP/1.1 Accept-Encoding: identity Host: imap.suse.de Connection: close User-agent: Python-urllib/2.4 --------------------------------------------------------------- 0.0033 (0.0025) S>C --------------------------------------------------------------- HTTP/1.0 400 Bad Request Server: squid/2.5.STABLE5 Mime-Version: 1.0 Date: Thu, 26 Oct 2006 07:59:25 GMT Content-Type: text/html Content-Length: 1056 Expires: Thu, 26 Oct 2006 07:59:25 GMT X-Squid-Error: ERR_UNSUP_REQ 0 X-Cache: MISS from proxy.suse.de X-Cache-Lookup: MISS from proxy.suse.de:3128 Proxy-Connection: close <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1"> <TITLE>ERROR: The requested URL could not be retrieved</TITLE> <STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE> </HEAD><BODY> <H1>ERROR</H1> <H2>The requested URL could not be retrieved</H2> <HR noshade size="1px"> <P> While trying to retrieve the URL: <A HREF="https://imap.suse.de">https://imap.suse.de</A> <P> The following error was encountered: <UL> <LI> <STRONG> Unsupported Request Method and Protocol </STRONG> </UL> <P> Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request. <P>Your cache administrator is <A HREF="mailto:sysadmin@suse.de">sysadmin@suse.de</A>. <BR clear="all"> <HR noshade size="1px"> <ADDRESS> Generated Thu, 26 Oct 2006 07:59:25 GMT by proxy.suse.de (squid/2.5.STABLE5) </ADDRESS> </BODY></HTML> --------------------------------------------------------------- 1 0.0033 (0.0000) S>C TCP FIN 1 0.0166 (0.0133) C>S TCP RST Instead, it needs to look like this: --------------------------------------------------------------- CONNECT imap.suse.de:443 HTTP/1.0 User-Agent: w3m/0.5.1 Accept: text/*, image/*, audio/*, application/*, video/*, off_image/* Accept-Encoding: gzip, compress, bzip, bzip2, deflate Accept-Language: en; q=1.0 Host: imap.suse.de --------------------------------------------------------------- But: # rpm -ql python | xargs grep "\<CONNECT\>" [1] 13171 done rpm -ql python | 13172 exit 123 xargs grep "\<CONNECT\>" So I *don't* think, that python's urllib2/httplib/socket modules *have* support for requests via HTTPS proxies. There is a recent posting which describes the same: http://mail.python.org/pipermail/python-list/2006-August/355285.html I wonder why urllib2 honours the https_proxy variable at all, even though it doesn't know how to deal with it. This patch http://mail.python.org/pipermail/python-dev/2005-December/058817.html supposedly fixes authentication for https proxies, which sounds like https proxies *should* be supported. Not sure if that patch in in our python release. Just for reference, here is some extension which adds a HTTPS Proxy handler: http://richard.jones.name/google-hacks/gmail-filesystem/pyOpenSSLProxy-0.1.t... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #6 from jmatejek@novell.com 2006-10-26 09:43 MST ------- i'm a bit confused now. so, urllib2 is supposed to connect to the proxy with SSL, right? and ask to CONNECT to https server. but what it does is that it connects to the proxy and asks for "https://", which makes no sense anyway. about the patch, we don't have it separately, but it is included in python 2.5. i'll check whether there is a way to fix it as bug, or i'll tell you if there is no such feature ;e) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #7 from jmatejek@novell.com 2006-10-26 10:20 MST ------- okay, confirmed that the feature is really -not- there, it's missing in the httplib module. http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195 this tells you how to build a HTTPS proxy with CONNECT for urllib2, if you need it right away. However, I don't think it is a good idea to add this "patch" right away. I'd rather wait for results of the posting Peter mentioned. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #8 from poeml@novell.com 2006-10-27 01:01 MST ------- I agree! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 maw@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |maw@novell.com ------- Comment #9 from maw@novell.com 2006-10-27 14:42 MST ------- This problem has existed forever. I worked around it for #141368 by using wget to do the heavy lifting, although that's not an adequate general-purpose solution. :/ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #10 from jmatejek@novell.com 2006-11-01 06:43 MST ------- more accordingly, the feature has never been there in the first place ;e) it's known for quite a long time, and nobody from the python community had bothered to fix it, so i guess it'll stay as is. what do you think, should we ship fixed version? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #11 from poeml@novell.com 2006-11-10 16:24 MST ------- Let's not deviate from upstream, it will only make it harder to get someone motivated to get the bug fixed. But I would strongly support a patch which makes python print out some error message, instead of pretending to use the https proxy but sending an illegible request. I'm not even sure why it does that, when I skimmed the code I didn't spot the place where the environment variable is evaluated. It should be ignored instead. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #12 from jmatejek@novell.com 2006-11-13 08:00 MST ------- httplib searches the environment for .*_proxy and then uses that. Warning is a good idea, it would be simple enough to say "warning, https proxy is not supported" when "https_proxy" is encountered. I'll do that. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 ------- Comment #13 from poeml@novell.com 2006-11-13 08:03 MST ------- Thanks! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 sndirsch@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |eich@novell.com, sndirsch@novell.com ------- Comment #14 from sndirsch@novell.com 2007-05-12 04:28 MST ------- Egbert, JFYI. Since Matthias or me is in Cc of this bugreport or the reported itself, it might be interesting for you as well. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983 jmatejek@novell.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED ------- Comment #15 from jmatejek@novell.com 2007-05-24 11:42 MST ------- i've submitted the warning along with update to py2.5.1 matejcik@titan:/data/yapt/stable-all> python -c "import urllib2; urllib2.urlopen('https://www.sf.net').read()" /usr/lib/python2.5/urllib2.py:662: RuntimeWarning: urllib can't handle https proxies, your https_proxy setting will not work proxies = getproxies() the URL is then retrieved without use of proxy -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
https://bugzilla.novell.com/show_bug.cgi?id=214983#c16 --- Comment #16 from Peter Poeml <poeml@novell.com> 2007-08-08 10:42:11 MST --- This issue showed up in bug 298378 -- which I didn't resolve as a duplicate, but I am mentioning it here for reference. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=214983 User jmatejek@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=214983#c17 --- Comment #17 from Jan Matejek <jmatejek@novell.com> 2008-09-19 12:54:03 MDT --- upstream has manned up and they have a working patch, i'm including it in our python -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com