[Bug 991680] New: zypper requires leading dot for domains in no_proxy setting
http://bugzilla.opensuse.org/show_bug.cgi?id=991680 Bug ID: 991680 Summary: zypper requires leading dot for domains in no_proxy setting Classification: openSUSE Product: openSUSE Distribution Version: Leap 42.1 Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: libzypp Assignee: zypp-maintainers@forge.provo.novell.com Reporter: eblock@nde.ag QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- Description of problem: Using zypper behind a proxy leads to errors for repositories hosted on local servers (e.g. spacewalk) because domains in the no_proxy settings are ignored if they don't have a leading dot. However, curl or wget accept domains without a leading dot. It works for: "no_proxy=localhost, 127.0.0.1, .example.org" It doesn't work for: "no_proxy=localhost, 127.0.0.1, example.org" This issue exists in - openSUSE_Leap_42.1 - SLES12-SP1 - openSUSE_13.1 This is NOT an issue in - SLES11-SP4 We haven't tested this on Leap_42.2. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c1
Michael Andres
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c2
Dominique Leuenberger
All proxy handling is delegated to libproxy and it looks indeed like libproxy actually requires a leading dot to match domains.
I forward this to the libprocxy maintainers as there should be no need to enforce the leading dot.
How do you distinguish the difference between a subdomain and a hostname then? It's not uncommon to start domains with a dot, and hostnames not. The syntax used by Libproxy is, by the way, identical to the one documented by Firefox too: http://www-archive.mozilla.org/quality/networking/docs/aboutno_proxy_for.htm...
From a libproxy PoV => ENOTABUG
-- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c3
Michael Andres
PROXY_ENABLED="yes" HTTP_PROXY="http://proxy.provider.de:3128" NO_PROXY="localhost, hostname.domain.com"
to filter 'host.hostname.domain.com' or 'host-hostname.domain.com'
"www.mozilla.org" Also blocks any hostnames or domains that end in the same string (other-www.mozilla.org)
But e.g. on my SLE12SP1 it does not:
$ proxy http://domain.com http://proxy.provider.de:3128
$ proxy http://hostname.domain.com direct://
$ proxy http://host.hostname.domain.com http://proxy.provider.de:3128
$ proxy http://host-hostname.domain.com http://proxy.provider.de:3128
Also the lower table IMO suggests this:
FQDNs hostname.domain.com domain.com proxy hostname.domain.com hostname.domain.com direct hostname.domain.com host.hostname.domain.com direct
-- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c4
Dominique Leuenberger
Also the lower table IMO suggests this:
FQDNs hostname.domain.com domain.com proxy hostname.domain.com hostname.domain.com direct hostname.domain.com host.hostname.domain.com direct
So the 3rd call does not return as you expect and is documented in the Mozilla table - I don't really see why they make a difference in the number of levels; that's bogus and rong (and we did indeed not copy this behavior). In UK for example, it's not uncommon to use .co.uk instead of .com, so you immediately get one level more and run into weird behaviors. Specifying when you talk about the domain and when about the host is the only sane solution to know what you want to exluce (Think corporate: with domain: example.co.uk having this internal network, but www.example.co.uk is hosted external.. ) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c5
--- Comment #5 from Michael Andres
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c6
--- Comment #6 from Dominique Leuenberger
Curl IMO is a bit more relaxed: Either an exact match, or the previous character is a '.', so a host is within the same domain. A leading '.' in the pattern is ignored.
Anyway - I don't mind so much _which_ semantic is applied, but consistency is IMO desirable.
The problem I see, is that /etc/sysconfig/proxy:NO_PROXY is exported into the environment, where curl/wget/etc. evaluate it.
Zypp - if /etc/sysconfig/proxy:PROXY_ENABLED="yes" - will have NO_PROXY evaluated by libproxy. OTOH, if /etc/sysconfig/proxy is off, but the same NO_PROXY value is manually exported into the environment, the underlying libcurl is allowed evaluate it.
That's actually only true for ROOT - as there the sysconfig module is loaded, which was intentionally written for libzypp back in the days. Other users won't load the sysconfig plugin to start with and will only evaluate the session config (that is GNOME settings in GNOME, KDE Settings in KDE and ENVVAR in other environments)
So zypp will behave differently depending where the NO_PROXY value is defined. That's bad.
the sysconfig module always wins in libproxy as root: envvar is a compat backdrop only and is used if 'no better matching config source can be found' sysconfig being 'the better config source' states PROXY_ENABLED=no, so libproxy takes the best match and returns direct://
...if there's no way to tweak libpoxy, maybe we should drop and do it the-curl-way.
https://github.com/libproxy/libproxy is opensource like so many other libs.. .there is always a way to improve things. Just because curl went the lazy way of weird matching does not make it right though (btw: wget uses libproxy - so it behaves the same... curl does not... imho, curl should be fixed) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c7
Jens-U. Mozdzen
(In reply to Michael Andres from comment #3)
Also the lower table IMO suggests this:
FQDNs hostname.domain.com domain.com proxy hostname.domain.com hostname.domain.com direct hostname.domain.com host.hostname.domain.com direct
So the 3rd call does not return as you expect and is documented in the Mozilla table - I don't really see why they make a difference in the number of levels; that's bogus and rong (and we did indeed not copy this behavior).
who's counting levels? All that is done is matching the tailing characters of the FQDN.
Specifying when you talk about the domain and when about the host is the only sane solution to know what you want to exluce
(Think corporate: with domain: example.co.uk having this internal network, but www.example.co.uk is hosted external.. )
I don't get your point/example? All internal hosts are to be skipped for proxying, so you'd put ".example.co.uk" into NO_PROXY - but that would include www.example.co.uk as well. No matter of specifying the leading sub-domain separator (".") or not. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c8
--- Comment #8 from Dominique Leuenberger
I don't get your point/example? All internal hosts are to be skipped for proxying, so you'd put ".example.co.uk" into NO_PROXY - but that would include www.example.co.uk as well. No matter of specifying the leading sub-domain separator (".") or not.
Oh right.. I consider that even worse... So company ABS (let's say the use 'abs.com' in their NO_PROXY) matching on tailing chars would mean this company can't reach 'fabs.com'; this sounds like way too many things that can break even worse - 'ABS" would just configure .abs.com and be done with it. IMHO, this is rather something YaST should guide the user(s) to configure correctly and a statement for documentation as for yast: it uses this in the help section:
No Proxy Domains is a comma-separated list of domains for which the requests should be made directly without caching, for example, localhost, .intranet.example.com, www.example.com
=> it does give the indication of the leading DOT in the help system. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c9
--- Comment #9 from Dominique Leuenberger
Specifying when you talk about the domain and when about the host is the only sane solution to know what you want to exluce
(Think corporate: with domain: example.co.uk having this internal network, but www.example.co.uk is hosted external.. )
I don't get your point/example? All internal hosts are to be skipped for proxying, so you'd put ".example.co.uk" into NO_PROXY - but that would include www.example.co.uk as well. No matter of specifying the leading sub-domain separator (".") or not.
Right - thinking error at my end on this one (I used to workaround those situations in companies using .pac files, which offers the flexibility needed for this case) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c10
--- Comment #10 from Michael Andres
So company ABS (let's say the use 'abs.com' in their NO_PROXY) matching on tailing chars would mean this company can't reach 'fabs.com'; this sounds like way too many things that can break even worse - 'ABS" would just configure .abs.com and be done with it.
The user experience however seems to be different. You configure 'suse.de' and it works for wget and curl, but not for zypper:
http_proxy=http://proxy.suse.de:3128/ no_proxy='localhost, 127.0.0.1, suse.de'
$ proxy http://www.suse.de # what zypper gets from libsolv http://proxy.suse.de:3128/ http://proxy.suse.de
$ wget http://www.suse.de Resolving www.suse.de (www.suse.de)... i.e. direct://
$ curl -v http://www.suse.de * Connected to www.suse.de... i.e. direct://
Wget probably uses the libproxy ENVVAR module which matches '*suse.de' (far more than you actually wanted, but who actually recognizes this...). Curl matches 'suse.de|*.suse.de', which might be 'lazy and weird' ;), but is probably close to what you actually wanted. Only poor zypper uses the libproxy SYSCONFIG module, which matches just 'suse.de' if you missed the leading dot. AFAIR the idea of the SYSCONFIG module was to let changes to sysconfig/proxy take effect immediately. Otherwise you had to logout and login again or to re-evaluate /etc/profile in order to get the right environment variables. The question is whether the SYSCONFIG module could/should be more relaxed (like curl, which is what zypp uses)? Otherwise the bug would be INVALID as the dot is needed (unless you use curl directly). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c11
--- Comment #11 from Dominique Leuenberger
(In reply to Dominique Leuenberger from comment #8)
So company ABS (let's say the use 'abs.com' in their NO_PROXY) matching on tailing chars would mean this company can't reach 'fabs.com'; this sounds like way too many things that can break even worse - 'ABS" would just configure .abs.com and be done with it.
The user experience however seems to be different. You configure 'suse.de' and it works for wget and curl, but not for zypper:
http_proxy=http://proxy.suse.de:3128/ no_proxy='localhost, 127.0.0.1, suse.de'
$ proxy http://www.suse.de # what zypper gets from libsolv http://proxy.suse.de:3128/ http://proxy.suse.de
$ wget http://www.suse.de Resolving www.suse.de (www.suse.de)... i.e. direct://
Did you run this as root or user? Root uses config_sysconfig, user uses config_{gnome3,kde,envvar}, depending on your current session (test with _PX_DEBUG=1 wget http://www.suse.com => it will tell you what config it found)
$ curl -v http://www.suse.de * Connected to www.suse.de... i.e. direct://
Wget probably uses the libproxy ENVVAR module which matches '*suse.de' (far more than you actually wanted, but who actually recognizes this...).
Curl matches 'suse.de|*.suse.de', which might be 'lazy and weird' ;), but is probably close to what you actually wanted.
Only poor zypper uses the libproxy SYSCONFIG module, which matches just 'suse.de' if you missed the leading dot.
the behavior, once the config is read, should not be different between the config modules, as the actual processing of the config is done in the ignore_* modules (ignore_ip, ignore_hostname, ignore_domain); ALL tools (incl. proxy test tool) use sysconfig when running as root.
AFAIR the idea of the SYSCONFIG module was to let changes to sysconfig/proxy take effect immediately. Otherwise you had to logout and login again or to re-evaluate /etc/profile in order to get the right environment variables.
That's correct; another idea was to be able to use config_sysconfig as 'chained' config reader, meaning one could set e.g. GNOME to read 'SYSTEM SETTINGS' (something that does not exist at this moment) and thus fall back to sysconfig (which would otherwise be ignored in GNOME). This part is not yet implemented in libproxy yet though
The question is whether the SYSCONFIG module could/should be more relaxed (like curl, which is what zypp uses)? Otherwise the bug would be INVALID as the dot is needed (unless you use curl directly).
as said, generally the 'config source' should not impact how things are interpreted by libproxy (but of course the 'config parser can add interpretation to obfuscate things) Generally speaking, libproxy would suggest to connect like this (valid for all config modules) URL IGNOREFILT CONNECTION http://www.suse.com suse.com PROXY http://suse.com suse.com DIRECT http://www.suse.com .suse.com DIRECT http://suse.com .suse.com PROXY (this one is actually arguable) http://www.suse.com *suse.com DIRECT http://suse.com *suse.com DIRECT -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c12
--- Comment #12 from Michael Andres
Did you run this as root or user? Root uses config_sysconfig, user uses
As root, as install/update/migration are the libzypp scenarios I'm actually interested in.
(test with _PX_DEBUG=1 wget
Results vary. Looks like my wget-1.16 uses some strange kind of logic. It appears as if libproxy is used only if no proxy is defined in the environment:
# login with sysconfig/ptroxy OFF
$ set | grep -i proxy
$ _PX_DEBUG=1 wget http://suse.de Configuration extension is: 26sysconfig_config_extension Ingored list is: localhost, 127.0.0.1, .suse.de Config is: direct:// --2016-08-04 13:35:37-- http://suse.de/
If proxy varialbles are defined, I get no _PX_DEBUG output.
# login with sysconfig/proxy ON
$ set | grep -i proxy NO_PROXY='localhost, 127.0.0.1, .suse.de' ftp_proxy= gopher_proxy= http_proxy=http://proxy.suse.de:3128/ https_proxy=http://proxy.suse.de:3128/ no_proxy='localhost, 127.0.0.1, .suse.de'
$ _PX_DEBUG=1 wget http://suse.de --2016-08-04 13:35:41-- http://suse.de/
But that's a different topic.... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c13
--- Comment #13 from Dominique Leuenberger
(In reply to Dominique Leuenberger from comment #11)
Did you run this as root or user? Root uses config_sysconfig, user uses
As root, as install/update/migration are the libzypp scenarios I'm actually interested in.
(test with _PX_DEBUG=1 wget
Results vary. Looks like my wget-1.16 uses some strange kind of logic. It appears as if libproxy is used only if no proxy is defined in the environment:
Ah right - the wget folks wanted it 'least surprising for users' - I did not agree but that was the closest they wanted to accept (not that they actually ever DID accept the patch) - and a user that could no longer specify HTTP_PROXY= would be a surprise to them. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=991680
http://bugzilla.opensuse.org/show_bug.cgi?id=991680#c14
Michael Andres
participants (1)
-
bugzilla_noreply＠novell.com