On Tue, Feb 24, 2009 at 12:33:01PM +0300, z wrote:

2009/2/23 Duncan Mac-Vicar P. <dmacvicar@suse.de>:

...

I have submited libzypp 6.1.0 to Factory. Which makes aria2c the default transfer backend for http/ftp.

Read about what this means and some new features here:

http://duncan.mac-vicar.com/blog/archives/507

I wanted to test a little more, but Factory still has 5.11.0 which is even older than 11.1 updates I think. So be sure to read the post and learn about ZYPP_ARIA2C=0 in case something breaks or you are hit by any regression (may be open issues with proxy, ipv6) ;-)

Have fun!

Btw, this new stuff just arrived in Factory an hour ago.

...

Duncan Mac-Vicar P. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

hi. Am i right, that this will affect proxy management for yast\zypper? There was issue https://bugzilla.novell.com/show_bug.cgi?id=458708 due to curl environment vars for proxy can it be handeled gracefully since now? Thanks in advance

It will affect proxy handling in so far that the new download backend has slightly different characteristics of how it opens connections, does its HTTP requests -- changes on that level. What's unchanged is that any system-wide proxy setting is propagated to the download backend as before, with the little difference that .curlrc isn't relevant anymore. (Unless the old download backend is used, that is; but it should not be used anymore by default now.) However, it is an interesting question how to actually deal best with proxies. Since the new download backend uses more than one connection and downloads files in chunks, this does not play well with caching, and I am considering a procedere where we actually use the old backend when a proxy is configured. This or a configuration of the new backend that has the same effect. And since HTTP redirects do also not play well with caching (regardless of download backend), I am further considering the introduction of server-side state tracking. Anyway, once we have the newk powerful client, which deals with all the error scenarios that plagued users the past, optimizing for intermediary caches is one of the next things to have a closer look on. And I expect that it'll take a while until the new technical possibilities become reflected in YaST. So I'm talking rather low-level here, which may not completely answer your question to your satisfaction. Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Thu, Feb 26, 2009 at 04:08:04PM +0300, z wrote:

...

...

hi. Am i right, that this will affect proxy management for yast\zypper? There was issue https://bugzilla.novell.com/show_bug.cgi?id=458708 due to curl environment vars for proxy can it be handeled gracefully since now? Thanks in advance

It will affect proxy handling in so far that the new download backend has slightly different characteristics of how it opens connections, does its HTTP requests -- changes on that level.

What's unchanged is that any system-wide proxy setting is propagated to the download backend as before, with the little difference that .curlrc isn't relevant anymore. (Unless the old download backend is used, that is; but it should not be used anymore by default now.)

Ok. What was bad in curl is requiring relogin to switch off proxy: "Switching proxy from On to Off: YaST module clears /root/.curlrc file, but until relogin, $http_proxy variable remains set and libcurl (thus, libzypp too) still uses that URL even though you don't want to use proxy anymore."

Maybe someone can tell, if there is any method to turn proxy off for aria2c, while environment still contains old proxy settings? Something like a settings file, which is preferred over the environment, and can force aria2 to not to use proxy?

YaST could simply read the current setting from /etc/sysconfig/proxy when it starts, and put them into the environment, so they would be used by both libcurl and aria2c. This would break with the principle of least surprise for some users, who expect the current *environment* to be decisive; things to do in a shell like # http_proxy= yast2 or # http_proxy=http://different-proxy.example.com:3128/ yast2 to temporarily override a setting would no longer be working. I tend to agree, however, that most "normal" users would benefit from yast2 reading the settings from /etc/sysconfig/proxy afresh, thus ignoring the enviromnent. It would give *those* users what they expect, I guess. Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Duncan Mac-Vicar P. wrote:

Right now, the MediaAria backend is just an extended MediaCurl. So when attaching a http media, it reads Sysconfig, and sets the proxy from there, unless the user override it in the url.

However, I am not sure how libcurl or aria handle the environment variable or curlrc.

Duncan

We checked and we handle everything right, except for the env variable which seems to be used by curl internally and ZYpp has no control over it. This is the only problem i see, if curl reads the env variable, and aria does not, then mediacurl will behave differently when downloading (aria2) than when checking if a file exist (curl) Duncan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Peter Poeml wrote:

Really? That means that /etc/sysconfig/proxy is always authoritative? Then it shouldn't (have) happen(ed) that any user needs to relogin after the setting is changed...

No, libzypp reads sysconfig when the media object is created. Any info there is overridden by url params if they exist. Then other settings (proxy auth) are read from curlrc when the media is attached. Then when a file is transfered, libcurl may be reads the env variable on _its own_. Duncan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Peter Poeml wrote:

aria2 does it the same way as libcurl and various other programs, so no difference here; except for the fact that yast (?) creates a /root/.curlrc file...

Yes, but curlrc is read by ZYpp, not by curl, and then the options are passed to libcurl or to aria2 (at least proxy options) Duncan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Dave Plater wrote:

Isn't passing the option the equivalent to a command line option? This would override the environment variable and enable the turning on and off of proxy use without a logout / login. Regards Dave P For aria, yes. But the ZYpp aria backend is an extension of the curl backend, and fallbacks to curl for some operations that aria can't do, like 1 byte range transfer as a hack to check for file existence on broken ftp servers.

My fear is that for those operations, libcurl could override what libzypp set using curl_easy_set_opt (command line parameters equivalent) with the environment variables, thus overriding arias behavior (actually inverting it) and making the env variable authoritative. Does anyone know if curl/libcurl override set_opt with env variables? I can test, but not now ;-) Duncan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Dave Plater wrote:

This doesn't answer your question but aria2c, from what I have observed, does a very good job of checking file existence on mirrors by itself. That is as long a http://download.opensuse.org is used and maybe the easier approach is to rely more on aria2c's reporting. Don't quite know how the refresh process works though but this is so as far as package download is concerned.

No, the usecase is different. Sometimes libzypp wants to know if the file is there, but if it is, not to download it. curl can do a head request only. This is use for repository probing for example.

To shed a bit of light according to https://bugzilla.novell.com/show_bug.cgi?id=458708#c7 /root/.curlrc overrides the environment variable and various program options I have seen override option files and environment variables. i.e. if libzypp passes an option from its config file it will override all others. Sorry I can't think of a zypp.conf option to actually test this with. Regards Dave P

I found my answer: libcurl respects the environment variables http_proxy, ftp_proxy, all_proxy etc, if any of those are set. The CURLOPT_PROXY <http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTPROXY> option does however override any possibly set environment variables. Setting the proxy string to "" (an empty string) will explicitly disable the use of a proxy, even if there is an environment variable set for it. I need to make sure we set the option to "" if the proxy was disabled on sysconfig, I guess. Duncan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Dave Plater wrote:

Does libzypp bypass aria2c and use curl under certain circumstances when refreshing repos? With the new libzypp when my packman repository is refreshed (the first one to be refreshed) libzypp doesn't use aria2c whilst "Retrieving repository 'Packman' metadata" which is quite a large transfer but appears to use aria2c for the other repositories. Regards Dave P

yes, MediaAria2c is actually an extended MediaCurl. For some operations like HEAD requests to check file existence, we use curl. Can you open a bug report, assign to me, and attach a log? -- Duncan Mac-Vicar P. - Engineering Manager, YaST SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Stephan Kulow wrote:

Does it make sense to use aria2c for repositories without metalinks?

I think yes, because the multiple connections, speed controls, etc. In the future we can add more grained failover, for example, allowing the user to manually say "this host is a mirror of..." and then aria makes easy to use this information. And, as we don't know which hosts have metalinks or not in advance, ZYpp would have some trouble knowing which backend to use. However, this could be done per repo (manually). Do you have anything against it? -- Duncan Mac-Vicar P. - Engineering Manager, YaST SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org