[zypp-devel] Way how separate download and install package
During testing I find way, how separate download and install of packages. It is quite easy with zypper. 1) enable keeping package for repository (zypper mr -k repo) 2) run zypper in -D (dry run, only test install) 3) disconnect from internet (rcnetwork stop) 4) rerun zypper in without -D and it works Possibly problems 1) metadata must be valid and autorefresh disabled 2) packages rpm is kept after install (delete after successful install can be useful) Pepa -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Josef Reidinger wrote:
During testing I find way, how separate download and install of packages. It is quite easy with zypper.
1) enable keeping package for repository (zypper mr -k repo) 2) run zypper in -D (dry run, only test install) 3) disconnect from internet (rcnetwork stop) 4) rerun zypper in without -D and it works
Possibly problems 1) metadata must be valid and autorefresh disabled 2) packages rpm is kept after install (delete after successful install can be useful)
Nice idea, but it doesn't fulfill the use-case: Users want libzypp to install packages while another ones are being downloaded. This can be solved by two threads - one for downloading packages and another one for installing them. And it must be solved in libzypp, not only in zypper, yast2-qt-pkg, yast2-ncurses-pkg ... Using zypper is a nice cheating you might want to post on openSUSE-FACTORY or blog about it :) ;) We actually want this feature (and not via 'keep-packages' because of limited resources during installation) for all user front-ends. Bye Lukas
Am Freitag 06 Juni 2008 schrieb Lukas Ocilka:
Josef Reidinger wrote:
During testing I find way, how separate download and install of packages. It is quite easy with zypper.
1) enable keeping package for repository (zypper mr -k repo) 2) run zypper in -D (dry run, only test install) 3) disconnect from internet (rcnetwork stop) 4) rerun zypper in without -D and it works
Possibly problems 1) metadata must be valid and autorefresh disabled 2) packages rpm is kept after install (delete after successful install can be useful)
Nice idea, but it doesn't fulfill the use-case:
You confuse use-cases. Other people want zypper to behave like smart: download all packages and then install in one transaction. Josef's hack only does the first part :) Greetings, Stephan -- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Stephan Kulow wrote:
Am Freitag 06 Juni 2008 schrieb Lukas Ocilka:
Josef Reidinger wrote:
During testing I find way, how separate download and install of packages. It is quite easy with zypper.
1) enable keeping package for repository (zypper mr -k repo) 2) run zypper in -D (dry run, only test install) 3) disconnect from internet (rcnetwork stop) 4) rerun zypper in without -D and it works
Possibly problems 1) metadata must be valid and autorefresh disabled 2) packages rpm is kept after install (delete after successful install can be useful) Nice idea, but it doesn't fulfill the use-case:
You confuse use-cases. Other people want zypper to behave like smart: download all packages and then install in one transaction. Josef's hack only does the first part :)
Greetings, Stephan
Does rpm allow something like emerge -> install to some fake root like /var/tmp/fakeroot/ and after successful install to this root install whole this fake root to real root??? Pepa -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Stephan Kulow wrote:
Am Freitag 06 Juni 2008 schrieb Lukas Ocilka:
Josef Reidinger wrote:
During testing I find way, how separate download and install of packages. It is quite easy with zypper.
1) enable keeping package for repository (zypper mr -k repo) 2) run zypper in -D (dry run, only test install) 3) disconnect from internet (rcnetwork stop) 4) rerun zypper in without -D and it works
Possibly problems 1) metadata must be valid and autorefresh disabled 2) packages rpm is kept after install (delete after successful install can be useful) Nice idea, but it doesn't fulfill the use-case:
You confuse use-cases. Other people want zypper to behave like smart: download all packages and then install in one transaction. Josef's hack only does the first part :)
OK, I was actually writing about the use-case from FATE and bugzilla and mailing-lists where users want exactly this ;) Yes, for zypper, this might work ... but does it make sense to implement code that couldn't be reused? In the times of libzypp :)? L.
Lukas Ocilka wrote:
Stephan Kulow wrote:
Am Freitag 06 Juni 2008 schrieb Lukas Ocilka:
Josef Reidinger wrote:
During testing I find way, how separate download and install of packages. It is quite easy with zypper.
1) enable keeping package for repository (zypper mr -k repo) 2) run zypper in -D (dry run, only test install) 3) disconnect from internet (rcnetwork stop) 4) rerun zypper in without -D and it works
Possibly problems 1) metadata must be valid and autorefresh disabled 2) packages rpm is kept after install (delete after successful install can be useful) Nice idea, but it doesn't fulfill the use-case:
You confuse use-cases. Other people want zypper to behave like smart: download all packages and then install in one transaction. Josef's hack only does the first part :)
OK, I was actually writing about the use-case from FATE and bugzilla and mailing-lists where users want exactly this ;)
Yes, for zypper, this might work ... but does it make sense to implement code that couldn't be reused? In the times of libzypp :)?
L.
It can be reused, it is dry run policy in commit in libzypp. Only keepPackages must be allowed to don't delete downloaded files after fake install. Pepa -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Josef Reidinger wrote:
Lukas Ocilka wrote:
Stephan Kulow wrote:
You confuse use-cases. Other people want zypper to behave like smart: download all packages and then install in one transaction. Josef's hack only does the first part :)
OK, I was actually writing about the use-case from FATE and bugzilla and mailing-lists where users want exactly this ;)
Yes, for zypper, this might work ... but does it make sense to implement code that couldn't be reused? In the times of libzypp :)?
L.
It can be reused, it is dry run policy in commit in libzypp. Only keepPackages must be allowed to don't delete downloaded files after fake install.
I'm adraid this doesn't work for installation with limited resources (memory, disk space), by default all sources during installation se keep_packages=false (please, remember the bug we had recently in installation). And moreover downloading and installing in threads is faster than first downloading and then installing. And this is what our users want too. L.
Lukas Ocilka wrote:
Josef Reidinger wrote:
Lukas Ocilka wrote:
Stephan Kulow wrote:
You confuse use-cases. Other people want zypper to behave like smart: download all packages and then install in one transaction. Josef's hack only does the first part :)
OK, I was actually writing about the use-case from FATE and bugzilla and mailing-lists where users want exactly this ;)
Yes, for zypper, this might work ... but does it make sense to implement code that couldn't be reused? In the times of libzypp :)?
L.
It can be reused, it is dry run policy in commit in libzypp. Only keepPackages must be allowed to don't delete downloaded files after fake install.
I'm adraid this doesn't work for installation with limited resources (memory, disk space), by default all sources during installation se keep_packages=false (please, remember the bug we had recently in installation).
And moreover downloading and installing in threads is faster than first downloading and then installing. And this is what our users want too.
L.
If everything work good this is not hard. In commit code it is separated, only we need instead providing file when needed, provide it to queue (producer and consumer) and install when queue is not empty. But if something happen what to do? Do we expect more then one failure (because download and install can both cause fail), also I am now sure if libzypp is prepared for multithread aplication. Also we must provide two progress, one for download and one for installation. Does YaST expect multithread libzypp? E.g. one callback is called and what happen if another is called and YaST still does work on first? Pepa Pepa -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Josef Reidinger wrote:
If everything work good this is not hard. In commit code it is separated, only we need instead providing file when needed, provide it to queue (producer and consumer) and install when queue is not empty. But if something happen what to do? Do we expect more then one failure (because download and install can both cause fail), also I am now sure if libzypp is prepared for multithread aplication. Also we must provide two progress, one for download and one for installation. Does YaST expect multithread libzypp? E.g. one callback is called and what happen if another is called and YaST still does work on first?
That sound like a very good deliberation about pros and cons of libzypp and YaST capabilities for running in multiple threads. About answers to your questions: Franky, I don't know, but the important part was to start a discussion :) I hope that both YaST and libzypp gurus will join it. PS: If we use separate threads for downloading and installation, there should be a limit on maximum of files / data size downloaded into queue (cache size). Thanks Lukas
On Fri, Jun 06, Lukas Ocilka wrote:
Josef Reidinger wrote:
If everything work good this is not hard. In commit code it is separated, only we need instead providing file when needed, provide it to queue (producer and consumer) and install when queue is not empty. But if something happen what to do? Do we expect more then one failure (because download and install can both cause fail), also I am now sure if libzypp is prepared for multithread aplication. Also we must provide two progress, one for download and one for installation. Does YaST expect multithread libzypp? E.g. one callback is called and what happen if another is called and YaST still does work on first?
The callbacks will be serialized, so yast/zypper can process them one by one. But each callback may be related to a different task, as there are multiple tasks running at the same time.
That sound like a very good deliberation about pros and cons of libzypp and YaST capabilities for running in multiple threads.
About answers to your questions: Franky, I don't know, but the important part was to start a discussion :) I hope that both YaST and libzypp gurus will join it.
PS: If we use separate threads for downloading and installation, there should be a limit on maximum of files / data size downloaded into queue (cache size).
Yes. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Michael Andres wrote:
On Fri, Jun 06, Lukas Ocilka wrote:
Josef Reidinger wrote: ...
OK, we had a discussion here and it seems all these issues, pros and cons have been already discussed. Anyway, I haven't seen (or I don't remember seeing) any document, wiki or plan where this would be summarized :) ;) Where can I find it? Thanks in advance Lukas
On Fri, Jun 06, 2008 at 04:06:24PM +0200, Lukas Ocilka wrote:
Michael Andres wrote:
On Fri, Jun 06, Lukas Ocilka wrote:
Josef Reidinger wrote: ...
OK, we had a discussion here and it seems all these issues, pros and cons have been already discussed. Anyway, I haven't seen (or I don't remember seeing) any document, wiki or plan where this would be summarized :) ;)
Where can I find it?
I don't know in the wiki, but in the Novell internal feature tracking tool it is tracked as issue #120340.
Thanks in advance Lukas
Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
Peter Poeml wrote:
On Fri, Jun 06, 2008 at 04:06:24PM +0200, Lukas Ocilka wrote:
Michael Andres wrote:
On Fri, Jun 06, Lukas Ocilka wrote:
Josef Reidinger wrote: ... OK, we had a discussion here and it seems all these issues, pros and cons have been already discussed. Anyway, I haven't seen (or I don't remember seeing) any document, wiki or plan where this would be summarized :) ;)
Where can I find it?
I don't know in the wiki, but in the Novell internal feature tracking tool it is tracked as issue #120340.
The mentioned FATE #120340 contains this link: http://en.opensuse.org/Libzypp/Failover Bye L.
On Mon, Jun 09, 2008 at 02:56:58PM +0200, Lukas Ocilka wrote:
Peter Poeml wrote:
On Fri, Jun 06, 2008 at 04:06:24PM +0200, Lukas Ocilka wrote:
Michael Andres wrote:
On Fri, Jun 06, Lukas Ocilka wrote:
Josef Reidinger wrote: ... OK, we had a discussion here and it seems all these issues, pros and cons have been already discussed. Anyway, I haven't seen (or I don't remember seeing) any document, wiki or plan where this would be summarized :) ;)
Where can I find it?
I don't know in the wiki, but in the Novell internal feature tracking tool it is tracked as issue #120340.
The mentioned FATE #120340 contains this link: http://en.opensuse.org/Libzypp/Failover
Indeed, and while that doesn't intend to cover parallel downloads, it is definitely related, and parallel downloads should be considered in that context. I would be very glad if (as Coolo suggested) downloading would be split out to a separate process, because I think that it would make implementing the failover proposal (and other things) much easier. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
Josef Reidinger wrote:
If everything work good this is not hard. In commit code it is separated, only we need instead providing file when needed, provide it to queue (producer and consumer) and install when queue is not empty. But if something happen what to do? Do we expect more then one failure (because download and install can both cause fail), also I am now sure if libzypp is prepared for multithread aplication. Also we must provide two progress, one for download and one for installation. Does YaST expect multithread libzypp? E.g. one callback is called and what happen if another is called and YaST still does work on first?
It depends on the callback type. I implemented some callbacks in a reentrant way because some callbacks are nested even in the current libzypp. Each callback has a unique ID (numericId()) so it is possible to distinguish them. But it's currently not used in yast (except for debug logging). -- Best Regards Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Dňa Friday 06 June 2008 13:49:33 Josef Reidinger ste napísal:
Lukas Ocilka wrote:
Josef Reidinger wrote:
Lukas Ocilka wrote:
Stephan Kulow wrote:
You confuse use-cases. Other people want zypper to behave like smart: download all packages and then install in one transaction. Josef's hack only does the first part :)
OK, I was actually writing about the use-case from FATE and bugzilla and mailing-lists where users want exactly this ;)
Yes, for zypper, this might work ... but does it make sense to implement code that couldn't be reused? In the times of libzypp :)?
L.
It can be reused, it is dry run policy in commit in libzypp. Only keepPackages must be allowed to don't delete downloaded files after fake install.
I'm adraid this doesn't work for installation with limited resources (memory, disk space), by default all sources during installation se keep_packages=false (please, remember the bug we had recently in installation).
And moreover downloading and installing in threads is faster than first downloading and then installing. And this is what our users want too.
L.
If everything work good this is not hard. In commit code it is separated, only we need instead providing file when needed, provide it to queue (producer and consumer) and install when queue is not empty. But if something happen what to do? Do we expect more then one failure (because download and install can both cause fail), also I am now sure if libzypp is prepared for multithread aplication. Also we must provide two progress, one for download and one for installation. Does YaST expect multithread libzypp?
IIRC, Duncan has a half-finished patch for download in threaded way. Stano -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Stanislav Visnovsky wrote:
IIRC, Duncan has a half-finished patch for download in threaded way.
Stano
(only internal) http://w3.suse.de/~dmacvicar/libzypp/0001-first-attempt-to-do-paralell-downl... It does a nice crash :-) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Duncan Mac-Vicar Prett wrote:
Stanislav Visnovsky wrote:
IIRC, Duncan has a half-finished patch for download in threaded way.
Stano
(only internal) http://w3.suse.de/~dmacvicar/libzypp/0001-first-attempt-to-do-paralell-downl...
It does a nice crash :-)
I see that duncan use boost::thread and because I never use it I write little performance test for boost::thread if pthreads isn't better. And result is that it is almost same (different is statistical error). boost threads: user time: 4032 system time: 24 pthreads: user time: 4076 system time: 14 source code attached. Pepa
Hi all, Lukas Ocilka wrote:
Josef Reidinger wrote:
Lukas Ocilka wrote: [....] It can be reused, it is dry run policy in commit in libzypp. Only keepPackages must be allowed to don't delete downloaded files after fake install.
I'm adraid this doesn't work for installation with limited resources (memory, disk space), by default all sources during installation se keep_packages=false (please, remember the bug we had recently in installation).
I think it even doesn't make sense in installation - the feature is useful in an installed system which you need to keep in a consistent system. If installation fails the system will be broken anyway.
And moreover downloading and installing in threads is faster than first downloading and then installing. And this is what our users want too.
That's something different. In the first case (download first then install) package update is safer, if there is a problem with downloading you don't break the system with partially updated packages. Parallel download and installation "just" speeds up the installation, but update is not as safe as in the first scenario. IMO both scenarios are valid and should be possible in the future. -- Best Regards Ladislav Slezák Yast Developer ------------------------------------------------------------------------ SUSE LINUX, s.r.o. e-mail: lslezak@suse.cz Lihovarská 1060/12 tel: +420 284 028 960 190 00 Prague 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Ladislav Slezak wrote:
That's something different. In the first case (download first then install) package update is safer, if there is a problem with downloading you don't break the system with partially updated packages.
Parallel download and installation "just" speeds up the installation, but update is not as safe as in the first scenario.
IMO both scenarios are valid and should be possible in the future.
Yes. Both features sound nice. Last night my VPN stopped half way through a large zypper transaction... no fun. -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Bart Whiteley wrote:
Ladislav Slezak wrote:
That's something different. In the first case (download first then install) package update is safer, if there is a problem with downloading you don't break the system with partially updated packages.
Parallel download and installation "just" speeds up the installation, but update is not as safe as in the first scenario.
IMO both scenarios are valid and should be possible in the future.
Yes. Both features sound nice. Last night my VPN stopped half way through a large zypper transaction... no fun.
For first case maybe this article[0] also solv if customer need safety upgrade and error happen during installation (after all downloaded). But this must be only optional, because it is slower and also consume more disk space. Pepa [0]http://www.linuxjournal.com/article/7034 -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Lukas Ocilka escribió:
And moreover downloading and installing in threads is faster than first downloading and then installing.
I guess you might want to look at libcurl "multi" implementation {1} http://curl.haxx.se/libcurl/c/libcurl-multi.html to make the downloading part parallel and reliable. -- "Progress is possible only if we train ourselves to think about programs without thinking of them as pieces of executable code.” - Edsger W. Dijkstra Cristian Rodríguez R. Platform/OpenSUSE - Core Services SUSE LINUX Products GmbH Research & Development http://www.opensuse.org/
Cristian Rodríguez napsal(a):
Lukas Ocilka escribió:
And moreover downloading and installing in threads is faster than first downloading and then installing.
I guess you might want to look at libcurl "multi" implementation
{1} http://curl.haxx.se/libcurl/c/libcurl-multi.html
to make the downloading part parallel and reliable.
If I understand it, this is something different. It is multiple download (good if you are faster then servers and also for small files on site with bad latency)..but we need download and install (which is faster if you doesn't have site at same speed as you hard disk :). Pepa -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Am Samstag, 7. Juni 2008 schrieb josef reidiner:
Cristian Rodríguez napsal(a):
Lukas Ocilka escribió:
And moreover downloading and installing in threads is faster than first downloading and then installing.
I guess you might want to look at libcurl "multi" implementation
{1} http://curl.haxx.se/libcurl/c/libcurl-multi.html
to make the downloading part parallel and reliable.
If I understand it, this is something different. It is multiple download (good if you are faster then servers and also for small files on site with bad latency)..but we need download and install (which is faster if you doesn't have site at same speed as you hard disk :).
I would move all downloading into a download manager process that can be queried over IPC once the main process is wanting to install the next package. Greetings, Stephan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (11)
-
Bart Whiteley -
Cristian Rodríguez -
Duncan Mac-Vicar Prett -
josef reidiner -
Josef Reidinger
-
Ladislav Slezak -
Lukas Ocilka -
Michael Andres -
Peter Poeml -
Stanislav Visnovsky -
Stephan Kulow