[zypp-devel] [patch] Keep downloaded packages
Hi! Could you, please, review the attached patch for libzypp? It implements the feature to keep the downloaded packages on disk cache and it's able to reuse existing cached files. The patch applies cleanly against trunk, almost applies against jump_sat ;-) The downloaded packages are stored (by default at) /var/cache/zypp/packages/<alias>/<arch>/<package>.rpm Keep packages behavior is enabled/disabled by repository. Add 'keeppackages=1' to .repo file. The configuration: zypp.conf ========= Added 'packagesdir' option to zypp.conf to re-define a prefix to keep the downloaded packages. Default is '/var/cache/zypp/packages' a.repo ===== Added 'keeppackages' option to .repo parser. Set to 1 to keep the packages. Stano
On Thu, Feb 14, 2008 at 01:04:40PM +0100, Stanislav Višňovský wrote:
Index: zypp/repo/RepoProvideFile.cc =================================================================== --- zypp/repo/RepoProvideFile.cc (revision 8642) +++ zypp/repo/RepoProvideFile.cc (working copy) @@ -28,6 +28,7 @@ #include "zypp/repo/SUSEMediaVerifier.h" #include "zypp/repo/RepoException.h" #include "zypp/FileChecker.h" +#include "zypp/Fetcher.h"
using std::endl; using std::set; @@ -214,7 +215,12 @@ repo_excpt.remember(RepoException(_("No url in repository."))); ZYPP_THROW(repo_excpt); } + + Fetcher fetcher; + fetcher.addCachePath( info.packagesPath() );
+ MIL << "Added cache path " << info.packagesPath() << endl; + for ( RepoInfo::urls_const_iterator it = info.baseUrlsBegin(); it != info.baseUrlsEnd(); /* incremented in the loop */ ) @@ -227,11 +233,17 @@ << "' from " << url << endl; shared_ptr<MediaSetAccess> access = _impl->mediaAccessForUrl(url); _impl->setVerifierForRepo(repo_r, access); + + fetcher.enqueue( loc_r ); + + // FIXME: works for packages only
you mean not for patches and patterns? so what?
+ fetcher.start( info.packagesPath(), *access );
- ManagedFile ret( access->provideFile(loc_r) ); + // reached if no exception has been thrown, so this is the correct file + ManagedFile ret( info.packagesPath() + loc_r.filename() );
std::string scheme( url.getScheme() ); - if ( scheme == "http" || scheme == "https" || scheme == "ftp" ) + if ( !info.keepPackages() )
so will this delete files from my DVD?
{ ret.setDispose( filesystem::unlink ); }
-- Martin Vidner, YaST developer http://en.opensuse.org/User:Mvidner Kuracke oddeleni v restauraci je jako fekalni oddeleni v bazenu -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Dňa Thursday 14 February 2008 13:21:06 Martin Vidner ste napísal:
On Thu, Feb 14, 2008 at 01:04:40PM +0100, Stanislav Višňovský wrote:
Index: zypp/repo/RepoProvideFile.cc =================================================================== --- zypp/repo/RepoProvideFile.cc (revision 8642) +++ zypp/repo/RepoProvideFile.cc (working copy) @@ -28,6 +28,7 @@ #include "zypp/repo/SUSEMediaVerifier.h" #include "zypp/repo/RepoException.h" #include "zypp/FileChecker.h" +#include "zypp/Fetcher.h"
using std::endl; using std::set; @@ -214,7 +215,12 @@ repo_excpt.remember(RepoException(_("No url in repository."))); ZYPP_THROW(repo_excpt); } + + Fetcher fetcher; + fetcher.addCachePath( info.packagesPath() );
+ MIL << "Added cache path " << info.packagesPath() << endl; + for ( RepoInfo::urls_const_iterator it = info.baseUrlsBegin(); it != info.baseUrlsEnd(); /* incremented in the loop */ ) @@ -227,11 +233,17 @@ << "' from " << url << endl; shared_ptr<MediaSetAccess> access = _impl->mediaAccessForUrl(url); _impl->setVerifierForRepo(repo_r, access); + + fetcher.enqueue( loc_r ); + + // FIXME: works for packages only
you mean not for patches and patterns? so what?
Just a reminder when bugs will pop up ;-)
+ fetcher.start( info.packagesPath(), *access );
- ManagedFile ret( access->provideFile(loc_r) ); + // reached if no exception has been thrown, so this is the correct file + ManagedFile ret( info.packagesPath() + loc_r.filename() );
std::string scheme( url.getScheme() ); - if ( scheme == "http" || scheme == "https" || scheme == "ftp" ) + if ( !info.keepPackages() )
so will this delete files from my DVD?
No, the code above will copy the package to the local cache always (not optimal, but so be it). Therefore, it will delete it from the cache. Stano -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Thu, Feb 14, Stanislav Visnovsky wrote:
========= Added 'packagesdir' option to zypp.conf to re-define a prefix to keep the downloaded packages. Default is '/var/cache/zypp/packages'
{ // does the current file exists in the current cache? Pathname cached_file = *it_cache + (*it_res)->location.filename(); + + MIL << "Trying cached file: " << cached_file << endl; + if ( PathInfo( cached_file ).isExist() ) { + MIL << "File exist, testing checksum " << (*it_res)->location.checksum() << endl; + // check the checksum
It might be worth maintaining an index by checksum for the cached files. If you look for a package in repo A, the same package (same checksum) might be already available in repo B. We should not download it to the new location as well. Another usefull feature would be to allow to 'inject' packages created by a delta into the chache. The package provider had to look for the package by checksum before trying to use delta. So there would be no need to apply the delta twice. If provided via a delta, the recreated package could be injected into the cache. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Dňa Thursday 14 February 2008 15:54:24 Michael Andres ste napísal:
On Thu, Feb 14, Stanislav Visnovsky wrote:
========= Added 'packagesdir' option to zypp.conf to re-define a prefix to keep the downloaded packages. Default is '/var/cache/zypp/packages'
{ // does the current file exists in the current cache? Pathname cached_file = *it_cache + (*it_res)->location.filename(); + + MIL << "Trying cached file: " << cached_file << endl; + if ( PathInfo( cached_file ).isExist() ) { + MIL << "File exist, testing checksum " << (*it_res)->location.checksum() << endl; + // check the checksum
It might be worth maintaining an index by checksum for the cached files.
If you look for a package in repo A, the same package (same checksum) might be already available in repo B. We should not download it to the new location as well.
This might be doable with having all existing caches for repos in Fetcher. But then it might slow things down. Or, having something smart.
Another usefull feature would be to allow to 'inject' packages created by a delta into the chache. The package provider had to look for the package by checksum before trying to use delta. So there would be no need to apply the delta twice. If provided via a delta, the recreated package could be injected into the cache.
Yes, that certainly a nice idea. Stano -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (3)
-
Martin Vidner -
Michael Andres -
Stanislav Visnovsky