[zypp-devel] zypper miscounting when DownloadInAdvance used
Hi there! When you switch on DownloadInAdvance, the DownloadResolvableReport is still call, even if the package is already in cache. This causes this callback is called twice for each package, and zypper has no good way of counting the so-far-downloaded packages. I see no easy way of avoiding this for 11.2. The information that the package is available in cache is buried deep in the call stack in Fetcher::Impl: target::TargetImpl::commit(with PackageCache) target::CommitPackageCache::get target::CommitPackageCache::sourceProvidePackage TargetImpl::RepoProvidePackage::operator() DownloadResolvableReport sent from here -. repo::PackageProvider::providePackage <-----------' repo::PackageProvider::doProvidePackage repo::Impl::RepoMediaAccess::provideFile Fetcher::start Fetcher::Impl::start Fetcher::Impl::provideToDest Fetcher::Impl::provideFromCache The way i see it, we'll need to add bool Fetcher::isCached public method and improve repo::PackageProvider to be able to use Fetcher::isCached before it calls any callback (well, in fact, it should report DownloadResolvableReport::gotFromCache). Maybe add bool repo::PackageProvider::provideFromCache and only use the current code flow if this returns false. Or (maybe even better) we should completely avoid calling repo::PackageProvider and get the files from cache in somewhat more direct way (e.g. by remembering the paths of files already provided by previous calls to PackageProvider during the download phase), if possible. Any ideas? (BTW, in the meantime i'll fix zypper by blindly resetting the counter when it reaches the expected number of packages to download.) https://bugzilla.novell.com/show_bug.cgi?id=545295 -- cheers, jano Ján Kupec YaST team ---------------------------------------------------------(PGP)--- Key ID: 637EE901 Fingerprint: 93B9 C79B 2D20 51C3 800B E09B 8048 46A6 637E E901 ---------------------------------------------------------(IRC)--- Server: irc.freenode.net Nick: jniq Channels: #zypp #yast #suse #susecz ---------------------------------------------------------(EOF)---
On Friday 06 November 2009 14:15:52 Jano Kupec wrote:
The way i see it, we'll need to add bool Fetcher::isCached public method and improve repo::PackageProvider to be able to use Fetcher::isCached before it calls any callback (well, in fact, it should report DownloadResolvableReport::gotFromCache). Maybe add bool repo::PackageProvider::provideFromCache and only use the current code flow if this returns false.
No. No one want's to know how the package is provided. I'll arrange the PackageProvider to check for a chache hit with the required checksum before actually starting the download.
Or (maybe even better) we should completely avoid calling repo::PackageProvider and get the files from cache in somewhat more direct way (e.g. by remembering the paths of files already provided by previous calls to PackageProvider during the download phase), if possible.
It's the PackageProviders task to get the package in the most direct way. If the provider is buggy or can be optimized, this should be done instead of working arround the bug. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Friday 06 November 2009 23:16:44 Michael Andres wrote:
On Friday 06 November 2009 14:15:52 Jano Kupec wrote:
The way i see it, we'll need to add bool Fetcher::isCached public method and improve repo::PackageProvider to be able to use Fetcher::isCached before it calls any callback (well, in fact, it should report DownloadResolvableReport::gotFromCache). Maybe add bool repo::PackageProvider::provideFromCache and only use the current code flow if this returns false.
No. No one want's to know how the package is provided. I'll arrange the PackageProvider to check for a chache hit with the required checksum before actually starting the download.
Or (maybe even better) we should completely avoid calling repo::PackageProvider and get the files from cache in somewhat more direct way (e.g. by remembering the paths of files already provided by previous calls to PackageProvider during the download phase), if possible.
It's the PackageProviders task to get the package in the most direct way. If the provider is buggy or can be optimized, this should be done instead of working arround the bug.
libzypp-6.22.1 no longer reports cached packages as being downloaded. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On 11/12/2009 12:33 PM, Michael Andres wrote:
On Friday 06 November 2009 23:16:44 Michael Andres wrote:
On Friday 06 November 2009 14:15:52 Jano Kupec wrote:
The way i see it, we'll need to add bool Fetcher::isCached public method and improve repo::PackageProvider to be able to use Fetcher::isCached before it calls any callback (well, in fact, it should report DownloadResolvableReport::gotFromCache). Maybe add bool repo::PackageProvider::provideFromCache and only use the current code flow if this returns false.
No. No one want's to know how the package is provided. I'll arrange the
You would be surprised :O) See below.
PackageProvider to check for a chache hit with the required checksum before actually starting the download.
Or (maybe even better) we should completely avoid calling repo::PackageProvider and get the files from cache in somewhat more direct way (e.g. by remembering the paths of files already provided by previous calls to PackageProvider during the download phase), if possible.
It's the PackageProviders task to get the package in the most direct way. If the provider is buggy or can be optimized, this should be done instead of working arround the bug.
libzypp-6.22.1 no longer reports cached packages as being downloaded.
Would be nice if it called back to app if it gets the file from cache. Without it it's difficult to show progress like retrieving X of Y, since we don't know Y. Looking into the future, it would be nice to be able to get such info in advance, before commit. That way the apps can show info like "need to get 100 MiB of packages (120 MiB is already cached)" or "need to get 20 of 30 packages (10 are already cached)". And then the commit progress. -- cheers, jano Ján Kupec YaST team ---------------------------------------------------------(PGP)--- Key ID: 637EE901 Fingerprint: 93B9 C79B 2D20 51C3 800B E09B 8048 46A6 637E E901 ---------------------------------------------------------(IRC)--- Server: irc.freenode.net Nick: jniq Channels: #zypp #yast #suse #susecz ---------------------------------------------------------(EOF)---
On 11/27/2009 01:16 PM, Jano Kupec wrote:
On 11/12/2009 12:33 PM, Michael Andres wrote:
On Friday 06 November 2009 23:16:44 Michael Andres wrote:
On Friday 06 November 2009 14:15:52 Jano Kupec wrote:
The way i see it, we'll need to add bool Fetcher::isCached public method and improve repo::PackageProvider to be able to use Fetcher::isCached before it calls any callback (well, in fact, it should report DownloadResolvableReport::gotFromCache). Maybe add bool repo::PackageProvider::provideFromCache and only use the current code flow if this returns false.
No. No one want's to know how the package is provided. I'll arrange the
You would be surprised :O) See below.
PackageProvider to check for a chache hit with the required checksum before actually starting the download.
Or (maybe even better) we should completely avoid calling repo::PackageProvider and get the files from cache in somewhat more direct way (e.g. by remembering the paths of files already provided by previous calls to PackageProvider during the download phase), if possible.
It's the PackageProviders task to get the package in the most direct way. If the provider is buggy or can be optimized, this should be done instead of working arround the bug.
libzypp-6.22.1 no longer reports cached packages as being downloaded.
Would be nice if it called back to app if it gets the file from cache. Without it it's difficult to show progress like retrieving X of Y, since we don't know Y.
Looking into the future, it would be nice to be able to get such info in advance, before commit. That way the apps can show info like "need to get 100 MiB of packages (120 MiB is already cached)" or "need to get 20 of 30 packages (10 are already cached)". And then the commit progress.
Another issue is calculation of bytes that need to be downloaded _before_ we start the commit [1][2]. We do want to know how the package is provided. Whether from cache and whether a delta/patch rpm will be used. And we want to know this before the commit. [1] https://bugzilla.novell.com/show_bug.cgi?id=325922 [2] https://bugzilla.novell.com/show_bug.cgi?id=410897 -- cheers, jano Ján Kupec YaST team ---------------------------------------------------------(PGP)--- Key ID: 637EE901 Fingerprint: 93B9 C79B 2D20 51C3 800B E09B 8048 46A6 637E E901 ---------------------------------------------------------(IRC)--- Server: irc.freenode.net Nick: jniq Channels: #zypp #yast #suse #susecz ---------------------------------------------------------(EOF)---
On Wednesday 16 December 2009 15:05:56 Jano Kupec wrote:
Looking into the future, it would be nice to be able to get such info in advance, before commit. That way the apps can show info like "need to get 100 MiB of packages (120 MiB is already cached)" or "need to get 20 of 30 packages (10 are already cached)". And then the commit progress.
Another issue is calculation of bytes that need to be downloaded _before_ we start the commit [1][2]. We do want to know how the package is provided. Whether from cache and whether a delta/patch rpm will be used. And we want to know this before the commit.
Ok. I was thinking about adding some convenience methods to allow downloading and removing packages to/from the local disk cache anyway. I often just want to get a .rpm to my local disk, without solving and installing it. I can add some more convenience to to tell you if, and how much will be downloaded. I can also guess whether a delta will be considered. For sure we know this at the time we actually need it, and whether it works we only know after the download. For factory we'll switch from the libzypp computed installation order to the one the satsolver provides. It does a better job in breaking up dependency cycles and detecting the need to delay package deletions. We also want to offer parallel downloads while installing. Together with the new install order there will be a transaction class that basically contains all the information the zypper installation summary shows. I take care it also offers a size and download statistic. Then you can use it as base for your summary screen. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (2)
-
Jano Kupec -
Michael Andres