[zypp-devel] Help needed for libzypp
Hi, I'm trying to get libzypp to not delete the /var/adm/mount/ temporary directory it uses when it downloads packages when retrying a download. This is to enable aria2c to use a partial download instead of starting from the beginning. I've used gdb to track down the creation of the directory in MediaHandler.cc but my time is limited and I would appreciate some help as to where the partially downloaded file is deleted on retry. Thanks Dave Plater -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Monday 12 October 2009 09:10:54 Dave Plater wrote:
Hi, I'm trying to get libzypp to not delete the /var/adm/mount/ temporary directory it uses when it downloads packages when retrying a download. This is to enable aria2c to use a partial download instead of starting from the beginning. I've used gdb to track down the creation of the directory in MediaHandler.cc but my time is limited and I would appreciate some help as to where the partially downloaded file is deleted on retry.
I'm not that familiar with the downloader workflows, but I suppose it's not that easy. The items are reference counted, so the directory gets deleted together with the last handler accessing it. The same way the (partially) downloaded file gets deleted as the last reference to it drops. Not deleting the directory won't help, as a new handler would create and use it's own dir. In order to retry an aborted partial download the error handling has to be changed. Currently the error is handled on a too high level. The file, the Handler and even the Fetcher class managing the additional checks (like checksum) are gone when you are asked whether to retry. A retry will not simply retry the last aria download, but it will restart the complete package providing workflow. Starting with checks for available delta rpms, it will process all the urls available for that repo (in case it has more than one). So one had to offer a retry at the level of MediaAria, i.e. while the partial file is still accessible. It needs some hack so the PackageProvider can make it's Report class available to MediaAria. Then MediaAria itself could report the partial download and ask whether to try to continue. You would stay inside MediaAria until the file is downloaded completely or failed. For testing one could auto-coninue a fix number of times and, if this works well, later replace the counter by a report. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+
On 10/12/2009 03:20 PM, Michael Andres wrote:
On Monday 12 October 2009 09:10:54 Dave Plater wrote:
Hi, I'm trying to get libzypp to not delete the /var/adm/mount/ temporary directory it uses when it downloads packages when retrying a download. This is to enable aria2c to use a partial download instead of starting from the beginning. I've used gdb to track down the creation of the directory in MediaHandler.cc but my time is limited and I would appreciate some help as to where the partially downloaded file is deleted on retry.
I'm not that familiar with the downloader workflows, but I suppose it's not that easy.
The items are reference counted, so the directory gets deleted together with the last handler accessing it. The same way the (partially) downloaded file gets deleted as the last reference to it drops. Not deleting the directory won't help, as a new handler would create and use it's own dir.
In order to retry an aborted partial download the error handling has to be changed. Currently the error is handled on a too high level. The file, the Handler and even the Fetcher class managing the additional checks (like checksum) are gone when you are asked whether to retry.
A retry will not simply retry the last aria download, but it will restart the complete package providing workflow. Starting with checks for available delta rpms, it will process all the urls available for that repo (in case it has more than one). When aria2c is restarted, if the partially downloaded file is still in the temp download directory it will resume that file, that's why I am trying to prevent its deletion before the download retry. My idea is to retry aria2c before it gets up to a higher level with a lower level retry counter as you seem to be suggesting below. I suppose I can look into MediaAria2c.cc and see if I can hook in there for now.
So one had to offer a retry at the level of MediaAria, i.e. while the partial file is still accessible. It needs some hack so the PackageProvider can make it's Report class available to MediaAria. Then MediaAria itself could report the partial download and ask whether to try to continue. You would stay inside MediaAria until the file is downloaded completely or failed.
For testing one could auto-coninue a fix number of times and, if this works well, later replace the counter by a report.
Regards Dave P -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Monday 12 October 2009 20:41:01 Dave Plater wrote:
On 10/12/2009 03:20 PM, Michael Andres wrote:
On Monday 12 October 2009 09:10:54 Dave Plater wrote:
When aria2c is restarted, if the partially downloaded file is still in the temp download directory it will resume that file, that's why I am trying to prevent its deletion before the download retry. My idea is to retry aria2c before it gets up to a higher level with a lower level retry counter as you seem to be suggesting below. I suppose I can look into MediaAria2c.cc and see if I can hook in there for now.
Yes. I'll have a look how we can get access ot the report callback, so we can replace the counter by asking the user whether to retry. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Tuesday 13 October 2009 10:59:43 Michael Andres wrote:
On Monday 12 October 2009 20:41:01 Dave Plater wrote:
On 10/12/2009 03:20 PM, Michael Andres wrote:
On Monday 12 October 2009 09:10:54 Dave Plater wrote:
When aria2c is restarted, if the partially downloaded file is still in the temp download directory it will resume that file, that's why I am trying to prevent its deletion before the download retry. My idea is to retry aria2c before it gets up to a higher level with a lower level retry counter as you seem to be suggesting below. I suppose I can look into MediaAria2c.cc and see if I can hook in there for now.
Yes. I'll have a look how we can get access ot the report callback, so we can replace the counter by asking the user whether to retry.
As zypper is already connected to MediaArias DownloadProgressReport, asking whether to retry an interrupted download seems to be quite easy. I checked in a new zypp/media/MediaAria2c.cc (0a19ae60): Prepare reporting of partial downloads so one could retry with --continue). If you set 'bool partialDownload' to true, the an interrupted download gets reported and you are a asked whether to retry. So you just have to detect the partial download and arrange the next try to use --continue. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (2)
-
Dave Plater
-
Michael Andres