[zypp-devel] Fetcher and hardlinking
Right now Fetcher copies identical files from the configured caches to the destination path. When downloading metadata, Fetcher uses a tmp directory to download all metadata, and it gets identical files from the repos raw metadata directories. Normally, the raw metadata dir of the same repo we are downloading is the best candidate to have the identical files. Then the original repo dir is renamed to something tmp too, and the tmp dir is renamed to the repo raw metadata dir (this allows to keep always consistant raw metadata copies). I have experimented locally so when a identical file is found in a previous cache, I hardlink instead of copy, at the end I don't care which directory is deleted first, because hard links are "refcounted". My refresh is much faster now. But Fetcher is also used in other places appart of repository metadata download, and there, I have no idea if the cache dirs are in different filesystem than the destination directory. I tested and ln will fail. Can I do hardlink by default, and fallback to copy? Is there anything else I could be missing? If not, I will commit this :-) Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Hi, On Mon, 18 Feb 2008, Duncan Mac-Vicar P. wrote:
But Fetcher is also used in other places appart of repository metadata download, and there, I have no idea if the cache dirs are in different filesystem than the destination directory. I tested and ln will fail.
Can I do hardlink by default, and fallback to copy?
Yes, this will always work. Ciao, Michael. -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Mon, Feb 18, Duncan Mac-Vicar P. wrote:
But Fetcher is also used in other places appart of repository metadata download, and there, I have no idea if the cache dirs are in different filesystem than the destination directory. I tested and ln will fail.
if ( PathInfo( cache dirs ).dev() != PathInfo( destination ).dev() )
Can I do hardlink by default, and fallback to copy?
You can easily adapt filesystem::hardlink to copy if ::link fails with EXDEV. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Michael Andres wrote:
On Mon, Feb 18, Duncan Mac-Vicar P. wrote:
But Fetcher is also used in other places appart of repository metadata download, and there, I have no idea if the cache dirs are in different filesystem than the destination directory. I tested and ln will fail.
if ( PathInfo( cache dirs ).dev() != PathInfo( destination ).dev() )
How expensive is this? If most of the cases will succeed with ln, I think it is faster to fallback to copy if ln fails. Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Mon, Feb 18, Duncan Mac-Vicar Prett wrote:
Michael Andres wrote:
On Mon, Feb 18, Duncan Mac-Vicar P. wrote:
But Fetcher is also used in other places appart of repository metadata download, and there, I have no idea if the cache dirs are in different filesystem than the destination directory. I tested and ln will fail.
if ( PathInfo( cache dirs ).dev() != PathInfo( destination ).dev() )
How expensive is this? If most of the cases will succeed with ln, I think it is faster to fallback to copy if ln fails.
Meant to be an example on how to detect different filesystems. The prefered way is of course to have a method like filesytem::hardlink, but trying a copy if ::link fails with the mentioned errno. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (4)
-
Duncan Mac-Vicar P. -
Duncan Mac-Vicar Prett
-
Michael Andres -
Michael Matz