[opensuse] rsync between two years
Hello, The web is filled with rsync examples, but I found none that fit my needs an find and rsync options are really hard to manage. I have a tree of various files I use frequently, that get larger each year. I want to archive them by year or range of years (example from start to 2010, from 2011 to 2012... at the end nothing must be ommitted I have a lot of 32Gb sd cards and the whole file shouldn't be more than 3 or 4 cards. What is the best way to do this? I would like to use the --delete option, because I may remove old files on source thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/11/2018 07:13 PM, jdd@dodin.org wrote:
Hello,
The web is filled with rsync examples, but I found none that fit my needs an find and rsync options are really hard to manage.
I have a tree of various files I use frequently, that get larger each year.
I want to archive them by year or range of years (example from start to 2010, from 2011 to 2012... at the end nothing must be ommitted
I have a lot of 32Gb sd cards and the whole file shouldn't be more than 3 or 4 cards.
What is the best way to do this?
TBH this sounds more like a one-time moving task rather than a regular sync. Of course, you can use rsync as part of the solution, but it would be interesting to understand more what you want to do. Do you have a random, nested directory structure like dir/a/file-from-2010 dir/a/file-from-2012 dir/a/x/file-from-2013 dir/b/c/file-from-2014 dir/file-from-2011 ... and want to archive the files based on their modification timestamp?
I would like to use the --delete option, because I may remove old files on source
The --delete option does not delete on SRC side, but would remove files/directories on the DST side if the file does not exist on SRC side. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 11/01/2018 à 19:24, Bernhard Voelker a écrit :
TBH this sounds more like a one-time moving task rather than a regular sync.
only in part. I often notice than some files are duplicated so I can remove them without loss, I try to keep the same on archives
Of course, you can use rsync as part of the solution, but it would be interesting to understand more what you want to do.
split the source by dates, to make browsing the archives easier
Do you have a random, nested directory structure like
dir/a/file-from-2010 dir/a/file-from-2012 dir/a/x/file-from-2013 dir/b/c/file-from-2014 dir/file-from-2011
no, not there. I have a nested by date structure for the real "collections" (photos, videos), but not for the day to day files that are more fitted by other classification
I would like to use the --delete option, because I may remove old files on source
The --delete option does not delete on SRC side, but would remove files/directories on the DST side if the file does not exist on SRC side.
exactly. I often find duplicates an want to remove the from all archives (I have several copies of the archives, some older than others) the folder I want to archive I name "mes-documents" (my-docs in french) and I drop them where I need them. I would like to use rsync to sort them by date while keeping the directory structure. Say, I have somewhere a folder "taxes" and I want to archive them as taxes from 2000 to 2010 for example jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-11 19:51, jdd@dodin.org wrote:
Le 11/01/2018 à 19:24, Bernhard Voelker a écrit :
I would like to use the --delete option, because I may remove old files on source
The --delete option does not delete on SRC side, but would remove files/directories on the DST side if the file does not exist on SRC side.
exactly. I often find duplicates an want to remove the from all archives (I have several copies of the archives, some older than others)
I don't think you understand the --delete action. The intention is that you do a copy, say from a directory to a memory card. Some days later you repeat the same operation: same directory and same card. Well, the --delete option will delete on the backup the files that are no longer in the original. Also, if you make a mistake and put the wrong memory card (a backup of another directory), it will delete the complete card. For example, you backup "photos" to card A, and "taxes" to card B. If another day you use rsync with --delete of "photos" to card B, then card B is deleted and you lose the backup of the taxes. So be very careful with --delete. Happened to me.
the folder I want to archive I name "mes-documents" (my-docs in french) and I drop them where I need them.
I would like to use rsync to sort them by date while keeping the directory structure.
Say, I have somewhere a folder "taxes" and I want to archive them as taxes from 2000 to 2010 for example
I don't see how to sort by date with rsync, but perhaps. Me, I would perhaps create a secondary tree of hardlinks of the files sorted by date, then backup that. Hard links do not use extra space. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Le 11/01/2018 à 23:52, Carlos E. R. a écrit :
I don't think you understand the --delete action.
yes I do, I use it all the time
Me, I would perhaps create a secondary tree of hardlinks of the files sorted by date, then backup that. Hard links do not use extra space.
hard link on other device?? jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2018-01-12 at 09:02 +0100, jdd@dodin.org wrote:
Le 11/01/2018 à 23:52, Carlos E. R. a écrit :
I don't think you understand the --delete action.
yes I do, I use it all the time
Me, I would perhaps create a secondary tree of hardlinks of the files sorted by date, then backup that. Hard links do not use extra space.
hard link on other device??
No, on the source device. You create a secondary tree built of hardlinks to the original tree, in the order that you want to do the backup. Then do the backup of that one. I use this when I want to have to different sort methods fo the same bunch of files. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpYuoIACgkQtTMYHG2NR9XVrQCfVLD6csekUV8Kw+m8lZPI9pIv yc4An1CJllxWQGIiP8oC92NGJfPSCq00 =ZyR5 -----END PGP SIGNATURE-----
On 01/11/2018 07:51 PM, jdd@dodin.org wrote:
Do you have a random, nested directory structure like
dir/a/file-from-2010 dir/a/file-from-2012 dir/a/x/file-from-2013 dir/b/c/file-from-2014 dir/file-from-2011
no, not there. I have a nested by date structure for the real "collections" (photos, videos), but not for the day to day files that are more fitted by other classification
you missed the 2nd half of the question: do you want to archive based on the timestamps of the files, or is the date part of the folders you're putting the files in? In the former case (timestamps), the solution may be a combination of of 'find' and 'rsync' (or 'mv'/'cp', depends); e.g. the following find the files with modification time in the year 2012: $ find . -newermt 2012-01-01 ! -newermt 2013-01-01 -ls So you could use that to move all those files into a 2012 folder (untested!): $ find . -type f -newermt 2012-01-01 ! -newermt 2013-01-01 -print0 \ | rsync -HAXaxi --from0 --files-from=- . /path2/archive-2012/. If you want to remove the files from the source directory, then I'd redirect the 'find' output to a file, and use it once for rsync and once for 'xargs -0 rm' as input. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 12/01/2018 à 00:46, Bernhard Voelker a écrit :
do you want to archive based on the timestamps of the files,
this. I want to keep the original directory tree, but only with the files of the given dates
So you could use that to move all those files into a 2012 folder (untested!):
$ find . -type f -newermt 2012-01-01 ! -newermt 2013-01-01 -print0 \ | rsync -HAXaxi --from0 --files-from=- . /path2/archive-2012/.
It's nearly what I expected, thanks. I use the (mounted) path of the sd card as destination, works as root (to preserve permissions) and can omit the first -newer for a simple "from date". Use of ! on the second newer is very smart. however, I don't fully understand how it works. Specially I guess find send the file names one at a time, but do rsync take them one by one, copying them one at a time, or as a list, waiting to the list to be complete to begin writing. In other words, will the option --delete works? I'm also surprised to see the "dot", I was thinking of the files as source and if I get it well, file list is only a test on the "." source that need to be set. the "-i" output is not that easy to understand neither anyway, I wouldn't have got that command line without help, thanks a lot! jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/12/2018 12:16 PM, jdd@dodin.org wrote:
Le 12/01/2018 à 00:46, Bernhard Voelker a écrit :
So you could use that to move all those files into a 2012 folder (untested!):
$ find . -type f -newermt 2012-01-01 ! -newermt 2013-01-01 -print0 \ | rsync -HAXaxi --from0 --files-from=- . /path2/archive-2012/.
It's nearly what I expected, thanks.
I use the (mounted) path of the sd card as destination, works as root (to preserve permissions) and can omit the first -newer for a simple "from date". Use of ! on the second newer is very smart.
however, I don't fully understand how it works.
you can set up a test directory hierarchy: $ mkdir -p src/a/b src/c $ touch -d 2012-06-06 src/file-from-2012 $ touch -d 2013-06-06 src/a/file-from-2013 $ touch -d 2014-06-06 src/a/b/file-from-2014 Now you have: $ find -type f -exec ls -log '{}' + -rw-r--r-- 1 0 Jun 6 2014 ./src/a/b/file-from-2014 -rw-r--r-- 1 0 Jun 6 2013 ./src/a/file-from-2013 -rw-r--r-- 1 0 Jun 6 2012 ./src/file-from-2012 Selecting with only -newermt for files newer than Jan 1st of 2013 gives also the file from 2014: $ find -type f -newermt 2013-01-01 -exec ls -log '{}' + -rw-r--r-- 1 0 Jun 6 2014 ./src/a/b/file-from-2014 -rw-r--r-- 1 0 Jun 6 2013 ./src/a/file-from-2013 Adding the "! -newermt" (spoken "not newer than") will find only the files from 2013: $ find -type f -newermt 2013-01-01 ! -newermt 2014-01-01 -exec ls -log '{}' + -rw-r--r-- 1 0 Jun 6 2013 ./src/a/file-from-2013
Specially I guess find send the file names one at a time, but do rsync take them one by one, copying them one at a time, or as a list, waiting to the list to be complete to begin writing.
yes, find passes the name of the files one-by-one to rsync, and that will start the sync as early the names are available on the pipe.
In other words, will the option --delete works?
NO, --delete is operating on the DESTINATION. As far as I understood you would like to remove all the archived files from the SOURCE, right? As mentioned in my previous mail, I'd save the output of find in a file, and then pass that file a) to rsync for archiving, and then b) to "xargs rm" to delete it in the SOURCE directory. 1. Get file names: $ find . -type f -newermt 2012-01-01 ! -newermt 2013-01-01 -print0 \ > list 2. Archive the files: $ rsync -HAXaxi --from0 --files-from=- . /path2/archive-2012/. \ < list 3. Remove the files from the SOURCE: $ xargs -0 rm -v < list
I'm also surprised to see the "dot", I was thinking of the files as source and if I get it well, file list is only a test on the "." source that need to be set.
Well, I didn't find a proper example with the --files-from option in the man page, so usin "." was just guesswork. ;-)
the "-i" output is not that easy to understand neither
you could change that to '-v' if you are more familiar with it.
anyway, I wouldn't have got that command line without help, thanks a lot!
no worries! Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 12/01/2018 à 17:36, Bernhard Voelker a écrit :
Adding the "! -newermt" (spoken "not newer than") will find only the files from 2013, yes I was only wondering if removing "!" makes a second set with all what is not on the first, not to forget something in between
In other words, will the option --delete works?
NO, --delete is operating on the DESTINATION. As far as I understood you would like to remove all the archived files from the SOURCE, right?
wrong. If I happen to remove a file from the source, I want this to be also done on the copy If I first copy the file list in a file and give it as argument for --files-from, will --delete works? else I need an other line to first remove the extra files
I'm also surprised to see the "dot", I was thinking of the files as source and if I get it well, file list is only a test on the "." source that need to be set.
Well, I didn't find a proper example with the --files-from option in the man page, so using "." was just guesswork. ;-)
good guess :-)
the "-i" output is not that easy to understand neither
you could change that to '-v' if you are more familiar with it.
I tried, but it's less informative thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 12/01/2018 à 18:43, jdd@dodin.org a écrit :
If I first copy the file list in a file and give it as argument for --files-from, will --delete works?
tested, no it don't works. rsync takes the files in a from-file one at a time. presently, the only thing I find is to blank completely the target and then, using rsync is pointless :-( (may be it's a better cp) or is there a way to: * compare the file list on source and target * remove files on target that where previously removed on source the reason is that I have from time to time to clean this folder tree and I don't want to have to clean it again if I have to recover from the archives thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/12/2018 07:17 PM, jdd@dodin.org wrote:
Le 12/01/2018 à 18:43, jdd@dodin.org a écrit :
If I first copy the file list in a file and give it as argument for --files-from, will --delete works?
tested, no it don't works. rsync takes the files in a from-file one at a time.
What else would you expect? It gets a list of files, and processes them one after another. What's wrong with that? Sorry, I'm lost again.
presently, the only thing I find is to blank completely the target and then, using rsync is pointless :-( (may be it's a better cp)
Why erase from the target?
or is there a way to:
* compare the file list on source and target
rsync *does* compare the files on source and target; well, for performance reasons, it compares file size and timestamps only, but you could use the -c option to compare the full content.
* remove files on target that where previously removed on source
ah, that's what the --delete option is for: if file A exists in the destination but no longer in source, then it deletes A in the destination. $ mkdir src dst $ touch dst/myfile $ rsync -avx --delete src/. dst/. sending incremental file list deleting myfile ... Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/01/18 18:17, jdd@dodin.org wrote:
Le 12/01/2018 à 18:43, jdd@dodin.org a écrit :
If I first copy the file list in a file and give it as argument for --files-from, will --delete works?
tested, no it don't works. rsync takes the files in a from-file one at a time.
presently, the only thing I find is to blank completely the target and then, using rsync is pointless :-( (may be it's a better cp)
or is there a way to:
* compare the file list on source and target * remove files on target that where previously removed on source
the reason is that I have from time to time to clean this folder tree and I don't want to have to clean it again if I have to recover from the archives
This sound to me then, like you want to write a little script that uses hardlinks as others have suggested. So what you would do is scan the current directory for all files within the date range you want and hard-link them to your temporary directory. Then you recurse your script into all the subdirectories, linking those files. This will give you a temporary directory with all the files you want backed up. You can rsync this directory onto your card, and then delete it - because all the files have been hard-linked the files will be untouched and the directory will take up very little disk space. Incidentally, you said you sometimes have duplicate files you'd like to clean up - I've got a little utility script I run (that I got from a magazine somewhere) that finds duplicates. It's safe to run - run it over a directory and it creates two output files, one of which is a script that will delete the duplicates. All the lines in the script are commented out so you can edit it, see which files are duplicates, and delete the hash in front of the files you want to get rid of before running it. Hope it's useful... I've also got a couple of perl scripts that traverse directories like I suggested, but they don't do what you want - they would only serve as a template example. Cheers, Wol
Le 14/01/2018 à 00:48, Wols Lists a écrit :
This sound to me then, like you want to write a little script that uses hardlinks as others have suggested.
at first glance, I didn't understand the hard link question, but now I do, it's a bit like the filename file, but more sophisticated, I was thinking there was a simpler solution, but may be there is none in summary: build a tree with hard links, then rsync --delete it.
Incidentally, you said you sometimes have duplicate files you'd like to clean up
yes, but duplicates have to be managed by hand (reason why I don't want to do it several times), because it's very difficult to be sure of what one is to be removed thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 14/01/18 12:49 PM, jdd@dodin.org wrote:
yes, but duplicates have to be managed by hand (reason why I don't want to do it several times), because it's very difficult to be sure of what one is to be removed
For that, you might look at DirSyncProc https://en.wikipedia.org/wiki/DirSync_Pro -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 15/01/2018 à 12:58, Anton Aylward a écrit :
On 14/01/18 12:49 PM, jdd@dodin.org wrote:
yes, but duplicates have to be managed by hand (reason why I don't want to do it several times), because it's very difficult to be sure of what one is to be removed
For that, you might look at DirSyncProc
I will drop a look, thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 15/01/2018 à 13:07, jdd@dodin.org a écrit :
Le 15/01/2018 à 12:58, Anton Aylward a écrit :
On 14/01/18 12:49 PM, jdd@dodin.org wrote:
yes, but duplicates have to be managed by hand (reason why I don't want to do it several times), because it's very difficult to be sure of what one is to be removed
For that, you might look at DirSyncProc
I will drop a look, thanks
jdd
yep, seems to be very flexible, even scriptable :-) and seems to do what I need - so far :-) thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 15/01/2018 à 13:25, jdd@dodin.org a écrit :
and seems to do what I need - so far :-)
well no. Or there is something I don't understand I can change filters dates and nothing change in target when executing the job, neither writing nor deletion :-( jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/12/2018 06:43 PM, jdd@dodin.org wrote:
Le 12/01/2018 à 17:36, Bernhard Voelker a écrit :
In other words, will the option --delete works?
NO, --delete is operating on the DESTINATION. As far as I understood you would like to remove all the archived files from the SOURCE, right?
wrong. If I happen to remove a file from the source, I want this to be also done on the copy
If I first copy the file list in a file and give it as argument for --files-from, will --delete works? else I need an other line to first remove the extra files
sorry, I'm lost. The --delete option clearly operates on the destination: $ rsync --help | grep -F ' --delete ' --delete delete extraneous files from destination dirs For further discussion, please provide simple example directory hierarchies for SRC and DST (like I did), jow it should look like before and after a sync/archive/copy/delete (or whatever you want to achieve) so we can understand what you want/need. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 13/01/2018 à 00:03, Bernhard Voelker a écrit :
sorry, I'm lost. The --delete option clearly operates on the destination:
of course if, for example, I delete cache files from source I want them to be also deleted from destination where they where copied previously
For further discussion, please provide simple example directory hierarchies for SRC and DST (like I did),
can be anything, but same on source and destination, restricted to the dates it's because it's random than I need all this thanks jdd (NB: I will be offline for the next 72h) -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/12/2018 06:03 PM, Bernhard Voelker wrote:
On 01/12/2018 06:43 PM, jdd@dodin.org wrote:
Le 12/01/2018 à 17:36, Bernhard Voelker a écrit :
In other words, will the option --delete works?
NO, --delete is operating on the DESTINATION. As far as I understood you would like to remove all the archived files from the SOURCE, right?
wrong. If I happen to remove a file from the source, I want this to be also done on the copy
If I first copy the file list in a file and give it as argument for --files-from, will --delete works? else I need an other line to first remove the extra files
sorry, I'm lost. The --delete option clearly operates on the destination:
If you use the option '--delete' *and* the file has been removed from the server it will be removed from the destination. -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-11 19:13, jdd@dodin.org wrote:
Hello,
The web is filled with rsync examples, but I found none that fit my needs an find and rsync options are really hard to manage.
I have a tree of various files I use frequently, that get larger each year.
I want to archive them by year or range of years (example from start to 2010, from 2011 to 2012... at the end nothing must be ommitted
I have a lot of 32Gb sd cards and the whole file shouldn't be more than 3 or 4 cards.
I don't know that rsync can split a file across several cards. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Le 11/01/2018 à 19:37, Carlos E. R. a écrit :
I don't know that rsync can split a file across several cards.
not a file, but files grouped (I'm a bit short of English vocabulary :-() jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-11 19:42, jdd@dodin.org wrote:
Le 11/01/2018 à 19:37, Carlos E. R. a écrit :
I don't know that rsync can split a file across several cards.
not a file, but files grouped (I'm a bit short of English vocabulary :-()
Even so, rsync can not copy to several cards distributing the files. It does a single operation to a single destination. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Le 11/01/2018 à 23:41, Carlos E. R. a écrit :
Even so, rsync can not copy to several cards distributing the files. It does a single operation to a single destination.
it is possible to copy only files within some criteria, using find, for example: https://unix.stackexchange.com/questions/87018/find-and-rsync but I'm short of syntax details for using years as criteria thanks jd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2018-01-12 at 08:59 +0100, jdd@dodin.org wrote:
Le 11/01/2018 à 23:41, Carlos E. R. a écrit :
Even so, rsync can not copy to several cards distributing the files. It does a single operation to a single destination.
it is possible to copy only files within some criteria, using find, for example:
https://unix.stackexchange.com/questions/87018/find-and-rsync
but I'm short of syntax details for using years as criteria
I'd have to think about it. The idea is to run a single rsync command for a large bunch of files, or it becomes inefficient. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpYuucACgkQtTMYHG2NR9UmRACffVNbDK8io7lNGHEXwZjVWQde tMIAnRKRXRuOmDnsi7rw2XGnecGnq4in =hpZM -----END PGP SIGNATURE-----
Am Donnerstag, Januar 11, 2018 19:13 CET, "jdd@dodin.org"
Hello,
The web is filled with rsync examples, but I found none that fit my needs an find and rsync options are really hard to manage.
I have a tree of various files I use frequently, that get larger each year.
I want to archive them by year or range of years (example from start to 2010, from 2011 to 2012... at the end nothing must be ommitted
rsync can't do that. It has no options to select files by date alone. Try to create a second folder structure using find(1) and the option "-newermt" (see https://unix.stackexchange.com/questions/73268/how-to-move-the-files-based-o... for an example how to use a date range). Use hardlinks (ln(1) without -s) to "create" the files in the per-year directory structure, for example: -exec ln "{}" "/path/to/backup/2012/{}" \; Afterwards, you can use "find . -type f | sort > files.lst" on both folder structures and compare the two outputs to make sure you didn't miss anything. If the modification dates/permissions are different, use -printf to just include only name and file size. When you're sure, you can backup each year. Afterwards, you can use the find(1) commands which you used to create the links and replace -exec... with -delete to get rid of the saved files. The folders with the hardlinks, you can simply delete. Regards, -- Aaron "Optimizer" Digulla a.k.a. Philmann Dark "It's not the universe that's limited, it's our imagination. Follow me and I'll show you something beyond the limits." http://blog.pdark.de/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 12/01/2018 à 09:46, Aaron Digulla a écrit :
Try to create a second folder structure using find(1) and the option "-newermt" (see https://unix.stackexchange.com/questions/73268/how-to-move-the-files-based-o... for an example how to use a date range).
Hello, thanks, I try to follow this track. I can't make it work. find . -type f ! -newermt 2017-01-01 or find `pwd` -type f ! -newermt 2017-01-01 gives me files names from which I can't make a link may be because some file names are odd /data-ssd480/mesdocs-jdd/livres/cal/Dick,Philip K_/L'oeil dans le ciel (763)/L'oeil dans le ciel - Dick,Philip K_.epub but also ln problems find . -type f ! -newermt 2017-01-01 -exec ln "{}" "/data-ssd480/tmp/{}" \; gives: ln: impossible de créer le lien direct '/data-ssd480/tmp/./culte/mulet-reborn.bk/etc/sysconfig/mail' → './culte/mulet-reborn.bk/etc/sysconfig/mail': Aucun fichier ou dossier de ce type
Use hardlinks (ln(1) without -s) to "create" the files in the per-year directory structure, for example:
-exec ln "{}" "/path/to/backup/2012/{}" \;
in ln, may not the target be before the source? (tried and not seems to give better result) thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/15/2018 09:00 AM, jdd@dodin.org wrote:
I can't make it work.
find . -type f ! -newermt 2017-01-01
or
find `pwd` -type f ! -newermt 2017-01-01
gives me files names from which I can't make a link
may be because some file names are odd
/data-ssd480/mesdocs-jdd/livres/cal/Dick,Philip K_/L'oeil dans le ciel (763)/L'oeil dans le ciel - Dick,Philip K_.epub
you have blanks and other special characters in file names. The only safe way to handle this between tool is to use NUL-terminated strings.
but also ln problems
find . -type f ! -newermt 2017-01-01 -exec ln "{}" "/data-ssd480/tmp/{}" \;
gives:
ln: impossible de créer le lien direct '/data-ssd480/tmp/./culte/mulet-reborn.bk/etc/sysconfig/mail' → './culte/mulet-reborn.bk/etc/sysconfig/mail': Aucun fichier ou dossier de ce type
AFAIR my French, this means that the parent directory in the target does not exist. I'm surprised why so many guys here suggest creating a hierarchy with hardlinks with the files falling in the year range. Well, you could also store it in a database, or write it on a paper and send it around the world before processing further. ;-) As I already wrote: - let find create a NUL-terminated list of files from e.g. 2013: $ cd DIR $ find . -type f -newermt 2013-01-01 ! -newermt 2014-01-01 -print0 > list - let e.g. copy the files from the list to the backup medium: $ mkdir BACKUP/2013 $ rsync -HAXaxv --from0 --files-from='list' '.' BACKUP/2013/. - remove the files from the source (after you have checked the backup): $ xargs -0 rm -v < 'list' Isn't this what you want? Actually that is just like moving, but ensuring that the directory structure is the same on the target. The following does the same with find/xargs/mkdir/mv: $ ( # gather list cd src \ && find . -type f -newermt 2013-01-01 ! -newermt 2014-01-01 -print0 \ ) | xargs -0 sh -c ' # move each file to dst, keeping the dir structure. while [ "$1" ]; do d="$(dirname "$1")" \ && mkdir -pv "dst/$d" \ && mv -v "src/$1" "dst/$d" \ && shift 1 \ || exit 255; # make xargs stop further processing done' ARCHIVE Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 15/01/2018 à 11:29, Bernhard Voelker a écrit :
you have blanks and other special characters in file names. The only safe way to handle this between tool is to use NUL-terminated strings.
hope it solve this part of the problem, but not all: ./culte/mulet-reborn.bk/etc/init.d/nmbln: failed to create hard link '/data-ssd480/tmp/./culte/mulet-reborn.bk/etc/init.d/nmb' => './culte/mulet-reborn.bk/etc/init.d/nmb': No such file or directory I think the problem come from the "/./" in the middle. This come from find issuing ./ on the beginning of every result? the blank spaces in filesnames are also a problem, mostly because there may also b single qotes in them (they are french books names, created by calibre)
I'm surprised why so many guys here suggest creating a hierarchy with hardlinks with the files falling in the year range. Well, you could also store it in a database, or write it on a paper and send it around the world before processing further. ;-)
the problem is the use of the --delete option. I want to be able to update the target from time to time and have it identical to the source. If a file is deleted from the source, it have to be also deleted from the target also. this should be allowed if the hard links folder have only the desired files
As I already wrote:
- let find create a NUL-terminated list of files from e.g. 2013:
$ cd DIR $ find . -type f -newermt 2013-01-01 ! -newermt 2014-01-01 -print0 > list
- let e.g. copy the files from the list to the backup medium:
$ mkdir BACKUP/2013 $ rsync -HAXaxv --from0 --files-from='list' '.' BACKUP/2013/.
this works, but I may get an error "no space left on device", so then I have to change the source range to be smaller, but I have to delete the now unneeded files *on the target*
- remove the files from the source (after you have checked the backup):
nope I want to keep all the files on the source and use several sd cards as target, each with different times thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/15/2018 12:23 PM, jdd@dodin.org wrote:
Le 15/01/2018 à 11:29, Bernhard Voelker a écrit :
you have blanks and other special characters in file names. The only safe way to handle this between tool is to use NUL-terminated strings.
hope it solve this part of the problem, but not all:
./culte/mulet-reborn.bk/etc/init.d/nmbln: failed to create hard link '/data-ssd480/tmp/./culte/mulet-reborn.bk/etc/init.d/nmb' => './culte/mulet-reborn.bk/etc/init.d/nmb': No such file or directory
I think the problem come from the "/./" in the middle. This come from find issuing ./ on the beginning of every result?
no, the problem is: "Not such file or directory" In the above context, I would think that ln did not find the file on the source tree.
the blank spaces in filesnames are also a problem, mostly because there may also b single qotes in them (they are french books names, created by calibre)
shouldn't be a problem (with correct quoting the file names).
I'm surprised why so many guys here suggest creating a hierarchy with hardlinks with the files falling in the year range. Well, you could also store it in a database, or write it on a paper and send it around the world before processing further. ;-)
the problem is the use of the --delete option.
I want to be able to update the target from time to time and have it identical to the source. If a file is deleted from the source, it have to be also deleted from the target also. this should be allowed if the hard links folder have only the desired files
...
I want to keep all the files on the source and use several sd cards as target, each with different times
ah, you want a repeatable script which manages the backup/sync/delete. Hmm, in that case having a "shadowed" tree with hardlinks might be okay: 1. create the hardlinks in a per-year directory tree. - remove all files from the hardlink-tree which have a link count of 1 (these are files which have been deleted from the source tree): $ find path-to-hardlink-tree -type f -link 1 -delete - invoke rsync with --delete. Well, depending on your data, you might get "filesystem full" errors when writing to the SD card, but you can think about it when that day comes. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 15/01/2018 à 18:10, Bernhard Voelker a écrit :
no, the problem is: "Not such file or directory" In the above context, I would think that ln did not find the file on the source tree.
found the problem. ln can't create path solution: use first find to do it find /data-ssd480/mesdocs-jdd/ -type d -exec mkdir -p /data-ssd480/tmp/{} \; then find /data-ssd480/mesdocs-jdd/ -type f ! -newermt 2016-01-01 -exec ln '{}' '/data-ssd480/tmp/{}' \; I can't use the date for folders, because the folders and the files dates do not match, but folders do not take much room let's try rsync after that :-) thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 16/01/18 08:52, jdd@dodin.org wrote:
Le 15/01/2018 à 18:10, Bernhard Voelker a écrit :
no, the problem is: "Not such file or directory" In the above context, I would think that ln did not find the file on the source tree.
found the problem. ln can't create path
solution: use first find to do it
find /data-ssd480/mesdocs-jdd/ -type d -exec mkdir -p /data-ssd480/tmp/{} \;
then
find /data-ssd480/mesdocs-jdd/ -type f ! -newermt 2016-01-01 -exec ln '{}' '/data-ssd480/tmp/{}' \;
That's why, in all my scripts that do this sort of thing, they take a directory argument and have three phases ... Phase 1 - set up the current directory as required Phase 2 - process all the files in the current directory (I tend to use Perl, and there's a function that reads the directory returning the names of all the files) Phase 3 - process all the sub-directories (same function, different args) and recurse into the directories. It might be a lot more verbose, but it's simpler to understand than one or two "find --exec" commands with lots of options you can never remember. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/11/2018 12:13 PM, jdd@dodin.org wrote:
I would like to use the --delete option, because I may remove old files on source
thanks jdd
My first thought is use use 'find' to dump the list of files you want to transfer for each period of time (each year) to a file (e.g. <yearfile>). Either using the -newermt <start date> ! -newermt <stop date> options, or using the same logic with -mtime. (I do this to shuffle package sets between different boxes) Then use the file you create containing all the files to tell rsync what files get transferred for each year. You can then use rsync with the options -uai --delete --no-R --files-from=<yearfile> / *note:* the '--no-R' (--no-relative) and '/' (or whatever your dest path is) are required when using --files-from= as --relative is implied when using --files-from= (the man page does a pretty good job explaining) -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 14/01/2018 à 08:47, David C. Rankin a écrit :
On 01/11/2018 12:13 PM, jdd@dodin.org wrote:
I would like to use the --delete option, because I may remove old files on source
thanks jdd
My first thought is use use 'find' to dump the list of files you want to transfer for each period of time (each year) to a file (e.g. <yearfile>). Either using the
-newermt <start date> ! -newermt <stop date>
I tested this and it don't seems to work. one of the reason I use it is when a target medium is full, with --delete-before to make room, this allows adding more target medium simply by changing one of the date with limited work
(the man page does a pretty good job explaining)
there is no real way to explain simply hard things :-(. The rsync man page is an example of very hard to follow text thanks jdd -- http://dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/14/2018 11:55 AM, jdd@dodin.org wrote:
Le 14/01/2018 à 08:47, David C. Rankin a écrit :
On 01/11/2018 12:13 PM, jdd@dodin.org wrote:
I would like to use the --delete option, because I may remove old files on source
thanks jdd
My first thought is use use 'find' to dump the list of files you want to transfer for each period of time (each year) to a file (e.g. <yearfile>). Either using the
-newermt <start date> ! -newermt <stop date>
I tested this and it don't seems to work.
Hmm.. Not sure where the hiccup is. Let's say you want all files from 2014, you can do: find . -maxdepth 1 -type f -newermt 2013-12-31 ! -newermt 2015-01-01
one of the reason I use it is when a target medium is full, with --delete-before to make room, this allows adding more target medium simply by changing one of the date with limited work
(the man page does a pretty good job explaining)
there is no real way to explain simply hard things :-(. The rsync man page is an example of very hard to follow text
Yes it can be clear as mud at times, but I suspect that is just due to how massively capable rsync is and the volume of information needed to accurately describe its workings. You could literally use the 'find` command above and parse per-year files with, e.g. for year in {2000..2018}; do find . -maxdepth 1 -type f -newermt $((year-1))-12-31 ! -newermt $((year+1))-01-01 done Whether you use --delete-before or --delete it should work the same. -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 14/01/18 12:55 PM, jdd@dodin.org wrote:
(the man page does a pretty good job explaining)
there is no real way to explain simply hard things :-(. The rsync man page is an example of very hard to follow text
+1 And, as JDD says, the examples on the WebOfABillionlies are never quite what one is looking for -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (8)
-
Aaron Digulla -
Anton Aylward -
Bernhard Voelker -
Carlos E. R. -
David C. Rankin -
jdd＠dodin.org -
Ken Schneider - openSUSE -
Wols Lists