[opensuse] I just found some thing about rsync I didn't know.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Many of us use rsync to copy files because it can be interrupted and it recovers where we left it. But there is one caveat: not the current file it is transmitting. I was sending a backup with partition images from one machine to another, and I interrupted it. Then I noticed that the directory shrinked in size: the 160000MiB file of which it had sent half of it, was deleted on interrupt, and will have to be sent in full again. Well, it makes sense, but I did not know. - -- Cheers Carlos E. R. (from 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCX4TbIRwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVf/8An3WdPW4JAdUh5l3syNpc J4/H+ALcAJ9KOF27s9mHQPxTTGq2LCaPRFooCQ== =agTf -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tuesday 13 October 2020, Carlos E. R. wrote:
Hi,
Many of us use rsync to copy files because it can be interrupted and it recovers where we left it. But there is one caveat: not the current file it is transmitting.
I was sending a backup with partition images from one machine to another, and I interrupted it. Then I noticed that the directory shrinked in size: the 160000MiB file of which it had sent half of it, was deleted on interrupt, and will have to be sent in full again.
Well, it makes sense, but I did not know.
-- Cheers
Carlos E. R. (from 15.1 x86_64 at Telcontar)
Note there is the --partial option for such cases. Cheers, Michael -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/12/20 4:00 PM, Michael Hamilton wrote:
On Tuesday 13 October 2020, Carlos E. R. wrote:
Hi,
Many of us use rsync to copy files because it can be interrupted and it recovers where we left it. But there is one caveat: not the current file it is transmitting.
I was sending a backup with partition images from one machine to another, and I interrupted it. Then I noticed that the directory shrinked in size: the 160000MiB file of which it had sent half of it, was deleted on interrupt, and will have to be sent in full again.
Well, it makes sense, but I did not know.
-- Cheers
Carlos E. R. (from 15.1 x86_64 at Telcontar)
Note there is the --partial option for such cases.
Interesting! But, if using --partial is without risk, why not is it the default? Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tuesday 13 October 2020, Lew Wolfgang wrote:
On 10/12/20 4:00 PM, Michael Hamilton wrote:
On Tuesday 13 October 2020, Carlos E. R. wrote:
Hi,
Many of us use rsync to copy files because it can be interrupted and it recovers where we left it. But there is one caveat: not the current file it is transmitting.
I was sending a backup with partition images from one machine to another, and I interrupted it. Then I noticed that the directory shrinked in size: the 160000MiB file of which it had sent half of it, was deleted on interrupt, and will have to be sent in full again.
Well, it makes sense, but I did not know.
-- Cheers
Carlos E. R. (from 15.1 x86_64 at Telcontar)
Note there is the --partial option for such cases.
Interesting! But, if using --partial is without risk, why not is it the default?
Regards, Lew
Perhaps it's not about risk, but more to do with consistency or correctness. I suspect the author(s) preferred a proper cleanup of temporary files so that the files resident and space consumed would at least be partially consistent with the original. Cheers, Michael -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13/10/2020 02.25, Michael Hamilton wrote:
On Tuesday 13 October 2020, Lew Wolfgang wrote:
On 10/12/20 4:00 PM, Michael Hamilton wrote:
On Tuesday 13 October 2020, Carlos E. R. wrote:
Hi,
Many of us use rsync to copy files because it can be interrupted and it recovers where we left it. But there is one caveat: not the current file it is transmitting.
I was sending a backup with partition images from one machine to another, and I interrupted it. Then I noticed that the directory shrinked in size: the 160000MiB file of which it had sent half of it, was deleted on interrupt, and will have to be sent in full again.
Well, it makes sense, but I did not know.
Ah! --partial keep partially transferred files --partial-dir=DIR put a partially transferred file into DIR -P same as --partial --progress ... --partial By default, rsync will delete any partially trans- ferred file if the transfer is interrupted. In some circumstances it is more desirable to keep partially transferred files. Using the --partial option tells rsync to keep the partial file which should make a subsequent transfer of the rest of the file much faster. --partial-dir=DIR A better way to keep partial files than the --par- tial option is to specify a DIR that will be used to hold the partial data (instead of writing it out to the destination file). On the next trans- fer, rsync will use a file found in this dir as data to speed up the resumption of the transfer and then delete it after it has served its pur- pose. ... Very interesting, thanks.
Note there is the --partial option for such cases.
Interesting! But, if using --partial is without risk, why not is it the default?
I thought you would be interested ;-)
Perhaps it's not about risk, but more to do with consistency or correctness.
I suspect the author(s) preferred a proper cleanup of temporary files so that the files resident and space consumed would at least be partially consistent with the original.
I think I will use: --partial --partial-dir=.rsync-partial -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On Tue, 13 Oct 2020 06:43:17 +0200 "Carlos E. R." <robin.listas@telefonica.net> wrote:
On 13/10/2020 02.25, Michael Hamilton wrote:
On Tuesday 13 October 2020, Lew Wolfgang wrote:
On 10/12/20 4:00 PM, Michael Hamilton wrote:
On Tuesday 13 October 2020, Carlos E. R. wrote:
Hi,
Many of us use rsync to copy files because it can be interrupted and it recovers where we left it. But there is one caveat: not the current file it is transmitting.
I was sending a backup with partition images from one machine to another, and I interrupted it. Then I noticed that the directory shrinked in size: the 160000MiB file of which it had sent half of it, was deleted on interrupt, and will have to be sent in full again.
Well, it makes sense, but I did not know.
Ah!
--partial keep partially transferred files --partial-dir=DIR put a partially transferred file into DIR
-P same as --partial --progress
...
--partial By default, rsync will delete any partially trans- ferred file if the transfer is interrupted. In some circumstances it is more desirable to keep partially transferred files. Using the --partial option tells rsync to keep the partial file which should make a subsequent transfer of the rest of the file much faster.
--partial-dir=DIR A better way to keep partial files than the --par- tial option is to specify a DIR that will be used to hold the partial data (instead of writing it out to the destination file). On the next trans- fer, rsync will use a file found in this dir as data to speed up the resumption of the transfer and then delete it after it has served its pur- pose. ...
Very interesting, thanks.
Note there is the --partial option for such cases.
Interesting! But, if using --partial is without risk, why not is it the default?
I thought you would be interested ;-)
Perhaps it's not about risk, but more to do with consistency or correctness.
I suspect the author(s) preferred a proper cleanup of temporary files so that the files resident and space consumed would at least be partially consistent with the original.
I think I will use:
--partial --partial-dir=.rsync-partial
I don't think you use --partial as well as --partial-dir, you use just one of them. Personally I would choose a name like RSYNC-PARTIAL rather than .rsync-partial so it was pretty obvious to me if anything had gone wrong/hadn't finished properly. PS I didn't know about this either so thanks to everybody who's contributed :) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13/10/2020 16.10, Dave Howorth wrote:
On Tue, 13 Oct 2020 06:43:17 +0200 "Carlos E. R." <> wrote:
Perhaps it's not about risk, but more to do with consistency or correctness.
I suspect the author(s) preferred a proper cleanup of temporary files so that the files resident and space consumed would at least be partially consistent with the original.
I think I will use:
--partial --partial-dir=.rsync-partial
I don't think you use --partial as well as --partial-dir, you use just one of them.
You are right. Using both rsync crashes after a restart.
Personally I would choose a name like RSYNC-PARTIAL rather than .rsync-partial so it was pretty obvious to me if anything had gone wrong/hadn't finished properly.
Yes, I used the exact example of the man page, but your version is more vissible. Although I see it instantly, I have a terminal with 'mc' view of the destination directory of the rsync to observe progress.
PS I didn't know about this either so thanks to everybody who's contributed :)
:-) I wonder what more surprises lurk in rsync to be discovered ;-D -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On 10/13/20 11:29 AM, Carlos E. R. wrote:
On 13/10/2020 16.10, Dave Howorth wrote:
On Tue, 13 Oct 2020 06:43:17 +0200 "Carlos E. R." <> wrote:
Perhaps it's not about risk, but more to do with consistency or correctness.
I suspect the author(s) preferred a proper cleanup of temporary files so that the files resident and space consumed would at least be partially consistent with the original. I think I will use:
--partial --partial-dir=.rsync-partial
I don't think you use --partial as well as --partial-dir, you use just one of them.
You are right. Using both rsync crashes after a restart.
Personally I would choose a name like RSYNC-PARTIAL rather than .rsync-partial so it was pretty obvious to me if anything had gone wrong/hadn't finished properly.
Yes, I used the exact example of the man page, but your version is more vissible. Although I see it instantly, I have a terminal with 'mc' view of the destination directory of the rsync to observe progress.
PS I didn't know about this either so thanks to everybody who's contributed :) :-)
I wonder what more surprises lurk in rsync to be discovered ;-D
Excellent thread. Follow up question . . . Why would --partial-dir "speed up the resumption of the transfer" (from the man page) unless that directory was on different storage? It seems to me that if it is on the same volume/disk, there would be no advantage. --dg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13/10/2020 17.47, DennisG wrote:
On 10/13/20 11:29 AM, Carlos E. R. wrote:
On 13/10/2020 16.10, Dave Howorth wrote:
On Tue, 13 Oct 2020 06:43:17 +0200 "Carlos E. R." <> wrote:
Perhaps it's not about risk, but more to do with consistency or correctness.
I suspect the author(s) preferred a proper cleanup of temporary files so that the files resident and space consumed would at least be partially consistent with the original. I think I will use:
--partial --partial-dir=.rsync-partial
I don't think you use --partial as well as --partial-dir, you use just one of them.
You are right. Using both rsync crashes after a restart.
with this error: rsync: connection unexpectedly closed (663 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.3] rsync: connection unexpectedly closed (664 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.3]
Personally I would choose a name like RSYNC-PARTIAL rather than .rsync-partial so it was pretty obvious to me if anything had gone wrong/hadn't finished properly.
Yes, I used the exact example of the man page, but your version is more vissible. Although I see it instantly, I have a terminal with 'mc' view of the destination directory of the rsync to observe progress.
PS I didn't know about this either so thanks to everybody who's contributed :) :-)
I wonder what more surprises lurk in rsync to be discovered ;-D
Excellent thread. Follow up question . . .
Why would --partial-dir "speed up the resumption of the transfer" (from the man page) unless that directory was on different storage? It seems to me that if it is on the same volume/disk, there would be no advantage.
My guess - because without it the partial file could be anywhere in the (partial) tree of received files, and the program has to scan it (and the source tree) trying to see if there is a partial file somewhere. rsync doesn't create a log of progress that it can use to recover on interrupt. I noticed that with "--partial-dir" the directory is created after I hit ctrl-c to stop the process, and the partial file is moved inside. And on restart, it is erased after the file is completely downloaded. I noticed because the file is 162000MB over 100M ethernet and I had time to see the changes :-) While the file is being received, it gets a randomized extension, and a preceding dot: .sda3_Crypt.khtk5X (the final file is "sda3_Crypt) I have not played with --partial alone. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On Tue, 13 Oct 2020 18:00:19 +0200 "Carlos E.R." <robin.listas@gmx.es> wrote:
On 13/10/2020 17.47, DennisG wrote:
On 10/13/20 11:29 AM, Carlos E. R. wrote:
On 13/10/2020 16.10, Dave Howorth wrote:
On Tue, 13 Oct 2020 06:43:17 +0200 "Carlos E. R." <> wrote:
I think I will use:
--partial --partial-dir=.rsync-partial
I don't think you use --partial as well as --partial-dir, you use just one of them.
You are right. Using both rsync crashes after a restart.
with this error:
rsync: connection unexpectedly closed (663 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.3] rsync: connection unexpectedly closed (664 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.3]
That's probably worth bug-reporting. Crashes are not nice. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13/10/2020 18.07, Dave Howorth wrote:
On Tue, 13 Oct 2020 18:00:19 +0200 "Carlos E.R." <> wrote:
On 13/10/2020 17.47, DennisG wrote:
On 10/13/20 11:29 AM, Carlos E. R. wrote:
On 13/10/2020 16.10, Dave Howorth wrote:
On Tue, 13 Oct 2020 06:43:17 +0200 "Carlos E. R." <> wrote:
I think I will use:
--partial --partial-dir=.rsync-partial
I don't think you use --partial as well as --partial-dir, you use just one of them.
You are right. Using both rsync crashes after a restart.
with this error:
rsync: connection unexpectedly closed (663 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.3] rsync: connection unexpectedly closed (664 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.3]
That's probably worth bug-reporting. Crashes are not nice.
You are right, it should report bad command line syntax. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2020-10-13 at 18:49 +0200, Carlos E. R. wrote:
On 13/10/2020 18.07, Dave Howorth wrote:
On Tue, 13 Oct 2020 18:00:19 +0200 "Carlos E.R." <> wrote:
On 13/10/2020 17.47, DennisG wrote:
On 10/13/20 11:29 AM, Carlos E. R. wrote:
On 13/10/2020 16.10, Dave Howorth wrote:
On Tue, 13 Oct 2020 06:43:17 +0200 "Carlos E. R." <> wrote: > I think I will use: > > --partial --partial-dir=.rsync-partial
I don't think you use --partial as well as --partial-dir, you use just one of them.
You are right. Using both rsync crashes after a restart.
with this error:
rsync: connection unexpectedly closed (663 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.3] rsync: connection unexpectedly closed (664 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.3]
That's probably worth bug-reporting. Crashes are not nice.
You are right, it should report bad command line syntax.
I will give more context on what I am doing before telling the failure I just saw (it is a bit long and not important, if you are busy). I use one external disk for doing backups of "old-laptop", booted with the openSUSE XFCE live stick (which has persistence), also called rescue stick. I decided I would instead install 15.2 on that same hard disk, and use the rest of the disk as a large backup space, using LUKS encrypted btrfs with compression (I don't like btrfs, but it is the only filesystem I know on Linux that supports compression). This allows to have transparent rsync with a compressed tree, and some compression on the partition images (I hope). To do this I had first to create a backup of the backup, as an image of the partition on my desktop machine. Done. Repartition and install 15.2 on said HD: done. Next task was recover the backup from the image: here comes the said rsync of this thread. So I have been running this script: +++.................. #!/bin/bash OPTIONS="--archive --acls --xattrs --hard-links --sparse --partial-dir=.rsync-partial --stats --human-readable --compress" # --partial # --password-file=/data/password Error: The --password-file option may only be used when accessing an rsync daemon DATE=`date --rfc-3339=seconds` echo "$DATE Starting rsync of: Portatil_entero_8" time rsync $OPTIONS root@telcontar.valinor://data/Datum/mountpoint/Portatil_entero_8/ /data/Portatil_entero_8 DATE=`date --rfc-3339=seconds` #skip 7, already transferred #echo "$DATE Empezando rsync de: Portatil_entero_7" #time rsync $OPTIONS root@telcontar.valinor://data/Datum/mountpoint/Portatil_entero_7/ /data/Portatil_entero_7 DATE=`date --rfc-3339=seconds` echo "$DATE Empezando rsync de: Portatil_entero_6" time rsync $OPTIONS root@telcontar.valinor://data/Datum/mountpoint/Portatil_entero_6/ /data/Portatil_entero_6 ..................++- Now, this was going to take days, because the Ethernet connection on that "old-laptop" is 100M, and the disk was on an USB2 converter and socket. Then I had a "brilliant idea™": use instead the "small-but-newer-laptop", which has an eth1000 and one USB3 socket. Why did I not think of this before? So I aborted the ongoing rsync: +++............ Erebor:/data # ./recrear_rsync_ssh 2020-10-13 13:30:55+02:00 Empezando rsync de: Portatil_entero_8 Password: ^Crsync error: received SIGUSR1 (code 19) at main.c(1455) [receiver=3.1.3] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(642) [generator=3.1.3] rsync: [generator] write error: Broken pipe (32) real 268m55.872s user 29m27.024s sys 33m17.537s 2020-10-13 17:59:52+02:00 Empezando rsync de: Portatil_entero_6 ............++- ... and switched hardware. I had to do some loops to get it to boot with that hard disk, but done. Notice it is the same physical hard disk and operating system instance, just that placed on a different USB bay, this time USB3. So I adjust the script a bit, because the "small-but-newer-laptop" is less powerful but has 1G eth, thus I disable compression: +++............ !/bin/bash OPTIONS="--archive --acls --xattrs --hard-links --sparse --partial-dir=.rsync-partial --stats --human-readable" # --partial # --compress remove when using the lenovo (less cpu power and eth 1000) # --password-file=/data/password Error: The --password-file option may only be used when accessing an rsync daemon DATE=`date --rfc-3339=seconds` echo "$DATE Empezando rsync de: Portatil_entero_8" time rsync $OPTIONS root@telcontar.valinor://data/Datum/mountpoint/Portatil_entero_8/ /data/Portatil_entero_8 DATE=`date --rfc-3339=seconds` echo "$DATE Empezando rsync de: Portatil_entero_7" time rsync $OPTIONS root@telcontar.valinor://data/Datum/mountpoint/Portatil_entero_7/ /data/Portatil_entero_7 DATE=`date --rfc-3339=seconds` echo "$DATE Empezando rsync de: Portatil_entero_6" time rsync $OPTIONS root@telcontar.valinor://data/Datum/mountpoint/Portatil_entero_6/ /data/Portatil_entero_6 ... etc ............++- And then I restart the rsync script, getting an error: +++............ Erebor:/data # ./recrear_rsync_ssh 2020-10-13 19:01:21+02:00 Empezando rsync de: Portatil_entero_8 Password: rsync: connection unexpectedly closed (663 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.3] rsync: connection unexpectedly closed (664 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.3] real 0m34.137s user 0m15.376s sys 0m3.173s 2020-10-13 19:01:56+02:00 Empezando rsync de: Portatil_entero_7 ............++- Huh? :-O Why? I looked at the script, at the receiving tree, saw the .rsync-partial directory with half of a six gig image file, and try again. Pains to waste it. Maybe I edited some white space in the script. I didn't see the reason for the cryptical error (I still don't). +++............ Erebor:/data # ./recrear_rsync_ssh 2020-10-13 19:06:23+02:00 Empezando rsync de: Portatil_entero_8 Password: Number of files: 25 (reg: 24, dir: 1) Number of created files: 12 (reg: 12) Number of deleted files: 0 Number of regular files transferred: 12 Total file size: 230.44G bytes Total transferred file size: 46.38G bytes Literal data: 42.43G bytes Matched data: 3.95G bytes File list size: 656 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 497.69K Total bytes received: 42.44G sent 497.69K bytes received 42.44G bytes 33.27M bytes/sec total size is 230.44G speedup is 5.43 real 21m15.388s user 10m19.641s sys 17m23.443s Erebor:/data # ............++- It worked. You know that humans are stupid and try again the same thing expecting a different result, but computers are stubborn and produce the same result. Except when they don't... :-) (ah, I have all the text output because I use the laptop from an ssh session from my desktop machine) And the entire exercise may be moot, if the hard links created in the original backup are not recreated: the disk will fill up before finishing. - -- Cheers, Carlos E. R. (from openSUSE 15.1 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCX4XrPxwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVAFIAoJMBgTCklPK3hdoMqX9v jKirTx5IAJ0ZrexXWBWjT7MiKsvG/PP47B/Kvw== =0rxu -----END PGP SIGNATURE-----
participants (6)
-
Carlos E. R. -
Carlos E.R. -
Dave Howorth -
DennisG -
Lew Wolfgang -
Michael Hamilton