[opensuse] puzzled by rsync
source & target have same filesystem size, block size, filesystem type OS 11.4 running on both # rsync -av --delete --stats --progress --exclude 'lost+found' source/ target Rsync halted because the target reached 100% full, even though the source was about 95% full at the start. I reduced source consumption to 85%, deleted some files on target to drop consumption to 90%, then reran rsync. At the end, source remained 85%, while target is at 94%. What's going on? Is my syntax not doing what I expect (mirroring)? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 09 Dec 2013 20:36:07 -0500 Felix Miata wrote:
source & target have same filesystem size, block size, filesystem type OS 11.4 running on both
# rsync -av --delete --stats --progress --exclude 'lost+found' source/ target
Rsync halted because the target reached 100% full, even though the source was about 95% full at the start. I reduced source consumption to 85%, deleted some files on target to drop consumption to 90%, then reran rsync. At the end, source remained 85%, while target is at 94%. What's going on? Is my syntax not doing what I expect (mirroring)?
Hi Felix, Whenever I run into puzzling results with rsync, I always use the '-n' flag ("dry run" so nothing is actually written) and pipe the verbose output to a text file. Then I can casually inspect what's going on. hth & regards, Carl -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-12-09 20:47 (GMT-0500) Carl Hartung composed:
Whenever I run into puzzling results with rsync, I always use the '-n' flag ("dry run" so nothing is actually written) and pipe the verbose output to a text file. Then I can casually inspect what's going on.
I used -n first, without piping, and saw nothing interesting in the scrollback. After seeing your and John's responses, I simply repeated, upon which all output messages were about deletes. Now both source and target are at 85% usage. Exactly what pipe syntax do you use, or do you simply redirect with ">"? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 09 Dec 2013 21:41:27 -0500 Felix Miata wrote:
On 2013-12-09 20:47 (GMT-0500) Carl Hartung composed:
Whenever I run into puzzling results with rsync, I always use the '-n' flag ("dry run" so nothing is actually written) and pipe the verbose output to a text file. Then I can casually inspect what's going on.
I used -n first, without piping, and saw nothing interesting in the scrollback. After seeing your and John's responses, I simply repeated, upon which all output messages were about deletes. Now both source and target are at 85% usage.
Exactly what pipe syntax do you use, or do you simply redirect with ">"?
Sorry for the sloppy language, Felix. In a local context it's just a redirect. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-12-10 02:36, Felix Miata wrote:
source & target have same filesystem size, block size, filesystem type
What filesystem? Not btrfs?
What's going on? Is my syntax not doing what I expect (mirroring)?
I would use 'mc', displaying both sides. Then make it to calculate directory sizes (command menu). If there is a directory with different sizes, get into it in both sides, and again calculate directory size. Repeat that till you find out what is different. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
On Tue, 10 Dec 2013, Felix Miata wrote:
source & target have same filesystem size, block size, filesystem type OS 11.4 running on both
# rsync -av --delete --stats --progress --exclude 'lost+found' source/ target
Rsync halted because the target reached 100% full, even though the source was about 95% full at the start. I reduced source consumption to 85%, deleted some files on target to drop consumption to 90%, then reran rsync. At the end, source remained 85%, while target is at 94%. What's going on? Is my syntax not doing what I expect (mirroring)? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation)
Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!
Felix Miata *** http://fm.no-ip.com/
Any sparse files? Perhaps try rsync's --sparse option. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
В Tue, 10 Dec 2013 17:01:07 +1300
Michael Hamilton
On Tue, 10 Dec 2013, Felix Miata wrote:
source & target have same filesystem size, block size, filesystem type OS 11.4 running on both
# rsync -av --delete --stats --progress --exclude 'lost+found' source/ target
Rsync halted because the target reached 100% full, even though the source was about 95% full at the start. I reduced source consumption to 85%, deleted some files on target to drop consumption to 90%, then reran rsync. At the end, source remained 85%, while target is at 94%. What's going on? Is my syntax not doing what I expect (mirroring)? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation)
Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!
Felix Miata *** http://fm.no-ip.com/
Any sparse files? Perhaps try rsync's --sparse option.
And --hard-links ... although they are a bit tricky when doing incremental updates. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/09/2013 10:01 PM, Michael Hamilton wrote:
On Tue, 10 Dec 2013, Felix Miata wrote:
source & target have same filesystem size, block size, filesystem type OS 11.4 running on both
# rsync -av --delete --stats --progress --exclude 'lost+found' source/ target
Rsync halted because the target reached 100% full, even though the source was about 95% full at the start. I reduced source consumption to 85%, deleted some files on target to drop consumption to 90%, then reran rsync. At the end, source remained 85%, while target is at 94%. What's going on? Is my syntax not doing what I expect (mirroring)? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation)
Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!
Felix Miata *** http://fm.no-ip.com/ Any sparse files? Perhaps try rsync's --sparse option.
I had trouble with the "delete" option because (I think) of the gnome fs directory "gvfs" (?) - anyway - if I told rsync to ignore it then the delete would start working again. Hope this helps, Duaine -- Duaine Hechler Piano, Player Piano, Pump Organ - Tuning, Servicing & Rebuilding (314) 838-5587 / dahechler@att.net / www.hechlerpianoandorgan.com Home & Business user of Linux - 13 years -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Monday 09 of December 2013 20:36:07 Felix Miata wrote:
source & target have same filesystem size, block size, filesystem type OS 11.4 running on both
# rsync -av --delete --stats --progress --exclude 'lost+found' source/ target
Rsync halted because the target reached 100% full, even though the source was about 95% full at the start. I reduced source consumption to 85%, deleted some files on target to drop consumption to 90%, then reran rsync. At the end, source remained 85%, while target is at 94%. What's going on? Is my syntax not doing what I expect (mirroring)?
Do both filesystems have the same size of reserved space? For ext4 check with tune2fs -l /dev/[partition node] | grep Reserved, the setting is "Reserved block count". I set this to 0 for the filesystems of a backup, because during file transfer disk usage is twice that of the file, the issue that Linda mentioned before. This has a negative effect on fragmentation, though. Regards, Peter -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Dec 9, 2013 at 8:36 PM, Felix Miata
source & target have same filesystem size, block size, filesystem type OS 11.4 running on both
# rsync -av --delete --stats --progress --exclude 'lost+found' source/ target
Rsync halted because the target reached 100% full, even though the source was about 95% full at the start. I reduced source consumption to 85%, deleted some files on target to drop consumption to 90%, then reran rsync. At the end, source remained 85%, while target is at 94%. What's going on? Is my syntax not doing what I expect (mirroring)?
Felix, Modern filesystems use preallocation. Per http://ext2.sourceforge.net/2005-ols/paper-html/node6.html ext2 already had it by 2005. That means when a file is created and written to they automatically allocate a unused tail at the end of each file. Then some time later (hours / days) they have a background kernel thread that scavenges any tails that are still unused. The positive is that files (like logs) growing slowly over time won't get fragmented so badly. The bad is that for highspeed filesystem filling tasks like a massive rsync, the disk usage is anomalously high for a while (hours / days). With XFS you can disable pre-allocation via the allocsize mount parameter. (That parameter has been around many years. so yes 11.4 has preallocation for XFS at a minimum and ext3/ext4 I think.) allocsize=size Sets the buffered I/O end-of-file preallocation size when doing delayed allocation writeout (default size is 64KiB). Valid values for this option are page size (typically 4KiB) through to 1GiB, inclusive, in power-of-2 increments. size = 0 disables preallocation and is probably smart on your distination backup disk. I assume other filesystems have a way to disable preallocation as well. FYI: I don't know how to determine the total amount of preallocation space on a filesystem. I'm sure it can be done somehow. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-12-10 15:48, Greg Freemyer wrote:
That means when a file is created and written to they automatically allocate a unused tail at the end of each file.
Then some time later (hours / days) they have a background kernel thread that scavenges any tails that are still unused.
Is it possible to do that scavenging on request and fast, on a requested partition? -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
On Tue, Dec 10, 2013 at 9:55 AM, Carlos E. R.
On 2013-12-10 15:48, Greg Freemyer wrote:
That means when a file is created and written to they automatically allocate a unused tail at the end of each file.
Then some time later (hours / days) they have a background kernel thread that scavenges any tails that are still unused.
Is it possible to do that scavenging on request and fast, on a requested partition?
Note that I know of. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-------- Original Message --------
Subject: Re: puzzled by rsync
Date: Tue, 10 Dec 2013 00:11:47 -0800
From: L.A. Walsh
source & target have same filesystem size, block size, filesystem type OS 11.4 running on both
Couple of scenarios to rule out: 1) going from disk1 a -> 500G file b->empty to disk2 a=>empty b->500G file ---- --delete = --delete-during which means it won't try to delete a later encountered directory until it gets to copying 'b'. If your disk is 750G, your copy will die in the middle of copying "a". Maybe delete-before would be better, though theoretically less safe, in that you might be deleting 500G before the copy on disk1 has safely made it to disk2. But if you are tight on disk space.... 2) If your target disk is more fragmented, then it will need more free-space extents(clusters) -- rather than 1 range with 2G, it might need 20 ranges with 100MB each. That means disk 2 will need more "indirect" space to hold the extra ranges. Those are possibilities that come to mind off the top of my head... -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (9)
-
Andrey Borzenkov -
auxsvr＠gmail.com -
Carl Hartung -
Carlos E. R. -
Duaine Hechler -
Felix Miata -
Greg Freemyer -
Linda Walsh -
Michael Hamilton