On Mon, Dec 9, 2013 at 8:36 PM, Felix Miata <mrmazda@earthlink.net> wrote:
source & target have same filesystem size, block size, filesystem type OS 11.4 running on both
# rsync -av --delete --stats --progress --exclude 'lost+found' source/ target
Rsync halted because the target reached 100% full, even though the source was about 95% full at the start. I reduced source consumption to 85%, deleted some files on target to drop consumption to 90%, then reran rsync. At the end, source remained 85%, while target is at 94%. What's going on? Is my syntax not doing what I expect (mirroring)?
Felix, Modern filesystems use preallocation. Per <http://ext2.sourceforge.net/2005-ols/paper-html/node6.html> ext2 already had it by 2005. That means when a file is created and written to they automatically allocate a unused tail at the end of each file. Then some time later (hours / days) they have a background kernel thread that scavenges any tails that are still unused. The positive is that files (like logs) growing slowly over time won't get fragmented so badly. The bad is that for highspeed filesystem filling tasks like a massive rsync, the disk usage is anomalously high for a while (hours / days). With XFS you can disable pre-allocation via the allocsize mount parameter. (That parameter has been around many years. so yes 11.4 has preallocation for XFS at a minimum and ext3/ext4 I think.) allocsize=size Sets the buffered I/O end-of-file preallocation size when doing delayed allocation writeout (default size is 64KiB). Valid values for this option are page size (typically 4KiB) through to 1GiB, inclusive, in power-of-2 increments. size = 0 disables preallocation and is probably smart on your distination backup disk. I assume other filesystems have a way to disable preallocation as well. FYI: I don't know how to determine the total amount of preallocation space on a filesystem. I'm sure it can be done somehow. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org