[opensuse] slightly off topic - rsync in cygwin
All, Is there a way with rsync to control the blocksize of data read / written at a time? For local directory copies, is there a higher performing tool? fyi: I know dd can do this faster, but in this case I'm making a copy of 261 files. rsync is in theory a perfect tool, but it is only running at 50% of theoretical max speed. I did a test in openSUSE a couple days ago and hit 100% by using a blocksize of 100 MB. I'd like to figure out how to optimize rsync, or find a similar recursive folder copy tool that can achieve close to 100% of theoretical max speed. == details == I'm starting more and more to use rsync to move large files from one USB-3 drive to another. (all files are 1.5 GB in my work today.) Often in openSUSE, but today I'm using cygwin/windows 8.1 I think this is a more generic rsync question. (I can test rsync in openSUSE later today.) My individual source and destination drives can hit 140 MB/sec (via USB 3). But I'm only getting a throughput of 70 MB/sec. I'm thinking something similar to this is happening. while (files) { read 1.5 GB file to ram write 1.5 GB file from ram fsync() ensure 1.5 GB file is on disk } endwhile I have no complaint about the fsync and I'm actually happy it is there. It would be great if I could control the read / write blocksize. I suspect 100MB or less is the best performance (1 MB is probably too small, but the current size is too big. My ultimate would be something like: while (files) { while (data_in_file) { read user_defined_blocksize to ram from file write user_defined_blocksize from ram to file } fsync() ensure 1.5 GB file is on disk } endwhile Greg -- Greg Freemyer www.IntelligentAvatar.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
11.04.2016 20:25, Greg Freemyer пишет:
All,
Is there a way with rsync to control the blocksize of data read / written at a time?
For local directory copies, is there a higher performing tool?
fyi: I know dd can do this faster, but in this case I'm making a copy of 261 files. rsync is in theory a perfect tool, but it is only running at 50% of theoretical max speed. I did a test in openSUSE a couple days ago and hit 100% by using a blocksize of 100 MB.
I'd like to figure out how to optimize rsync, or find a similar recursive folder copy tool that can achieve close to 100% of theoretical max speed.
== details ==
I'm starting more and more to use rsync to move large files from one USB-3 drive to another. (all files are 1.5 GB in my work today.)
Often in openSUSE, but today I'm using cygwin/windows 8.1
I think this is a more generic rsync question. (I can test rsync in openSUSE later today.)
My individual source and destination drives can hit 140 MB/sec (via USB 3).
But I'm only getting a throughput of 70 MB/sec.
Try "rsync -W" to disable delta computation; otherwise rsync may read each file twice (first to compute checksum, second to actually copy it).
I'm thinking something similar to this is happening.
while (files) { read 1.5 GB file to ram write 1.5 GB file from ram fsync() ensure 1.5 GB file is on disk } endwhile
I have no complaint about the fsync and I'm actually happy it is there. It would be great if I could control the read / write blocksize.
I suspect 100MB or less is the best performance (1 MB is probably too small, but the current size is too big.
My ultimate would be something like:
while (files) { while (data_in_file) { read user_defined_blocksize to ram from file write user_defined_blocksize from ram to file } fsync() ensure 1.5 GB file is on disk } endwhile
Greg -- Greg Freemyer www.IntelligentAvatar.net
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Apr 11, 2016 at 1:39 PM, Andrei Borzenkov
I think this is a more generic rsync question. (I can test rsync in openSUSE later today.)
My individual source and destination drives can hit 140 MB/sec (via USB 3).
But I'm only getting a throughput of 70 MB/sec.
Try "rsync -W" to disable delta computation; otherwise rsync may read each file twice (first to compute checksum, second to actually copy it).
No impact. (At least not in cygwin. I just tried it.) Note that my destination files don't exist, so there is no destination file for rsync to read. Greg -- Greg Freemyer www.IntelligentAvatar.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 2016-04-11 at 13:25 -0400, Greg Freemyer wrote:
All,
Is there a way with rsync to control the blocksize of data read / written at a time?
Not that comes to mind and I'm sure you have RTFM as well as I might.
For local directory copies, is there a higher performing tool?
cp -a generally works well
fyi: I know dd can do this faster, but in this case I'm making a copy of 261 files. rsync is in theory a perfect tool, but it is only running at 50% of theoretical max speed. I did a test in openSUSE a couple days ago and hit 100% by using a blocksize of 100 MB.
I'd like to figure out how to optimize rsync, or find a similar recursive folder copy tool that can achieve close to 100% of theoretical max speed.
You have disabled compression and chosen appropriate difference detection as Andrei suggests?
== details ==
I'm starting more and more to use rsync to move large files from one USB-3 drive to another. (all files are 1.5 GB in my work today.)
Often in openSUSE, but today I'm using cygwin/windows 8.1
I think this is a more generic rsync question. (I can test rsync in openSUSE later today.)
My individual source and destination drives can hit 140 MB/sec (via USB 3).
But I'm only getting a throughput of 70 MB/sec.
I'm thinking something similar to this is happening.
while (files) { read 1.5 GB file to ram write 1.5 GB file from ram fsync() ensure 1.5 GB file is on disk } endwhile
I have no complaint about the fsync and I'm actually happy it is there. It would be great if I could control the read / write blocksize.
I suspect 100MB or less is the best performance (1 MB is probably too small, but the current size is too big.
My ultimate would be something like:
while (files) { while (data_in_file) { read user_defined_blocksize to ram from file write user_defined_blocksize from ram to file } fsync() ensure 1.5 GB file is on disk } endwhile
That algorithm would indeed result in 70 MB/sec throughput. It would be better if the program did both reading and writing simultaneously, using a circular buffer chasing its tail. I have no idea whether rsync, or for that mater cp, does something like that. But with open source at least it's possible to check :) Cheers, Dave
Greg -- Greg Freemyer www.IntelligentAvatar.net
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Apr 11, 2016 at 4:39 PM, Dave Howorth
Is there a way with rsync to control the blocksize of data read / written at a time?
Not that comes to mind and I'm sure you have RTFM as well as I might.
I did RTFM before asking. Nothing obvious in there, but there could be side effects from some of the options.
For local directory copies, is there a higher performing tool?
cp -a generally works well
When doing a copy of a TB or so of data, I prefer a tool that is more robust. rsync if killed and restarted will simply restart where it left off. Note, I'm not only talking about an error condition causing the copy to die. If I'm beating the hell out of disk with a large copy and I need to use the drive otherwise for half an hour, I just kill the rsync and re-run it when the disk can be dedicated again. Greg -- Greg Freemyer www.IntelligentAvatar.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 2016-04-11 at 17:03 -0400, Greg Freemyer wrote:
On Mon, Apr 11, 2016 at 4:39 PM, Dave Howorth
wrote: Is there a way with rsync to control the blocksize of data read / written at a time?
Not that comes to mind and I'm sure you have RTFM as well as I might.
I did RTFM before asking. Nothing obvious in there, but there could be side effects from some of the options.
For local directory copies, is there a higher performing tool?
cp -a generally works well
When doing a copy of a TB or so of data, I prefer a tool that is more robust. rsync if killed and restarted will simply restart where it left off.
Note, I'm not only talking about an error condition causing the copy to die. If I'm beating the hell out of disk with a large copy and I need to use the drive otherwise for half an hour, I just kill the rsync and re-run it when the disk can be dedicated again.
I've never had a problem with cp either on local disks or on NFS mounts. I have some jobs that take days and happily coexist with other work. For some long-running jobs I do start them with nice or ionice so other jobs can take priority if necessary. Cheers, Dave
Greg -- Greg Freemyer www.IntelligentAvatar.net
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Apr 11, 2016 at 5:24 PM, Dave Howorth
On Mon, 2016-04-11 at 17:03 -0400, Greg Freemyer wrote:
On Mon, Apr 11, 2016 at 4:39 PM, Dave Howorth
wrote: Is there a way with rsync to control the blocksize of data read / written at a time?
Not that comes to mind and I'm sure you have RTFM as well as I might.
I did RTFM before asking. Nothing obvious in there, but there could be side effects from some of the options.
For local directory copies, is there a higher performing tool?
cp -a generally works well
When doing a copy of a TB or so of data, I prefer a tool that is more robust. rsync if killed and restarted will simply restart where it left off.
Note, I'm not only talking about an error condition causing the copy to die. If I'm beating the hell out of disk with a large copy and I need to use the drive otherwise for half an hour, I just kill the rsync and re-run it when the disk can be dedicated again.
I've never had a problem with cp either on local disks or on NFS mounts. I have some jobs that take days and happily coexist with other work. For some long-running jobs I do start them with nice or ionice so other jobs can take priority if necessary.
Cheers, Dave
Well, Linux is pretty good at sharing at rotating disk. But remember I said I was running rsync in cygwin (windows). It absolutely sucks at multitasking a disk. Throughput can drop a factor of 10 easy because you are doing 2 things on the same disk. :( Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Apr 11, 2016 at 6:46 PM, Greg Freemyer
On Mon, Apr 11, 2016 at 5:24 PM, Dave Howorth
wrote: On Mon, 2016-04-11 at 17:03 -0400, Greg Freemyer wrote:
On Mon, Apr 11, 2016 at 4:39 PM, Dave Howorth
wrote: Is there a way with rsync to control the blocksize of data read / written at a time?
Not that comes to mind and I'm sure you have RTFM as well as I might.
I did RTFM before asking. Nothing obvious in there, but there could be side effects from some of the options.
For local directory copies, is there a higher performing tool?
cp -a generally works well
When doing a copy of a TB or so of data, I prefer a tool that is more robust. rsync if killed and restarted will simply restart where it left off.
Note, I'm not only talking about an error condition causing the copy to die. If I'm beating the hell out of disk with a large copy and I need to use the drive otherwise for half an hour, I just kill the rsync and re-run it when the disk can be dedicated again.
I've never had a problem with cp either on local disks or on NFS mounts. I have some jobs that take days and happily coexist with other work. For some long-running jobs I do start them with nice or ionice so other jobs can take priority if necessary.
Cheers, Dave
Well, Linux is pretty good at sharing at rotating disk.
But remember I said I was running rsync in cygwin (windows). It absolutely sucks at multitasking a disk. Throughput can drop a factor of 10 easy because you are doing 2 things on the same disk. :(
Greg
Actually I just moved my HDDs from Windows to an old openSUSE laptop. It was one of the first laptops to have USB-3. I'm getting much better speeds from a dedicated rsync. More like 90 MB/sec instead of 70 MB/sec That might even be the max speed of the source disk, so this whole thread is looking totally off-topic. Closing it down. Thanks Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-04-11 19:25, Greg Freemyer wrote:
I'd like to figure out how to optimize rsync, or find a similar recursive folder copy tool that can achieve close to 100% of theoretical max speed.
As compared to dd? I don't think you can. rsync, or any file copy program, has to search files. Ie, read directory entries, inodes, then data, which can also be non contiguous. To speed the process, you need to cache the entire metadata of the partition in RAM. Reading from a flash disk would be much faster: if this is close to the theoretical max speed, it would prove my hypothesis. I'm assuming the data is only read once, though, but rsync does, or can do, read/write verification. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlcMTeIACgkQja8UbcUWM1wYPQD/SP1kCnQHg7CJNDr0NuX4obME CweXz7g3Xx9pU5qeEQoA/1AcjJ5S8Jao2TC15ReLxOPqY+4sGg9++T92RnxFbqVm =NtHC -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (4)
-
Andrei Borzenkov
-
Carlos E. R.
-
Dave Howorth
-
Greg Freemyer