On July 2, 2014 2:01:08 PM EDT, Anton Aylward
On 07/02/2014 12:58 PM, Greg Freemyer wrote: <snip>
But when I first copied the data between 2 drives I was only getting about 60 MB /sec throughput (120 MB combined reads and writes).
Suppose you use the 'copy' program from software bytes using the Fread() type buffering. You have a 512 byte read buffer and 512 byte write buffer. The core of you code does putchar(getchar()) that is, one at a time. And don't forget those 512 byte buffers.
Then there's using 'dd' with big buffers between files. Oh, right, files, which means the file system overhead, and quite possibly the allocation of new file segments and putting those refernces in the file map.
For my main use case I'm using ewfacquire as the user space app. It can compress the data stream, but I find the optimum clock time for the whole copy is with no compression. Ewfacquire works more like dd with tunable block sizes. I was testing with 32 KB blocks as I recall (I use the default transfer size). I will try even bigger blocks, but my testing with sata drives and dd showed 4KB blocks were only slightly slower than 1MB blocks.
Some file systems are faster than others.
I hate to admit that for the data transfer scenario I'm writing to ntfs formatted drives, so ntfs-3g is getting a major workout. I will test with ext4, btrfs and xfs just to see if the filesystem is a major bottleneck.
See http://www.phoronix.com/scan.php?page=article&item=linux_311_filesystems
as an example of different file systems under different loads and conditions.
You may want to look specifically at
http://www.phoronix.com/scan.php?page=article&item=usb20_usb30_flash&num=1
and
http://www.phoronix.com/scan.php?page=article&item=usb20_usb30_flash&num=1
You might also look at
http://www.phoronix.com/scan.php?page=article&item=linux_iosched_2012&num=1
There are a lot of things you can tune under /proc/
It would be great if I could get a faster transfer than 60 MB/sec.
Compared to the phoronix to flash drive you seem to be doing well.
No time to review tonight, but I will. I had to ewfacquire about 50 drives a couple weeks ago, and the throughput really was a bottleneck in me getting through the project. Trouble is I had to ship the data (forensic images) to my client and ntfs or fat32 are the only 2 realistic filesystems for the delivery. Thanks Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org