On Wed, Feb 3, 2010 at 1:41 PM, Sampsa Riikonen <sampsa.riikonen@iki.fi> wrote: <snip>

- Per is using blocksize of 65536 bytes ~ 65 Mb

Thats 64KB not MB. And your command "dd if=/dev/sda of=/dev/sdb bs=X" should be dd if=/dev/sda of=/dev/sdb bs=X conv=noerror,sync noerror says to continue the copy even if there is a media error on the /dev/sda. sync says when that media error occurs replace the failed data with zeros (nulls). Without sync it just copies truncated data. Really bad because you won't be able to restore the now corrupted data at all. X is obviously important for performance, but it you go with 1MB and there is a read error you can end up with a full MB of missing data even though only one sector was actually bad. I have done a lot of benchmarking of this process. As a company that has 5 computers primarily dedicated to running the above command to duplicate our clients data for preservation / analysis we have chosen eSata with 4K blocks as our optimum solution. We get good throughput with minimal risk of data loss due to a bad sector. With fast eSata drives I've seen dd speeds approaching 4GB/min or 240GB/hr. But that is pretty rare and takes good fast drives. A lot of drives just can't deliver data that fast. FYI: We actually do something closer to dd if=/dev/sda of=/evidence_drive/computer_name.img bs=4K conv=noerror, sync The resulting file is called a dd forensic image and is the basis of many civil and criminal court actions. If you master typing in the above you have mastered the first step of being a computer forensic examiner. (There are lots of ways to create a forensic image, but the above really is how we do it.) disclaimer: The State of Georgia, USA says that if you use the above command to collect potential evidence for a paying client to use in a court case you have to have a PI license. Same for South Carolina. Why that makes sense I have no idea. Bet you never saw that disclaimer before!! Greg PI license # available upon request :) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-02-03 19:41, Sampsa Riikonen wrote:

Dear All,

Thank you for your replies to my question. Here is a summary:

The question was, how to copy an entire hard disk to another hard disk in a usb box most conveniently.. the command is:

dd if=/dev/sda of=/dev/sdb bs=X

(X=blocksize).

...

(.. some contradictions here with the optimal blocksize).

:-) Probably the best thing is to test it. Ie, copy, say, 250 MiB, and time it, using several buffer sizes. The top size would be, I think (very wild guess) the size of the ram cache inside the hard disk; but via USB that would not be an issue (you can not feed the disk fast enough to fill that cache). Perhaps a small buffer is slowing down the read operation :-?

- Carlos said implicitly that USB might be the bottleneck.. and yes, it is a real hard disk and I have a mouse connected to my computer via usb as well. According to Carlos'es experience, eSATA is much more "rapido".

I copied 291GiB some hours ago, from one eSATA to an internal SATA, with rsync. It ran at about 60 MB/s, in the gkrellm display. Ah, I still have the xterm output: sent 311.99G bytes received 25.98K bytes 54.74M bytes/sec total size is 311.95G speedup is 1.00 real 94m59.787s <== output of command "time". user 37m56.172s sys 27m50.542s The checksum test runs faster - it is only "read" plus cpu, I suppose: bombadillo:/data/storage_a # time md5sum --check --quiet ~/checksum real 52m29.083s user 15m9.319s sys 4m0.977s And, according to hdparm, the disk can do: # hdparm -tT /dev/sdd1 ; hdparm -tT /dev/sda12 /dev/sdd1: <== external disk Timing cached reads: 11832 MB in 2.00 seconds = 5923.92 MB/sec Timing buffered disk reads: 204 MB in 3.00 seconds = 67.97 MB/sec /dev/sda12: <== internal disk Timing cached reads: 12366 MB in 2.00 seconds = 6192.10 MB/sec Timing buffered disk reads: 350 MB in 3.01 seconds = 116.40 MB/sec The sdd1 disk is the external one, the older unit, and sda12 the internal, destination unit. Mmm.... curious... I'll measure another one, same kind as the internal (both bought a month or two ago). # hdparm -tT /dev/sde7 /dev/sde7: Timing cached reads: 12162 MB in 2.00 seconds = 6089.21 MB/sec Timing buffered disk reads: 312 MB in 3.00 seconds = 103.98 MB/sec Yes, indeed, I have one slow disk. I'll have to check the specs some day, and learn if it is the disk or the interface (it is on a different port group on the motherboard). In any case, rsync was reading at about the maximun speed the disk and eSATA port is capable of.

.. btw, I have tried to copy my 500Gb hard disk only during ~ 12 hrs (error in the original e-mail) with the default 512 byte blocksize, i.e. during the night. In the morning it was still copying.. :( Well.. I guess I'll try tonight again with the 65 Mb blocksize and if that is unacceptable (more than one night) I'll be off to by an eSATA thingie for my laptop.

I bought a new desktop, and I was, and am, trhilled by the eSATA ports. I dunno how expensive they will be for a laptop, though, but I think it is worth it if you do large backups. Plus, don't forget SMART.

I am very familiar with rsync and I am using it all the time, but I just want an identical copy of my hard drive every now and then.. just in case.

Yes, it is another method. - -- Cheers / Saludos, Carlos E. R. (from 11.2 "Emerald" GM (bombadillo)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAktp6AwACgkQU92UU+smfQX8TwCeINN05Ga+3gbyXJghbgyY2qEU hUwAn2usuthLaPReIXq4N3QD5afxng2G =iKqG -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On 2010/02/03 20:41 (GMT+0200) Sampsa Riikonen composed:

- There is a general consensus that eSATA should be more efficient.

The "e" in eSATA means external. eSATA is nothing but SATA plus two bonuses: 1-HD cabling designed for the repeated plug/unplug cycles expected of external devices 2-hardware support necessary for hot-plugging to be possible IOW, speed from eSATA, like that from SATA, is all in the particular implementation from the driver, firmware, chip and bus combination. Any speed differential between SATA and eSATA on any given system is more likely because the eSATA controller and SATA controller are (commonly) separate chips from different manufacturers than anything else, assuming identical HDs on both. -- "Our Constitution was made only for a moral and religious people. It is wholly inadequate to the government of any other." John Adams, 2nd US President Team OS/2 ** Reg. Linux User #211409 Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org