Dear List, A question concerning data backups and transfer speed.. I am trying to do a backup using dd if=/dev/sda of=/dev/sdb From my internal hard disk to an external usb hard disk (i.e. I am just dumping the disk image to an empty disk). I am doing this, of course, while the filesystems are unmounted. I imagine that the bottleneck is the usb (2.0) connection. What I read from wikipedia, the speed of that is.. 60 Mbytes (not bits) per second ~ 0.06 Gbytes per second So this means that a 500 Gbytes hard disk should copy in aprox. 2.3 hours .. but know it seems it takes more than 24 hours! What seems to be the problem? Am I missing something in the math? My external usb hard disk box has also an option to use an esata connector (but then I should buy an adaptor to my laptop), but the speed difference to usb 2.0 is not that big, I imagine..? Regards, Sampsa -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Sampsa Riikonen wrote:
Dear List,
A question concerning data backups and transfer speed..
I am trying to do a backup using
dd if=/dev/sda of=/dev/sdb
From my internal hard disk to an external usb hard disk (i.e. I am just dumping the disk image to an empty disk). I am doing this, of course, while the filesystems are unmounted.
I imagine that the bottleneck is the usb (2.0) connection. What I read from wikipedia, the speed of that is..
60 Mbytes (not bits) per second ~ 0.06 Gbytes per second
So this means that a 500 Gbytes hard disk should copy in aprox. 2.3 hours
.. but know it seems it takes more than 24 hours!
What seems to be the problem? Am I missing something in the math?
Blocksize? -- Per Jessen, Zürich (4.3°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday 03 February 2010 11:58:49 am Per Jessen wrote:
Sampsa Riikonen wrote:
Dear List,
A question concerning data backups and transfer speed..
I am trying to do a backup using
dd if=/dev/sda of=/dev/sdb
From my internal hard disk to an external usb hard disk (i.e. I am just dumping the disk image to an empty disk). I am doing this, of course, while the filesystems are unmounted.
I imagine that the bottleneck is the usb (2.0) connection. What I read from wikipedia, the speed of that is..
60 Mbytes (not bits) per second ~ 0.06 Gbytes per second
So this means that a 500 Gbytes hard disk should copy in aprox. 2.3 hours
.. but know it seems it takes more than 24 hours!
What seems to be the problem? Am I missing something in the math?
Blocksize?
Thanks Per, This seems to be the case.. http://en.wikipedia.org/wiki/Dd_(Unix) Do you know if I can use any multiple of 512 bytes? Not just 4*512 but even even 100 * 512 ? The bigger blocksize the faster I presume.. Regards, Sampsa -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Sampsa Riikonen wrote:
Blocksize?
Thanks Per,
This seems to be the case..
http://en.wikipedia.org/wiki/Dd_(Unix)
Do you know if I can use any multiple of 512 bytes? Not just 4*512 but even even 100 * 512 ?
The bigger blocksize the faster I presume..
Yes that's the rule - as far as I know, any multiple of 512 is fine, but the optimal blocksize will depend on the hardware. I often use 65536. /Per -- Per Jessen, Zürich (5.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Sampsa Riikonen wrote:
The bigger blocksize the faster I presume..
Yes that's the rule - as far as I know, any multiple of 512 is fine, but the optimal blocksize will depend on the hardware. I often use 65536.
Traditionally there were two limits to optimise the transfer: (1) make sure that the blocksize*block count will fit in real memory! (2) typically, use a multiple of the disk's track size (depending on the exact actuator mechanism and/or sector skew) I've no idea how those criteria translate with current disks. Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Dave Howorth wrote:
Per Jessen wrote:
Sampsa Riikonen wrote:
The bigger blocksize the faster I presume..
Yes that's the rule - as far as I know, any multiple of 512 is fine, but the optimal blocksize will depend on the hardware. I often use 65536.
Traditionally there were two limits to optimise the transfer: (1) make sure that the blocksize*block count will fit in real memory! (2) typically, use a multiple of the disk's track size (depending on the exact actuator mechanism and/or sector skew)
I've no idea how those criteria translate with current disks.
Same here. I think there are too many variables to work it out, so it's trial and error. /Per -- Per Jessen, Zürich (6.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Am 03.02.2010 10:52, schrieb Sampsa Riikonen:
[...]
I am trying to do a backup using
dd if=/dev/sda of=/dev/sdb
[...] What I read from wikipedia, the speed of that is..
60 Mbytes (not bits) per second ~ 0.06 Gbytes per second
So this means that a 500 Gbytes hard disk should copy in aprox. 2.3 hours
.. but know it seems it takes more than 24 hours!
What seems to be the problem? Am I missing something in the math?
Yes, you have to use a larger block size "dd if=/dev/sda of=/dev/sdb bs=1M" should speed up your backup.
My external usb hard disk box has also an option to use an esata connector (but then I should buy an adaptor to my laptop), but the speed difference to usb 2.0 is not that big, I imagine..?
As far as I know esata can be much faster, but to get the higher speed, you have to use larger block sizes, too. Herbert -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Feb 3, 2010 at 5:04 AM, Herbert Graeber
Am 03.02.2010 10:52, schrieb Sampsa Riikonen:
[...]
I am trying to do a backup using
dd if=/dev/sda of=/dev/sdb
[...] What I read from wikipedia, the speed of that is..
60 Mbytes (not bits) per second ~ 0.06 Gbytes per second
So this means that a 500 Gbytes hard disk should copy in aprox. 2.3 hours
.. but know it seems it takes more than 24 hours!
What seems to be the problem? Am I missing something in the math?
Yes, you have to use a larger block size "dd if=/dev/sda of=/dev/sdb bs=1M" should speed up your backup.
My external usb hard disk box has also an option to use an esata connector (but then I should buy an adaptor to my laptop), but the speed difference to usb 2.0 is not that big, I imagine..?
As far as I know esata can be much faster, but to get the higher speed, you have to use larger block sizes, too.
I do this a lot and have run benchmarks. With raw disks (/dev/sda, etc) even a 512 byte blocksize works surprisingly well / efficiently But if your destination is a file on a filesystem, then 512 bytes can be horrible. 4K is the minimum to use in this case. Over 4K actually does very little to improve speed if your destination is a hdd. If your destination is a tape drive, you want big blocks. 1 MB or bigger would be reasonable. Also instead of dd, the mbuffer program is great for pulling a lot of data into ram (think a GB or more if you have the ram) and getting it ready to send to the tape in a large continuous stream. mbuffer is in 11.2, not sure about earlier. Having a continuous stream of data can drastically increase the speed of tape. Greg -- Greg Freemyer Head of EDD Tape Extraction and Processing team Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer Preservation and Forensic processing of Exchange Repositories White Paper - http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Sampsa Riikonen schreef:
Dear List,
A question concerning data backups and transfer speed..
I am trying to do a backup using
Hi, I also have a question about backup-speed. I use tar -cf /mnt/net.tar.bz2 --bzip2 --exclude="*.iso" /data/net to backup to a Iomega Rev-drive, connected via sata. I see the light every three or four seconds. Backup of about twenty GB takes hours. At work with the same type of drive via USB I can backup about 100GB in less time, with backup-software for Windows. There the light of the drive almost never goes out when backing up. Is there a way to speed things up ? Regards, Koenraad Lelong. P.S. I hope I didn't hi-jack this thread. Is so, I'll remember not to do this in the future. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Koenraad Lelong wrote:
Sampsa Riikonen schreef:
Dear List,
A question concerning data backups and transfer speed..
I am trying to do a backup using
Hi,
I also have a question about backup-speed. I use tar -cf /mnt/net.tar.bz2 --bzip2 --exclude="*.iso" /data/net to backup to a Iomega Rev-drive, connected via sata. I see the light every three or four seconds. Backup of about twenty GB takes hours. At work with the same type of drive via USB I can backup about 100GB in less time, with backup-software for Windows. There the light of the drive almost never goes out when backing up.
Is there a way to speed things up ?
Regards,
Koenraad Lelong.
P.S. I hope I didn't hi-jack this thread. Is so, I'll remember not to do this in the future.
Did you think bzip2 compression comes for free??? Typical unnamed backup software built-in compression either uses the hardware compression built-in to the tape drive (I don't remember if rev has hardware compression or not) Or, uses compression about equivalent to gzip -7 Also, real backup programs employ double buffering and multiple concurrent processes and other tricks to keep the hardware (both the source and the destination) as busy as possible. Use backupedge, not tar. At the very least you might try using star and piping through pbzip2 instead of using tar -j star -c /data/net |pbzip2 >/mnt/net.star.bz2 Neither star nor pbzip2 are probably installed on your machine already, but they are available. star is in the standard oss repos, which means if you go into yast and search, it'll probably already be there and you can just select it to be installed. pbzip2 is in obs Archiving repo which you will have to add to software sources first. http://download.opensuse.org/repositories/Archiving/openSUSE_11.2/ If pbzip2 is still too slow, switch to gzip, then gzip -7 (less compression than default) then lzop (less compression but very fast, better compression and speed than lower values for gzip). Or if your device supports it, use the devices built-in hardware compression instead of any software compression. -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2010-02-03 at 11:19 +0100, Koenraad Lelong wrote:
I also have a question about backup-speed. I use tar -cf /mnt/net.tar.bz2 --bzip2 --exclude="*.iso" /data/net to backup to a Iomega Rev-drive, connected via sata. I see the light every three or four seconds. Backup of about twenty GB takes hours. At work with the same type of drive via USB I can backup about 100GB in less time, with backup-software for Windows. There the light of the drive almost never goes out when backing up.
Is there a way to speed things up ?
bzip2 compression is slow, it uses a lot of cpu. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAktpcMcACgkQtTMYHG2NR9X82gCeNb7LMcng+6WYvJf60n3fzKKM 0sYAn03V8eAOflDWVSr42Wi8Y5WhiX4i =IAuy -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 03/02/10 09:52, Sampsa Riikonen wrote:
What seems to be the problem? Am I missing something in the math? My external usb hard disk box has also an option to use an esata connector (but then I should buy an adaptor to my laptop), but the speed difference to usb 2.0 is not that big, I imagine..?
ESATA has a capacity of 3Gbits/sec vs USB2.0's 480Mbits/sec. In fact with ESATA the choke point is usually the read speed of the drive, not the bus speed. The disadvantage is that your computer must have a ESATA port (desktops can fit an expansion card into a PCI-Express slot, but it's difficult on laptops) and also ESATA doesn't carry power. Tom's Hardware recently had an article[1] comparing the speeds of USB2.0, Firewire800, and ESATA and found a clear difference. For instance using a 1TB Hitachi G-Drive they were getting 100MBytes/sec read speeds on ESATA vs 35MBytes/sec on USB2.0. After saying all this though, USB3.0 at 4.8Gbits/sec will be introduced over the next few years. Regards, Tejas [1] http://www.tomshardware.co.uk/usb-firewire-esata,review-31793-8.html -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Tejas Guruswamy wrote:
On 03/02/10 09:52, Sampsa Riikonen wrote:
What seems to be the problem? Am I missing something in the math? My external usb hard disk box has also an option to use an esata connector (but then I should buy an adaptor to my laptop), but the speed difference to usb 2.0 is not that big, I imagine..?
ESATA has a capacity of 3Gbits/sec vs USB2.0's 480Mbits/sec. In fact with ESATA the choke point is usually the read speed of the drive, not the bus speed. The disadvantage is that your computer must have a ESATA port (desktops can fit an expansion card into a PCI-Express slot, but it's difficult on laptops) and also ESATA doesn't carry power.
Tom's Hardware recently had an article[1] comparing the speeds of USB2.0, Firewire800, and ESATA and found a clear difference. For instance using a 1TB Hitachi G-Drive they were getting 100MBytes/sec read speeds on ESATA vs 35MBytes/sec on USB2.0.
After saying all this though, USB3.0 at 4.8Gbits/sec will be introduced over the next few years.
Regards, Tejas
[1] http://www.tomshardware.co.uk/usb-firewire-esata,review-31793-8.html
You are also forgetting that different ports consume different amounts of cpu. A 400mbit firewire port can be faster than a 480mbit usb2.0 port because the firewire uses a lot less cpu than the usb. There are also differences in motherboard design. Usb ports and (e)sata ports (nor gigabit lan, nor anything else that's built-in) aren't all the same. Some are implemented very well and some are implemented very cheaply. I would expect an esata port to be pretty efficient compared to a usb port on the same motherboard in pretty much all cases, cheap or good, but it's not an automatic given just because it says esata on the side. And the less efficient usb port on a good motherboard could be faster than the more efficient esata port on a cheap motherboard. -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Also a good idea is to do the backup with compression, then you don't have to copy all the empty blocks. A good way to do this is with the the systemrescuecd and the partimage program on this cd. http://www.sysresccd.org/Main_Page Also a good idea is to do your backup on the filesystem level, this can also be done with the systemrescucd and the program FSArchiver. FSArchiver is still under development but very good to use, more flexible at restore and handles more filesystems. And very good for online backups is the program rsync, but you have to do some study about rsync. Good luck, Hans On 03/02/10 10:52, Sampsa Riikonen wrote:
Dear List,
A question concerning data backups and transfer speed..
I am trying to do a backup using
dd if=/dev/sda of=/dev/sdb
From my internal hard disk to an external usb hard disk (i.e. I am just dumping the disk image to an empty disk). I am doing this, of course, while the filesystems are unmounted.
I imagine that the bottleneck is the usb (2.0) connection. What I read from wikipedia, the speed of that is..
60 Mbytes (not bits) per second ~ 0.06 Gbytes per second
So this means that a 500 Gbytes hard disk should copy in aprox. 2.3 hours
.. but know it seems it takes more than 24 hours!
What seems to be the problem? Am I missing something in the math? My external usb hard disk box has also an option to use an esata connector (but then I should buy an adaptor to my laptop), but the speed difference to usb 2.0 is not that big, I imagine..?
Regards,
Sampsa -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hans de Faber wrote:
Also a good idea is to do the backup with compression, then you don't have to copy all the empty blocks.
Not necessarily. When dd is running efficiently it is much faster than any file-based scheme especially one using compression. So if the disk is full of things that change a lot it can be faster. It's also simple! But usually, yes, a dedicated backup program can be faster for typical filesystems and provide extras such as multiple backup copies on a single backup drive etc. Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Feb 3, 2010 at 8:08 AM, Dave Howorth
Hans de Faber wrote:
Also a good idea is to do the backup with compression, then you don't have to copy all the empty blocks.
Not necessarily. When dd is running efficiently it is much faster than any file-based scheme especially one using compression. So if the disk is full of things that change a lot it can be faster. It's also simple!
But usually, yes, a dedicated backup program can be faster for typical filesystems and provide extras such as multiple backup copies on a single backup drive etc.
Cheers, Dave
I agree, but it really depends on where the bottleneck is. If you can read data much faster than you can write it, then compression can accelerate the process by allowing our write process to go faster. If read and write are the same speed and they don't interfere with each other, then compression can not accelerate the process. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-02-03 14:08, Dave Howorth wrote:
Hans de Faber wrote:
Also a good idea is to do the backup with compression, then you don't have to copy all the empty blocks.
Not necessarily. When dd is running efficiently it is much faster than any file-based scheme especially one using compression. So if the disk is full of things that change a lot it can be faster. It's also simple!
Depends on the cpu speed. If the cpu is able to compress faster than the destination media subsystem can write, then it is faster to backup with compresssion. You can sometimes test with compression ratio.
But usually, yes, a dedicated backup program can be faster for typical filesystems and provide extras such as multiple backup copies on a single backup drive etc.
And comercial backup programs may have such interesting things as forward recovery data, so that if the backup is damaged it can be still retrieved. - -- Cheers / Saludos, Carlos E. R. (from 11.2 "Emerald" GM (bombadillo)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAktpg7EACgkQU92UU+smfQXD5QCfd0mM5NJerII4jIw2J5MpBgjR UokAnieNEgOMwwiIQ+QjmGJ7zYww1GgK =cUkH -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2010-02-03 at 11:52 +0200, Sampsa Riikonen wrote:
Dear List,
A question concerning data backups and transfer speed..
I am trying to do a backup using
dd if=/dev/sda of=/dev/sdb
From my internal hard disk to an external usb hard disk (i.e. I am just dumping the disk image to an empty disk). I am doing this, of course, while the filesystems are unmounted.
I imagine that the bottleneck is the usb (2.0) connection. What I read from wikipedia, the speed of that is..
60 Mbytes (not bits) per second ~ 0.06 Gbytes per second
Yes. That's the maximum, the maintained speed is quite lower.
So this means that a 500 Gbytes hard disk should copy in aprox. 2.3 hours
Yes... :-? I have copied about 50..80 GiB over usb at a time, and it is very slow. I don't remember exactly how slow, though. I should have written it somewhere.
.. but know it seems it takes more than 24 hours!
Could be.
What seems to be the problem? Am I missing something in the math?
Is it a real hard disk, not flash? Something else connected to the usb bus, perhaps?
My external usb hard disk box has also an option to use an esata connector (but then I should buy an adaptor to my laptop), but the speed difference to usb 2.0 is not that big, I imagine..?
It is a "huge" difference, believe me. I have some usb/esata HD boxes, and the speed is so much greater. It goes as fast as an internal HD. There is another advantage: you can run smart tests and learn of the health of the HD. One of my disks was dying, at a young age, and I had no idea. I could have lost the entire backup. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAktpb+kACgkQtTMYHG2NR9VXMACghqqwjwUeN6yMzeZNpO44BIPR DDoAnR+cd5iys7Hpv73hw4N74zMr+kaJ =Cvmt -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Feb 3, 2010 at 4:52 AM, Sampsa Riikonen
Dear List,
A question concerning data backups and transfer speed..
I am trying to do a backup using
dd if=/dev/sda of=/dev/sdb
From my internal hard disk to an external usb hard disk (i.e. I am just dumping the disk image to an empty disk). I am doing this, of course, while the filesystems are unmounted.
I imagine that the bottleneck is the usb (2.0) connection. What I read from wikipedia, the speed of that is..
60 Mbytes (not bits) per second ~ 0.06 Gbytes per second
So this means that a 500 Gbytes hard disk should copy in aprox. 2.3 hours
.. but know it seems it takes more than 24 hours!
I use usb-2 a fair amount. You should expect 1GB/min or 60GB/hr in the real world. So 8 or 9 hrs would be more typical. No idea why your running slow. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (10)
-
Brian K. White -
Carlos E. R. -
Dave Howorth -
Greg Freemyer -
Hans de Faber -
Herbert Graeber -
Koenraad Lelong -
Per Jessen -
Sampsa Riikonen -
Tejas Guruswamy