[opensuse] ddrescue bottleneck on faulty drive
Hi ListMates, I'm not sure if this is the correct forum but hopefully someone will have an answer for me. I have a faulty 500Gb notebook disk that I want to clone and recover. I use ddrescue (with the -n option) to clone the faulty hdd to a new hdd. The problem is that based on the current stats, it will take days/weeks to complete the clone!!! I know that I have a faulty drive, I just want to clone the faulty drive unto a new drive, simply skip the faulty sectors (zero fill on the destination drive) and continue the clone - I don't want the system to keep retrying over and over to read the faulty sector - try once and die and set the destination sector zero-filled. Where there are no errors on the original drive the copy goes fast, but once it hits a bad sector it is painfully slow!!!! So where is the bottle neck? 1) ddrescue - according to the docs if one uses the -n option it wont bother to read or recover the faulty sectors on the faulty drive (I also tried -r1 - read/retry once) but to no avail. BTW I tried both read from beginning of disk as well as reading from back (-R), but alas to no avail as once it gets to the faulty sectors it becomes painfully slow. 2) is it a kernel parameter - ie how to turn of retrying to read the faulty sector n..times 3) is it a sata driver parameter - ie likewise how to turn of retrying to read the faulty sector n..times 4) or is it the firmware on the faulty hdd? (in that case I'm not sure there is a solution to turn of retrying faulty sectors or remapping them) I'm not sure there is a solution but if anyone knows any help would be most appreciated. Thanks in advance for any help. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Otto Rodusek wrote:
Hi ListMates,
I'm not sure if this is the correct forum but hopefully someone will have an answer for me.
I have a faulty 500Gb notebook disk that I want to clone and recover. I use ddrescue (with the -n option) to clone the faulty hdd to a new hdd. The problem is that based on the current stats, it will take days/weeks to complete the clone!!!
It's a fualty drive. What do you expect?
I know that I have a faulty drive, I just want to clone the faulty drive unto a new drive, simply skip the faulty sectors (zero fill on the destination drive) and continue the clone
Then your copied data will be unreliable., so what's the point of using ddrescue? Do you realize that you are risking having severe filesystem corruption with this scheme? At the very minimum, you will have to run an EXTREMELY thorough fsck on the new disk.
- I don't want the system to keep retrying over and over to read the faulty sector - try once and die and set the destination sector zero-filled. Where there are no errors on the original drive the copy goes fast, but once it hits a bad sector it is painfully slow!!!!
So where is the bottle neck?
apparently the -n flag is not working properly. try dd_rescue instead.
1) ddrescue - according to the docs if one uses the -n option it wont bother to read or recover the faulty sectors on the faulty drive (I also tried -r1 - read/retry once) but to no avail. BTW I tried both read from beginning of disk as well as reading from back (-R), but alas to no avail as once it gets to the faulty sectors it becomes painfully slow. 2) is it a kernel parameter - ie how to turn of retrying to read the faulty sector n..times 3) is it a sata driver parameter - ie likewise how to turn of retrying to read the faulty sector n..times 4) or is it the firmware on the faulty hdd? (in that case I'm not sure there is a solution to turn of retrying faulty sectors or remapping them)
I'm not sure there is a solution but if anyone knows any help would be most appreciated.
Thanks in advance for any help.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/11/2014 07:56 PM, Otto Rodusek wrote:
faulty 500Gb notebook disk that I want to clone and recover
- maybe try "rsync" to new disk ? or : run command : tar clf - . | ( umask 0; cd /mnt; tar xvf - ) { starting at / root . . . to new disk on /mnt : then afterwards try to clean up new disk with fsck } ................ regards -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/11/2014 07:56 PM, Otto Rodusek wrote:
faulty 500Gb notebook disk that I want to clone and recover
- maybe try "rsync" to new disk ? : perhaps something like : rsync -av --delete-after --exclude='*/Cache/*' --exclude=/media --exclude=/mnt --exclude=/proc --exclude=/run --exclude=/sys --exclude=/tmp --exclude=/var/run / /mnt [ whereas start at root / and copy to /mnt as mount-point for new disk : some kindly knowledgeable list-member can tell you which excludes you should and should not have for rsync command ] ................ regards -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/11/2014 12:36 PM, ellanios82 wrote:
On 10/11/2014 07:56 PM, Otto Rodusek wrote:
faulty 500Gb notebook disk that I want to clone and recover
- maybe try "rsync" to new disk ?
: perhaps something like :
rsync -av --delete-after --exclude='*/Cache/*' --exclude=/media --exclude=/mnt --exclude=/proc --exclude=/run --exclude=/sys --exclude=/tmp --exclude=/var/run / /mnt
[ whereas start at root / and copy to /mnt as mount-point for new disk : some kindly knowledgeable list-member can tell you which excludes you should and should not have for rsync command ]
................
regards
I did this sort of recovery a while ago, using Clonezilla. As far as I could tell, I recovered everything. see: http://lists.opensuse.org/opensuse/2014-09/msg00172.html -- _____________________________________ ---This space for rent--- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On October 11, 2014 12:56:53 PM EDT, Otto Rodusek
Hi ListMates,
I'm not sure if this is the correct forum but hopefully someone will have an answer for me.
I have a faulty 500Gb notebook disk that I want to clone and recover. I
use ddrescue (with the -n option) to clone the faulty hdd to a new hdd.
The problem is that based on the current stats, it will take days/weeks
to complete the clone!!!
I know that I have a faulty drive, I just want to clone the faulty drive unto a new drive, simply skip the faulty sectors (zero fill on the destination drive) and continue the clone - I don't want the system to keep retrying over and over to read the faulty sector - try once and die and set the destination sector zero-filled. Where there are no errors on the original drive the copy goes fast, but once it hits a bad sector
it is painfully slow!!!!
So where is the bottle neck?
1) ddrescue - according to the docs if one uses the -n option it wont bother to read or recover the faulty sectors on the faulty drive (I also tried -r1 - read/retry once) but to no avail. BTW I tried both read from beginning of disk as well as reading from back (-R), but alas to no avail as once it gets to the faulty sectors it becomes painfully slow. 2) is it a kernel parameter - ie how to turn of retrying to read the faulty sector n..times 3) is it a sata driver parameter - ie likewise how to turn of retrying to read the faulty sector n..times 4) or is it the firmware on the faulty hdd? (in that case I'm not sure there is a solution to turn of retrying faulty sectors or remapping them)
I'm not sure there is a solution but if anyone knows any help would be most appreciated.
Thanks in advance for any help.
Are you sure you are using ddrescue from the gnu_ddrescue package. It should not work as you describe. Hopefully you ran it with the log option. If so you can control-c it anytime and restart it. Every time you want it use the same log file. Ddrescue will read the log file and know what work it has already done. On the first pass use a big skipsize (--skipsize=10G) with -n and -N. Thus "ddrescue --skipsize=10G -n -N /dev/sdb /dev/sdc logfile" should make a single pass through the disk and skip ahead 10 gigs when it hits an error. Logfile will have the info the next invocation needs to not copy the good parts a second time. The second time try the same call but from the opposite end and with 2048 sectors per read. (Eg. Add -c 2048 -R) That too should run pretty quick. For a third pass I would go back to forward, but maybe a 100 MB skipsize, then repeat in reverse. Each pass should get faster and faster you because there is less and les data not yet copied. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/10/14 6:15, Greg Freemyer wrote:
On October 11, 2014 12:56:53 PM EDT, Otto Rodusek
wrote: Hi ListMates,
I'm not sure if this is the correct forum but hopefully someone will have an answer for me.
I have a faulty 500Gb notebook disk that I want to clone and recover. I
use ddrescue (with the -n option) to clone the faulty hdd to a new hdd.
The problem is that based on the current stats, it will take days/weeks
to complete the clone!!!
I know that I have a faulty drive, I just want to clone the faulty drive unto a new drive, simply skip the faulty sectors (zero fill on the destination drive) and continue the clone - I don't want the system to keep retrying over and over to read the faulty sector - try once and die and set the destination sector zero-filled. Where there are no errors on the original drive the copy goes fast, but once it hits a bad sector
it is painfully slow!!!!
So where is the bottle neck?
1) ddrescue - according to the docs if one uses the -n option it wont bother to read or recover the faulty sectors on the faulty drive (I also tried -r1 - read/retry once) but to no avail. BTW I tried both read from beginning of disk as well as reading from back (-R), but alas to no avail as once it gets to the faulty sectors it becomes painfully slow. 2) is it a kernel parameter - ie how to turn of retrying to read the faulty sector n..times 3) is it a sata driver parameter - ie likewise how to turn of retrying to read the faulty sector n..times 4) or is it the firmware on the faulty hdd? (in that case I'm not sure there is a solution to turn of retrying faulty sectors or remapping them)
I'm not sure there is a solution but if anyone knows any help would be most appreciated.
Thanks in advance for any help. Are you sure you are using ddrescue from the gnu_ddrescue package. It should not work as you describe.
Hopefully you ran it with the log option. If so you can control-c it anytime and restart it. Every time you want it use the same log file. Ddrescue will read the log file and know what work it has already done.
On the first pass use a big skipsize (--skipsize=10G) with -n and -N.
Thus "ddrescue --skipsize=10G -n -N /dev/sdb /dev/sdc logfile" should make a single pass through the disk and skip ahead 10 gigs when it hits an error. Logfile will have the info the next invocation needs to not copy the good parts a second time.
The second time try the same call but from the opposite end and with 2048 sectors per read. (Eg. Add -c 2048 -R)
That too should run pretty quick.
For a third pass I would go back to forward, but maybe a 100 MB skipsize, then repeat in reverse.
Each pass should get faster and faster you because there is less and les data not yet copied.
Greg
Hi Greg, Thanks for the feedback. Yes I am using the correct gnu ddrescue, and yes I always use the logfile . Thanks for the insight into (--skip-size and -N) this did the trick. I do basically most of what you wrote (except the --skip-size). The disk is now recovered 95% (all the important stuff anyway), the rest I don't really care. Thanks for your help and best regards. Otto. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-10-11 18:56, Otto Rodusek wrote:
Hi ListMates,
I'm not sure if this is the correct forum but hopefully someone will have an answer for me.
I have a faulty 500Gb notebook disk that I want to clone and recover. I use ddrescue (with the -n option) to clone the faulty hdd to a new hdd. The problem is that based on the current stats, it will take days/weeks to complete the clone!!!
Yes, when dd hits a faulty sector it tries many times (perhaps 20, I don't remember), before going on to the next. I'm more familiar with dd_rescue and dd_rhelp. You call dd_rhelp, and this one calls dd_rescue, automatically telling it how to skip the problematic sectors first, starting with the good ones, then continuing with the bad ones, till it finishes or you stop it. You can let it run for some hours, up to days, till it gets everything or a what you consider a fair percent. GNU ddrescue is supposed to include the functionality of both in a single program. I have used it, but I'm not familiar with it. I think I prefer dd_rhelp. You can try it :-) -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On 12/10/14 10:26, Carlos E. R. wrote:
On 2014-10-11 18:56, Otto Rodusek wrote:
Hi ListMates,
I'm not sure if this is the correct forum but hopefully someone will have an answer for me.
I have a faulty 500Gb notebook disk that I want to clone and recover. I use ddrescue (with the -n option) to clone the faulty hdd to a new hdd. The problem is that based on the current stats, it will take days/weeks to complete the clone!!! Yes, when dd hits a faulty sector it tries many times (perhaps 20, I don't remember), before going on to the next.
I'm more familiar with dd_rescue and dd_rhelp. You call dd_rhelp, and this one calls dd_rescue, automatically telling it how to skip the problematic sectors first, starting with the good ones, then continuing with the bad ones, till it finishes or you stop it. You can let it run for some hours, up to days, till it gets everything or a what you consider a fair percent.
GNU ddrescue is supposed to include the functionality of both in a single program. I have used it, but I'm not familiar with it. I think I prefer dd_rhelp. You can try it :-)
Hi Carlos, Thanks for your feedback. Yes I've used both dd_rescue and ddrescue (different programs) and I have found that ddrescue is the better product. Greg pointed me in the correct direction with the parameters (--skip-size and -N) which more or less resolved the issues. I would still be curios as to where/what causes the bottleneck - yes I do know that when an error sector is found the system tries (many times) to recover the sector with multiple reads - this is the part that I was hoping to resolve - whether there is a parameter or command to ignore the error sector and don't waste time on recovery when I know it's bad/dead. If it's a function of the drive firmware then there's no hope I think, but if it's the linux kernel or a linux driver that is causing the re-reads than maybe there is a parameter to simply not retry?? For example if I ( cat anyfile-with-a-bad-sector(s) ) it could potentially take a LONG time to display. I know it's not the "cat" command that is doing the retries on bad sectors. I was hoping that it was a kernel parameter that can be tweaked to make this re-read ignored. Anyways thanks for your feedback. Best regards. Otto. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-10-12 14:02, Otto Rodusek wrote:
On 12/10/14 10:26, Carlos E. R. wrote:
Hi Carlos,
Thanks for your feedback. Yes I've used both dd_rescue and ddrescue (different programs) and I have found that ddrescue is the better product.
If you compare dd_rescue and ddrescue, yes. But the trick is to use tha wrapper script dd_rhelp instead. It does those adjustments automatically, on the fly. I think it calls and perhaps kills dd_rescue as needed, many times.
Greg pointed me in the correct direction with the parameters (--skip-size and -N) which more or less resolved the issues.
I would still be curios as to where/what causes the bottleneck - yes I do know that when an error sector is found the system tries (many times) to recover the sector with multiple reads - this is the part that I was hoping to resolve -
And it can be many sectors, nor only one. Also, with many faulty sectors I understand the head has problems to locate the tracks. It needs reading tracks to know where it is. And each failed reading causes a delay: if I recall correctly, the head is reinitialized on each failed attempt.
whether there is a parameter or command to ignore the error sector and don't waste time on recovery when I know it's bad/dead. If it's a function of the drive firmware then there's no hope I think, but if it's the linux kernel or a linux driver that is causing the re-reads than maybe there is a parameter to simply not retry??
It is both. Both have a retry count. You control only the kernel side.
For example if I ( cat anyfile-with-a-bad-sector(s) ) it could potentially take a LONG time to display. I know it's not the "cat" command that is doing the retries on bad sectors. I was hoping that it was a kernel parameter that can be tweaked to make this re-read ignored.
Yes, these rescue tools adjust that count. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On October 12, 2014 8:02:37 AM EDT, Otto Rodusek
Greg pointed me in the correct direction with the parameters (--skip-size and -N) which more or less resolved the issues.
I would still be curios as to where/what causes the bottleneck - yes I do know that when an error sector is found the system tries (many times) to recover the sector with multiple reads - this is the part that I was
hoping to resolve - whether there is a parameter or command to ignore the error sector and don't waste time on recovery when I know it's bad/dead. If it's a function of the drive firmware then there's no hope
If a large skipsize helped then you likely had a true head crash. A track is roughly 1 MB on a modern drive. Let's say you had a head crash in the middle of writing some data, that could take out numerous physically adjacent tracks on one platter surface. (Ie. The head is wider than one track, so if it hits the surface it takes out multiple tracks in a single disk revolution.) But the tracks from different surfaces are interlaced so is logically more spread out. Thus if you have damage to 5 physically adjacent tracks(5 MB lost), the logical spread of that damage is likely 4x that or 20 MB. I've seen damaged areas on a platter clearly visible to the human eye (assuming you disassemble the drive after boudoir recovery effort). Ddrescue with default params would hit that 20 MB spot and only bump ahead 64 k after each error as I recall. If so it would take 300 or so bumps to get past the damage. If each bump takes a minute because of retries, that's 5 hours delay because of that one section of bad media. A large skipsize let's you quickly jump past areas where a disk head crash happened. If Turkey curious, once you are done doing the recoveries up the drive and inspect the platters for physical damage. If you can see it, then it was bad. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 12/10/2014 16:34, Greg Freemyer a écrit :
the logical spread of that damage is likely 4x that or 20 MB. I've seen damaged areas on a platter clearly visible to the human eye (assuming you disassemble the drive after boudoir recovery effort).
look at this one (glass platters) http://dodin.info/piwigo/picture.php?/93010-dsc00009_14781/search/131 :-) clearly visible :-) jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On October 12, 2014 11:01:17 AM EDT, jdd
Le 12/10/2014 16:34, Greg Freemyer a écrit :
the logical spread of that damage is likely 4x that or 20 MB. I've seen damaged areas on a platter clearly visible to the human eye (assuming you disassemble the drive after boudoir recovery effort).
look at this one (glass platters)
http://dodin.info/piwigo/picture.php?/93010-dsc00009_14781/search/131
:-)
Nice, that must have been hundreds of tracks! Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (7)
-
Banga Gong
-
Carlos E. R.
-
ellanios82
-
Greg Freemyer
-
jdd
-
John Andersen
-
Otto Rodusek