On 2014-02-12 01:01 (GMT+0100) Carlos E. R. composed:

Felix Miata wrote:

...

I seem to have a failing STB HD, a "refurb" Samsung HD155UI relabeled as Seagate ST1500DL004 Barracuda Green, bought 11 months ago. Log from latest rsync retry:

You can query the on disk SMART data with "smartctl -a /dev/devname". Then you could run the short test (not the long one) and again read the data.

That would say more about the health of the disk.

...

Anyone here familiar with ddrescue know what I should expect of it? Suggestions?

Instead of ddrescue, I find better to run "dd_rhelp" (it is a script). Read "/usr/share/doc/packages/dd_rhelp/README" for details. Then the "example.txt" file.

The syntax is very simple:

minas-tirith:~ # dd_rhelp Need 2 arguments... usage: dd_rhelp {filename|device} {output-file} [{info}] or dd_rhelp --help or dd_rhelp --version minas-tirith:~ #

The advantage on ddrescue is that when there is an unreadable section on the disk, ddrescue will take ages, and maybe fail to clone god areas that are behind that bad area. dd_rhelp will copy everything that is readable, and then slowly read the rest, for as long as you leave it running. And the target image will have the correct size and location, no gaps. It is also abortable, it will restart skipping already copied areas.

Alternatively, you could use gnu ddrescue, which integrates both programs.

To be clear, you mean that gnu ddrescue is an integration of the dd_rhelp script and smartctl? Or, are there more than one ddrescue apps?

The syntax is quite different, so read the manual first (I don't have it on memory).

I looked at http://www.gnu.org/software/ddrescue/ before writing OP. I just don't want to take an unfortunate next step and lose a rescue possibility I might have had otherwise. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 02/12/2014 02:32 AM, Carlos E. R. wrote:

You can use any of those two, either dd_rhelp or ddrescue.

Just a side note: while such specialized tools have further functionality like adapting the read block size, merging two such rescue images, etc., it is still basically possible with plain dd: you have to use some options to avoid short reads (iflag=fullblock), to add padding of unreadable parts (conv=sync), and to continue on read errors (conv=noerror): dd \ if=/dev/sda1 \ of=sda1.img \ iflag=fullblock \ conv=noerror,sync We'll add such an example to the upstream coreutils Texinfo manual - as well as a link to http://www.gnu.org/software/ddrescue/ Discussion there: http://lists.gnu.org/archive/html/coreutils/2014-02/msg00011.html Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Hi Greg, On 02/13/2014 04:26 PM, Greg Freemyer wrote:

On Wed, Feb 12, 2014 at 8:03 AM, Bernhard Voelker wrote: I don't know:

...

iflag=fullblock

What does that really do? I've never seen it recommended before for handling a drive with bad sectors.

Well spotted, we're discussing the same question upstreams. ;-) I'm not sure it is required for block devices, but however ... depending on the ibs size I could imagine that short reads cannot happen for block devices, too. AFAIK fullblock cannot do any harm.

I checked the man page, but it is uninformative.

The man page of coreutils is effectively the same as "dd --help". Use "info coreutils 'dd invocation'" or shorter "info dd" instead. `fullblock' Accumulate full blocks from input. The `read' system call may return early if a full block is not available. When that happens, continue calling `read' to fill the remainder of the block. This flag can be used only with `iflag'. This flag is useful with pipes for example as they may return short reads. In that case, this flag is needed to ensure that a `count=' argument is interpreted as a block count rather than a count of read operations. The (probably ugly) point is that POSIX requires short reads per default, so GNU dd(1) provides fullblock as an allowed extension to fit better into daily usage.

fyi: The above defaults to 512 byte blocks. That can be extremely inefficient with linux. The trouble is if you bump the blocksize up, dd will only partially fill the block with valid data, then 00 fill the rest of the block after the error.

Yes, true. That's why other tools with adaptive ibs size settings usually fit better ... well, that's what they are designed for. ;-) However, given the failing blocks on the disk are located on the disk where no or only unimportant data is stored, I would think that dd(1) could do a pretty good job. Furthermore, you don't have a learning curve if you are familiar with dd(1) ... and finally, it's available everywhere. My mail now also serves as a heads-up again to make 'fullblock' more known to the public ... and to avoid short read issues in other (pipe) cases. ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 02/13/2014 06:05 PM, Greg Freemyer wrote:

Since you seem to know the details, can walk me through a simple example.

As I already said, I'm not yet sure about the correlation between the two cases where a) the unreadable part of a bad read (on failing disks) is padded out with NULs with "conv=noerror,sync", versus b) the regular "short read" case which mainly happens on pipes/fifos where the provider does not have enough data ready to be read by dd(1).

Assume we have a block device with only 8 sectors and the 4th sector is bad:

GGGBGGGG - G=Good - B=Bad

"dd bs=4K conv=noerror,sync" will produce:

GGG00000 - 0= a sector full of nulls.

It does that because it only invokes read once, and when if gets a short read, it null fills the rest of the block. The end result is 4 sectors of valid data is not copied, even though it is readable.

As said above, I can't tell for bad reads (yet), but for "short" reads, e.g. in a fifo case, the result is completely different ... probably as a big surprise to everyone who does not know the implication of a short read playing together with conv=sync and *bs= sizes. I reduced your example to bs=4 for illustration purposes. $ mkfifo fifo # Start dd to read from the fifo with bs=ibs=obs=4. $ dd bs=4 conv=noerror,sync < fifo > out 2> err & # Now write 3 bytes, then sleep and write the other 5 bytes. $ { printf GGG ; sleep 10 ; printf GGGGG; } > fifo # Look at the result: oops 12 bytes! $ ls -ldog out -rw-r--r-- 1 12 Feb 14 01:21 out Oops, 12 bytes? How that? # Look at the stderr diagnostic: $ cat err 1+2 records in 3+0 records out 12 bytes (12 B) copied, 10.0007 s, 0.0 kB/s Ah, there were 3 read() invocations where 2 were "short" reads (1+2 records in), and 3 records out. # Now looking at the output file with od(1): $ od -tx1 out 0000000 47 47 47 00 47 47 47 47 47 00 00 00 0000014 The first read() has returned 3 'G's. and since the obs size is the same as ibs, it has written 4 bytes (G G G NUL). The second read() was successful, and the 4x 'G's are written to the output file as expected. The third read() is a short read and returns only the 8th 'G'; as obs=ibs=4, four bytes are written padded out with 3 NULs. I'd guess 90% of the users would not have expected this, right? The reason for this is the conv=sync padding. Without it, the result looks as expected: $ rm fifo out err $ mkfifo fifo $ dd bs=4 conv=noerror < fifo > out 2> err & $ { printf GGG ; sleep 10 ; printf GGGGG; } > fifo $ cat err $ 1+2 records in $ 1+2 records out 8 bytes (8 B) copied, 10.0018 s, 0.0 kB/s $ od -tx1 out 0000000 47 47 47 47 47 47 47 47 0000010 Although there were 2 short reads, the output has been copied as expected. Looking at ith with "strace -e read,write dd ...", you can see it: ,,, read(0, "GGG", 4) = 3 write(1, "GGG", 3) = 3 read(0, "GGGG", 4) = 4 write(1, "GGGG", 4) = 4 read(0, "G", 4) = 1 write(1, "G", 1) = 1 read(0, "", 4) = 0 ... Okay, this is all the pipe/fifo case. For a failing disk, the user would want to have the same size of the output file as the input block device file. Therefore, he would add the conv=sync option again plus the iflag=fullblock option. I think it's most obvious with the strace output how fullblock works: $ rm fifo out err $ mkfifo fifo $ strace -e read,write -o log dd bs=4 conv=noerror,sync iflag=fullblock < fifo > out 2> err & $ { printf GGG ; sleep 10 ; printf GGGGG; } > fifo $ cat err 2+0 records in 2+0 records out 8 bytes (8 B) copied, 9.99723 s, 0.0 kB/s $ od -tx1 out 0000000 47 47 47 47 47 47 47 47 0000010 $ cat log ... read(0, "GGG", 4) = 3 read(0, "G", 1) = 1 write(1, "GGGG", 4) = 4 read(0, "GGGG", 4) = 4 write(1, "GGGG", 4) = 4 read(0, "", 4) = 0 ... With iflag=fullblock, dd reads until the bs=4 buffer is completely filled before writing it to stdout. As said several times, this is the behavior for "short" reads. I have to dive into the sources again to see how "conv=noerror,sync" interferes with "iflag=fullblock" for *bad* reads. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Felix Miata wrote:

To be clear, you mean that gnu ddrescue is an integration of the dd_rhelp script and smartctl? Or, are there more than one ddrescue apps?

I think there are 3: Ddrescue, Dd_rescue, gnu_ddrescue are 3 different packages iirc. The names of the executables overlap iirc, so it is easy to get confused so I only install one at a time. Gnu_ddrescue has the functionality of Dd_rhelp integrated in. If you don't know any of them, you might as well use the gnu version since it is more sophisticated. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2014-02-12 09:54, Felix Miata wrote:

On 2014-02-12 01:07 (GMT-0500) Greg Freemyer composed:

man ddrescue produced man page for 1.17, so that's what I tried first after not actually understanding dd_rhelp's description of the last input option about info. The 1.17 page had so many options for telling it how to do its job I missed -v. :-( Of the three source files, two result files played without apparent flaws. One "played" nothing but silent black for several minutes before I gave up trying. Mc F3 viewing the black file I see from front well into the file nothing but nulls, binary stuff in its tail.

I told you to simply use dd_rhelp and it's very simple syntax: usage: dd_rhelp {filename|device} {output-file} [{info}] or dd_rhelp --help or dd_rhelp --version Just that. it is a wrapper script around dd_rescue, and it takes care of its complex options. You do not have to care about them nor study manuals. dd_rhelp source destination

Logs from the first three done with 1.17: OK http://fm.no-ip.com/Tmp/Linux/bbt0512-201312181730.log bad http://fm.no-ip.com/Tmp/Linux/closer0710-201312190130R1.log OK http://fm.no-ip.com/Tmp/Linux/closer0720-201304040230.log

Looking at those logs it wasn't clear to me what they were saying, whether those numbers represent bits, bytes or sectors, and whether relative to file start and end, or filesystem addressing, until finding the man page discussion of them.

It doesn't matter. dd_rhelp knows.

I retried the one that failed with dd_rhelp, but it produced only 15%-20% of the total before it quit trying. What the log doesn't indicate is it actually ended with a segfault: http://fm.no-ip.com/Tmp/Linux/closer0710GDMX09-201312190130.ts.log

bad luck.

I retried same one again with 1.17: http://fm.no-ip.com/Tmp/Linux/closer0710-201312190130R2.log

You mean you tried with GNU ddrescue. And it worked. So, use that one instead of dd_rhelp. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlL7XOwACgkQja8UbcUWM1zwfwD9H1AMk0WMx3XZefNz6coh4BxR +oj2RZsqkFwI9XEqEVgA/0pjKUrAtc1V4i+A+0NbXA88Pu4Y4w0xm6yZNxAVOlEE =gD2d -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2014-02-12 09:22 (GMT+0100) jdd composed:

Felix Miata composed:

...

I seem to have a failing STB HD, a "refurb" Samsung HD155UI relabeled as Seagate ST1500DL004 Barracuda Green, bought 11 months ago. Log from latest

there are usually test programms from the disk maker. The drawback is that often they only run on windows :-(, but at least once this utility could revive a failing disk (I never understood why, but after usage the disk was as good as new).

using these apps are mandatory to have them accept a hardware return

Usually my HDs are old enough to use whatever diag is on the last ultimatebootcd I burned, usually easier to locate than those on HD maker web sites.

http://www.seagate.com/fr/fr/support/warranty-and-replacements/

My focus had to be on file recovery. WRT to warranty it doesn't matter. It had a typical refurb warranty, 90 days. I've never seen any refurb HD with more than 6 months warranty. It failed the Seatools 1.09 short test @10% complete in 6 seconds even though it says SMART is enabled and was not tripped. Long doesn't look like it will be done for quite some time. It's only done 6% in 15 minutes... -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 02/12/2014 12:23 PM, Carlos E. R. wrote:

On 2014-02-12 11:14, Radule Šoškić wrote:

...

Various ddrescue (and other *rescue) programs are meant mainly for rescuing deleted or otherwise lost data -- they are very useful in carving artifacts of lost files and try to recover them into whole files. They act as a sort of digital forensic tools.

No, absolutely not. You are confusing tools.

"No"? --Hmm, might apply to some of the tools in the broad group... "Aboslutely not"? --This I think is a bit of an overstatement...

...

Using them on damaged disks might help but is very likely to make even more damage. If they encounter any I/O errors, they go to a loop of retries, which is OK on a sane disk, but not on a damaged one. On damaged disk you need as little retries as possible. The best is to copy everything that can be copied without retries first and then to repeat copying incomplete files with increased number of retries, etc.

That's precisely what dd_rhelp does.

Frankly, for dd_rhelp in particular, I don't know exactly. I've never used it. I only checked dd_rhelp command line switches before posting. In "dd_rhelp --help" output I didn't see any parameters for error handling control. So, I presumed it has to hva some max number of retries on errors (that would be normal for a decent copy tool, as errors are always expected). Sorry if I misled OP.

...

It might surprise you, but simple xcopy command in Windows might be of good help. Check this link: http://djlab.com/2010/12/windows-ignore-errors-with-xcopy-and-robocopy/

Huh?

How are you going to run 'xcopy' in Linux? Using wine? Yiks!

Well, when in such a trouble, one might try booting winPE, or even attaching HDD to another machine (running Windows). R.Soskic -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2014-02-12 13:05, Radule Šoškić wrote:

And now I checked dd_rescue. It defaults to maxerr=0, meaning "infinite". Take care.

But not dd_rhelp. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlL7bCMACgkQja8UbcUWM1whnAD/dIzGqbEM/dklgdRV6pwUL9l1 IUvCJsVIRHnWX6//bwUA/3otdpApMxDqe9EUqTkRxNTUpVbmuMdJcDV67Efgl/de =IXTN -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 El 2014-02-12 a las 15:48 +0100, Radule Šoškić escribió:

...

But not dd_rhelp.

Yes. Not dd_rhelp.

It (dd_rhelp) just wraps around dd_rescue (as you already mentioned before), without control over its maxerr parameter.

That's why I said "take care".

On the contrary, it does limit the number of read operations tried. - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlL8sRwACgkQja8UbcUWM1yunAD/ciES0p3Rw7wsXBD9/guygIto Wkx5OcJowbe3wxaE5C4A/0UzoO0tLEK5bn5RZYFJjMQiEAaP5BPBb1FVqd81sBWQ =xudm -----END PGP SIGNATURE-----

On 02/13/2014 03:40 PM, Greg Freemyer wrote:

"Radule Šoškić" wrote:

...

On 02/12/2014 12:23 PM, Carlos E. R. wrote:

...

On 2014-02-12 11:14, Radule Šoškić wrote:

...

Various ddrescue (and other *rescue) programs are meant mainly for rescuing deleted or otherwise lost data -- they are very useful in carving artifacts of lost files and try to recover them into whole files. They act as a sort of digital forensic tools.

No, absolutely not. You are confusing tools.

"No"? --Hmm, might apply to some of the tools in the broad group... "Aboslutely not"? --This I think is a bit of an overstatement...

Radule,

You might not like how *rescue work, but they are very focused tools designed to work with failing disks. Personally I use ewfacquire from libewf-tools as my first choice of tools for backing up a drive with minimal hardware failures.

Ewfacquire defaults to trying each read twice before moving on. It also allows you to specify a block size of say 64KB for working parts of the drive, but step down to a single sector on failing sections. I always make "images". Those can be restored to a new drive.

Ewfacquire does not do things like reverse reads, nor does it log progress such that future runs can work only on the failing sections of the drive. Gnu_ddrescue can do both of those. Thus if a drive has more than a couple bad sectors, I would use gnu_ddrescue instead.

Fyi: I make my living doing computer forensics and I can assure you *rescue have nothing to do with data carving.

Greg

Uh, oh. I already had a feeling that something went wrong with my post, after reading Mr. Carlos' reactions. Now, after reading yours, I went to reread my first post again. And the whole thread. I think I see now what happened. My intention was just to suggest two simple things: -- avoid a lot of re-reads on hdd and choose the tool that gives you good control on number of retries. -- even a simple general purpose copy tool might help better than a super-power specialized forensics tool if the latter defaults to a hard-coded high max_no_of_retries. But, while addressing the other class ot tools as a bunch, I used terms "various ddrescue" and "*rescue", which was obviously unfortunate choice. It was meant to mean: "various '*' tools exist around that are built for purpose of rescuing deleted and lost files -- dont use them..." But it happened that my random pick of terms matched the exact names of real existing tools. I got that only after reading the other branch of the thread. Foreign languages. The wrong wording happens from time to time. But this was exact hitting the extreme opposite of the intended. Not funny. R.Soskic -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org