On 01/19/2010 09:25 PM, David Haller wrote:

After 72 Starts/Stops???

...

...

...

...

197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 207 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 207

This might be important. That _is_ important. "Pending Sectors" are defective, but just not yet "Reallocated Sectors", ditto "Offline Uncorrectable".

217 defects after 72 starts/stops is "DOA" ("Dead On Arrival").

...

...

You _must_ run the long test as soon as possible. Do not rely only on the short test. The long test includes a surface scan. NO!!! (not yet)

Replace the disc. NOW! (and get an image with GNU ddrescue[1] or dd_rescue if possible or at least get your data off that disc right effing now).

That disc is as good as dead!

After the data is saved, you can run the long test, and "wait and see" (and use the disc for temp stuff, such as rpm-caches, ccache, squid-cache and the like, readily expendable stuff anyway).

But better wipe the disc with GNU ddrescue or dd_rescue and input /dev/zero, and then get it replaced by Seagate (I'm rather sure it'll qualify for replacement, unless it's an OEM disc on which there's no guarantee).

Oh, yes: http://en.wikipedia.org/wiki/S.M.A.R.T.#ATA_S.M.A.R.T._attributes

HTH, -dnh

Dave, Thank you a lot. The disk is part of a mirrored set so basically I just need to split the array and run both standalone until I get the new drive back from seagate. I have a another new 750G drive that I could use in the interim or just leave them running standalone until the replacement gets here. I bet the bad sectors haven't been remapped due to the dmraid setup. I don't know if the disk can remap the sectors when the drive is part of an array -- I don't know what that would do to the sync among the disks?? Off to the seagate self-service site once again... -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Carlos E. R. wrote:

Except... The HD tries to move the data in a bad sector to a spare sector; but the data itself might be bad (read failure). Notice that this is an operation that happens totally inside the HD, without any consideration for the mirrored disk. Nobody outside of the HD knows about the problem.

Surely the user (or operating system at least) learns of a read failure? Wouldn't that lead to system/user-level retries and if they failed to rewriting of the sector from the mirror, which in turn would cause the remap? Just trying to understand how all the cogs interlock :) Cheers, Dave

The result could be that both sides had different data after the operation. However, the remapping occurs only during a write operation, so the preceding scenario would not happen.

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Thu, Jan 21, 2010 at 3:50 PM, Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Thursday, 2010-01-21 at 12:31 -0000, Dave Howorth wrote:

...

Carlos E. R. wrote:

...

Except... The HD tries to move the data in a bad sector to a spare sector; but the data itself might be bad (read failure). Notice that this is an operation that happens totally inside the HD, without any consideration for the mirrored disk. Nobody outside of the HD knows about the problem.

Surely the user (or operating system at least) learns of a read failure? Wouldn't that lead to system/user-level retries and if they failed to rewriting of the sector from the mirror, which in turn would cause the remap?

Just trying to understand how all the cogs interlock :)

No, the operating system doesn't know a thing, because this is completely internal to the HD firmware. I don't know the details, that is, I haven't seen a paper from a manufacturer explaining how exactly they do it. From what I gathered, when the HD attempts to write to a sector and it fails, and determines (somehow?) that that sector is bad and not recoverable, it decides to write the data to another sector, a spare sector defined as such during design by the manufacturer. Somehow, somewhere, external to the filesystem data, it stores that any read/write operation destined to the "bad" sector will happen instead on the remapped sector: meaning that the head has to move there, and operation is a tad slower.

All the system notices is that the original write operation went slower. The HD disk reports success... nothing happened. Afterward, if you run smartctl, you see the remap counter has gone one up, that's all.

It is possible that there is a protocol for the operating system to learn of this in real time, and perhaps do something. I'm not aware of that, but then, I'm not that expert :-)

It is different, though, if the problem occurs during a read. The system will probably get a read failure code, but the HD will do no remapping; I guess because it doesn't know what the correct data to write should be.

Again, it is possible that there is a protocol defined (perhaps it is manufacturer dependent) for the operating system to intervene and trigger a remap. I haven't heard of such, but certainly, in case of a raid, it would be very interesting to have.

- -- Cheers,       Carlos E. R.

Carlos, I think you have part of this wrong. I've never heard of a slow disk write. Disk drives do NOT verify what they write, they just write. The way it works is that the drive maintains crc info for every physical sector. On read it verifies the crc is correct. If not, then: Consumer drives assume that the only copy of the data is on the drive, so they have built in retry logic that can slow down the read by pretty extended times. (Seconds). Enterprise / raid drives assume you can get the data from other portions of the raid, so they have fast fail logic and don't bother to retry. They just immediately return a crc/media error to the kernel. Either way the drive tags that sector as bad. It does not relocate at this point, so you can keep trying data recovery techniques to your hearts content. ie. The infamous put your drive in the freezer overnight before trying to copy the data off. Eventually when new data is written to the bad sector, the drive internally remaps it to one of the spares, but this is fast. Thus writes are always fast as I understand it. === I have not been following this thread, but I'll assume mdraid is involved. mdraid monitors all sector reads for media errors. When detected it obviously rebuilds that sectors data from other drives. But, it also writes the recreated data back to the bad sector thus triggering the drive to do a remap. So if you have a large raid 1/5/etc. you should run a scan routinely. The scan will find the bad sectors and trigger a remap as described above. You can setup your scan to run routinely via cron. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Carlos E. R. said the following on 01/21/2010 03:50 PM:

On Thursday, 2010-01-21 at 12:31 -0000, Dave Howorth wrote:

No, the operating system doesn't know a thing, because this is completely internal to the HD firmware.

It is now. It didn't used to be, and I suspect it isn't always.

I don't know the details, that is, I haven't seen a paper from a manufacturer explaining how exactly they do it.

You go on to make a remarkably good guess.

From what I gathered, when the HD attempts to write to a sector and it fails, and determines (somehow?) that that sector is bad and not recoverable,

And that's the important point - is it recoverable? See later.

it decides to write the data to another sector, a spare sector defined as such during design by the manufacturer. Somehow, somewhere, external to the filesystem data, it stores that any read/write operation destined to the "bad" sector will happen instead on the remapped sector: meaning that the head has to move there, and operation is a tad slower.

All the system notices is that the original write operation went slower. The HD disk reports success... nothing happened. Afterward, if you run smartctl, you see the remap counter has gone one up, that's all.

It is different, though, if the problem occurs during a read. The system will probably get a read failure code, but the HD will do no remapping; I guess because it doesn't know what the correct data to write should be.

I don't know if that's the case now, but see below.

Again, it is possible that there is a protocol defined (perhaps it is manufacturer dependent) for the operating system to intervene and trigger a remap. I haven't heard of such, but certainly, in case of a raid, it would be very interesting to have.

That would be interesting .. Anyway: Back at the beginning the 1980s I was working for a UNIX "OEM" shop. Mostly we were porting UNIX to the new microprocessors. If you recall, that was when there were lots of 16-bit micros coming onto the market, most of which aren't around today. There were also a lot of manufacturers trying to do a computer-in-a -box, and wanted an OS, a *real* OS, not a single user thing like MS-DOS. Just as DEC had found a niche below IBM, they were finding niches below DEC/DG. I took an idea that a colleague had sketched out and wrote a disk driver for the PDP-11 under UNIX Version 7, you know, the tapes with "Love, Denis". I also back ported it to Version 6 for a Northern Telecom site. I still have the backup tapes of that project but they are probably unreadable now. The big difference between what you described and what I built was that those old huge drives had a CRC checksum at the end of each sector, and it contained enough information to perform at least one bit of correction. If you trusted the CRC absolutely you could perform a few more bits and do some run-length correction. However all this was carried out during an interrupt, so you didn't want to do too much computation. Once a bad read was detected and corrected the corrected sector was written to one of the spare sectors and the mapping table updated. That early correction and early re-mapping is the basic difference between what you described and what I built. I strongly suspect that modern on-board drive controllers do much the same. Well, OK it was a bit more than that. If every slight read error resulted in a remap those old drives would be remapped to hell and back! No, a first error caused a re-read. On some controllers you could pre-program that so by the time the s/w driver saw the error the hardware & microprogramming had given up. Of course an on-board drive controller has better low-level access to the drive and the raw signals than my host-based driver. However, I recall when using AIX with a large IBM RAID array which was supporting a extensive DB2 database that I once had to update the microcode on each and every disk drive in the array -- while it was running. (Yes, there were 'pauses' in performance.) This makes me think that at least some modern machines are well integrated with the internals of the disk drives, which addresses your final point. -- The scientific name for an animal that doesn't either run from or fight its enemies is lunch. - Michael Friedman -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Greg Freemyer said the following on 01/21/2010 05:09 PM:

Next generation drives will be 4K physical sectors. That way they can more effectively handle small media glitches without having to remap.

And about time too! I recall when Berkeley moved to 4K logical sectors for improved performance with their Fast File system back in the 1980s. And MS-DOS/Wondows has been able to use 4K logical sectors sicne .. when?

fyi: drives with 4k physical sectors are already shipping as of a couple months ago. The one I've heard of (WD?) still presents a 512byte logical sector interface to the controller.

Funny: my car has steering wheel and pedals, not a rein and stirrups. -- Sacred cows make the best hamburgers. --Mark Twain -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Hello, On Thu, 21 Jan 2010, Carlos E. R. wrote:

On Wednesday, 2010-01-20 at 04:25 +0100, David Haller wrote:

...

After 72 Starts/Stops???

And 6619 hours of use. Almost a year if it was continuous use.

More like 3/4ths. It's still a lot of defects and I guess the've all appeared rather lately (or dcr should have noticed earlier). So, it actually depends on when the defects appeared. 6k hours is not "infancy"/DOA, but already in the "stablest" phase of "bathtub curve of disc-deaths". And in that case e.g. googles data suggests immanent and permanent death and/or at least more defects and corrupted data "soonish". I currently have a disc (also a Seagate, but 1.5 TB), that had "suddenly" 5 defective sectors (but rather early in its use, I'd have to boot the server to look it up, a month or three IIRC) which I currently still use for replaceable data, the replacement disc already lies beside me, I just haven't got round to shuffle the data so that I can "migrate" to the new disc. Problem is: in that box I've got 10 discs and only 10 SATA-Ports and drive-bays are also quite full. Reminds me of (time-consuming) juggling ;) At least I already ran an rsync of the /-partition (and /home) to the temp-partition (>15G of stuff). BTW: of the IIRC 4 1.5TB Seagates, that's the only one with defective sectors (so far). But last year, I had 2(!) 500G Samsungs (same model) die on me just a few days apart (one died just like that, completely inaccessible). But I've got discs with >20k hrs (and start_stop_counts >4k) that have no defects (detected by SMART) (I recently switched the older 160G IDE, that had IIRC ~23k hours, for a new 500G in the old box). I think I've used and use >24 discs in the last 5 years (got 4 IDE discs in the old box, 2x500G, 300G, 160G), so I've got a currently running sample of 14 discs of varying ages and sizes, IDE and SATA, with IIRC at least 5 disc replaced (size/death reasons) in the last 2 years. So, I retract my "DOA" (didn't think of the Power On Hours then, and 72 Start/Stop would be <= 36 days here ;), but 270 defects after a mere ~6.6k hours is way too much for my liking. Less than 10 (appearing soonish), and then stable for at least a month, I'd still not trust the disc, but might use it for a while longer (as I do with above mentioned 1.5TB) for expendable data (caches, tempfiles, whatever one can lose or get corrupted at any time and have no worries). -dnh PS: total is currently 1.3 TB (IDE, old box) + IIRC ~12 TB (both formatted and in 2^40Bytes sizes!). And, as usual, see non-random sig, is >95% full, which quite throws Yast (and anything else) off whack, as the "not enough", say, 5% of free space amount to a whopping ~500G or so, and say 10G free on /, which is plenty for updates, or another install alongside ;) I guess I should file a bug that it should look at the space, not the percentage. Though: Yast hasn't complained about discspace running low lately. Maybe it's already fixed :) Oh, and: I've had "full" discs since my first HDD. A then huge 850MB Maxtor in my very first PC. I _so_ agree with Ken ... *sigh* -- The steady state of disks is full. -- Ken Thompson -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On 01/19/2010 05:44 PM, Carlos E. R. wrote:

That you have remapped sectors is important, but not definitive. You have to watch it, and see if the count increases. When the count increses so much that the disk runs out of spare sectors, then you have to switch the disk fast.

Hmm, this one is still under warranty. At least seagate will 'pre-ship' a replacement for $20 that comes with the pre-paid return envelope (packing too :-). What that doesn't explain is why in the heck have I had to send back 3 out of 4 of my 750G drives in the past 12 months?? Shame on you Seagate. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

ka1ifq said the following on 01/20/2010 10:23 AM:

"On 02:14:46 am "David C. Rankin" said"

...

Hmm, this one is still under warranty. At least seagate will 'pre-ship' a replacement for $20 that comes with the pre-paid return envelope (packing too :-).

What that doesn't explain is why in the heck have I had to send back 3 out of 4 of my 750G drives in the past 12 months?? Shame on you Seagate.

--

I had a 500g go bad in less than a year, also did the pre-ship, worth the $20.

Its all about statistics. Somewhere out there is a 500G drive that will work perfectly for another 50 years. But its the ones that fail we hear complaints about. Of course no-one will be using those 500G drives 50 years from now :-) -- There is no use whatever trying to help people who do not help themselves. You cannot push anyone up a ladder unless he be willing to climb himself. - Andrew Carnegie -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

kanenas@hawaii.rr.com said the following on 01/20/2010 04:26 PM:

On Wednesday 20 January 2010 05:33:09 am Anton Aylward wrote:

...

Its all about statistics. Somewhere out there is a 500G drive that will work perfectly for another 50 years. But its the ones that fail we hear complaints about.

That would be so *if* they sold enough of them.

but it is also all about statistics when 3 out of 4 drives have to be returned. That almost by itself leads into the conclusion that the good old 6 sigma production ethic (it *is* an ethic more than a process) is out the window...

No, its about economics. Manufacturing gets to a point where increasing the reliability by another 1% adds another 10%, 15%, 30%, 100%, 600% ... exponentially to the production cost. At some point its cheaper to live with what reliability figures you've got and unconditionally replace any failed units than it is to try an improve the reliability. After all, its the bean counters that run the company, the investors wanting a ROI, the executives who have to face up to the Wall Street analysts scathing comments about the performance of the stock, not the engineers and certainly not the customers. -- If you obey all the rules, you miss all the fun. - Katharine Hepburn -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On 01/20/2010 10:03 AM, Jeffrey L. Taylor wrote:

Quoting ka1ifq :

...

"On 02:14:46 am "David C. Rankin" said"

...

Hmm, this one is still under warranty. At least seagate will 'pre-ship' a replacement for $20 that comes with the pre-paid return envelope (packing too :-).

What that doesn't explain is why in the heck have I had to send back 3 out of 4 of my 750G drives in the past 12 months?? Shame on you Seagate.

Were the 4 from the same batch (i.e., ordered together)? There are occasional bad batches.

Also, statistics: assuming a Gaussian distribution, 10% will fail in less than 1/10 of the MTBF (Mean Time Between Failures). Plus infant mortality, many things will have high rates of failure for the first part of their expected lifetime, then the failure rate will be low and flat for many months/years and then the failure rate will start rising. A bunch of disk failures in the first months is unfortunate, but not terribly surprising. A bunch of 2 year drives failing would be surprising and probably points to environmental concerns, typically heat.

Have you checked ambient and drive temperatures?

Jeffrey

That's the strange part, the drives were two 500G and two 750G drives ordered at different time. Both 500's were bad (one replaced twice) and this is the second 750 that has gone bad. The 500's are now perfect. In the 17 years before that (probably 20 hard drives), the ONLY time a hard drive ever failed on me was after the computer had been stored in the attic for a while. Good luck maybe :p As far as temps go, they are always perfect (mid 30s). The case is an Antec 180ss with 4 120mm fans. Example: 10:39 archangel:~> hdtemp /dev/sdb: ST3750330AS: 34°C /dev/sdc: ST3500630AS: 38°C /dev/sdd: ST3750330AS: 35°C /dev/sda: ST3500630AS: 35°C (sdb is the current failing drive) Note: hdtemp is just a simple script that calls hddtemp for all 4 drives. I don't know what the issue is here, but I do know -- it's not heat.... -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org