https://bugzilla.novell.com/show_bug.cgi?id=461277
User teheo@novell.com added comment
https://bugzilla.novell.com/show_bug.cgi?id=461277#c27
Tejun Heo changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
Info Provider| |quentin.jackson@exclamation
| |.co.nz
--- Comment #27 from Tejun Heo 2009-02-03 20:05:38 MST ---
Hello, all.
Christoph, I don't think you're seeing the same problem. Can you please file a
separate bug report? Quentin, Kenn, you guys seem to be experiencing the same
problem. I'm sorry about the hassles you had to go through but I cannot help
finding the pattern very interesting. :-)
Anyways, you guys are both seeing PHYRDY_CHG/DEVXCHG on 0xEA (FLUSH_CACHE) and
following command failures and data loss/corruption on RAID arrays.
PHYRDY_CHG/DEVXCHG on heavy load followed by data loss is highly indicative of
power problem. The power supply can't hold 12v steady on sudden spike, voltage
drops briefly (really briefly), the drive powers down briefly performing
emergency head unload and forgets everything yet to be flushed in its write
cache. When the power supply catches up, the drive powers up again, PHY comes
back online triggering DEVXCHG on the controller side. From the controller's
view point, the event looks just like a brief communication failure. It simply
has no way of determining that the drive has lost content in its write buffer,
so the drive gets reset and access resumes. If the drive is kicked out of the
array, it's lucky. If not, massive file system corruption is likely to follow.
Now, why whould 11.1 be different from 11.0? Upto 11.0, md devices didn't have
barrier support, which means that regardless of whatever the filesystem does or
wants, each hard drive in the array would write out cache in small steps when
it sees fit most likely when the head is passing over the area for other tasks.
11.1 enabled barrier support for md, so now when the filesystem wants some
data to be on platter before other data, barrier gets forwarded to all member
drives as the FLUSH command which forces the drives to flush their cache at
once. So, now, when a FS issues a barrier, all drives begin writing out like
crazy at the _exact_ same moment. Under normal circumstances, this _should_ be
fine but seriously there are a lot of crappy power supplies out there
regardless how high wattage number they have printed on themselves.
Of course, it can be something different but given the symptoms, I don't think
it's very likely. You can confirm the problem by doing the followings.
1. The drive is likely to increment one or more of start_stop_count, head
unload count and emergency unload count after such failure. Run smartctl -a on
boot and after such failure, run smartctl -a again and compare which counts
have changed. If any of those changed, the drive lost power over the event.
2. If the system is relative quiet, you can often hear these failures. The
drives often make distinctive clunking sound doing emergency head unload. Put
your ears close to the drives and trigger the problem and see whether you can
hear anything.
3. Cure is sometimes the best diagnosis. Prepare a separate power supply and
offload some drives to it (say, half of them). Expensive doesn't always equal
good, especially the multi rail ones. 300w single rail one is often much
better than 450w multi rail one. You can power up power supply separately
by...
http://modtown.co.uk/mt/article2.php?id=psumod
So, can you guys please try to verify the problem?
Thanks.
--
Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.