[opensuse-kernel] aacraid driver crash when it shouldn't

15 Sep 2013

      Hi there, I'm facing an issue with 3.11 coming from our kernel-stable repos

A intel motherboard + i7 3770 32Gb ram with an adaptec 6805 adapter.
(8 intel ssd 510 series 120 go attached in a raid6 array)

The bundle adaptec + ssd in raid6 has worked during months (with openSUSE 12.1)
without any glitches.

Now this night we received that kind of error 

Sep 15 02:06:52 clochette.disney.interne kernel: aacraid: Host adapter abort request (6,0,0,0)
Sep 15 02:06:52 clochette.disney.interne kernel: aacraid: Host adapter abort request (6,0,0,0)
Sep 15 02:06:52 clochette.disney.interne kernel: aacraid: Host adapter reset request. SCSI hang ?
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: [sde] Medium access timeout failure. Offlining disk!
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: Device offlined - not ready after error recovery
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: Device offlined - not ready after error recovery
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: [sde] Unhandled error code
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: [sde]
Sep 15 02:07:07 clochette.disney.interne kernel: Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: [sde] CDB:
Sep 15 02:07:07 clochette.disney.interne kernel: Write(10): 2a 00 02 b7 f4 00 00 00 20 00
Sep 15 02:07:07 clochette.disney.interne kernel: end_request: I/O error, dev sde, sector 45609984
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-4, logical block 5252224
Sep 15 02:07:07 clochette.disney.interne kernel: lost page write due to I/O error on dm-4
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-4, logical block 5252225
Sep 15 02:07:07 clochette.disney.interne kernel: lost page write due to I/O error on dm-4
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-4, logical block 5252226
Sep 15 02:07:07 clochette.disney.interne kernel: lost page write due to I/O error on dm-4
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-4, logical block 5252227
Sep 15 02:07:07 clochette.disney.interne kernel: lost page write due to I/O error on dm-4
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: [sde] Unhandled error code
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: [sde]
Sep 15 02:07:07 clochette.disney.interne kernel: Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: [sde] CDB:
Sep 15 02:07:07 clochette.disney.interne kernel: Write(10): 2a 00 31 3e f6 20 00 00 08 00
Sep 15 02:07:07 clochette.disney.interne kernel: end_request: I/O error, dev sde, sector 826209824
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-3, logical block 853188
Sep 15 02:07:07 clochette.disney.interne kernel: lost page write due to I/O error on dm-3
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: rejecting I/O to offline device
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-3, logical block 2981929
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: rejecting I/O to offline device
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: rejecting I/O to offline device
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-3, logical block 2981929
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: rejecting I/O to offline device
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-3, logical block 45451
Sep 15 02:07:07 clochette.disney.interne kernel: lost page write due to I/O error on dm-3
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: rejecting I/O to offline device
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-3, logical block 785908
Sep 15 02:07:07 clochette.disney.interne kernel: lost page write due to I/O error on dm-3
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: rejecting I/O to offline device
Sep 15 02:07:07 clochette.disney.interne kernel: Buffer I/O error on device dm-3, logical block 810442
Sep 15 02:07:07 clochette.disney.interne kernel: lost page write due to I/O error on dm-3
Sep 15 02:07:07 clochette.disney.interne kernel: sd 6:0:0:0: rejecting I/O to offline device

But after checking kernel changelog I've found this commit for 3.10 series 
which should have fix it?

Author: Mahesh Rajashekhara 
Date:   Tue Jun 18 17:02:07 2013 +0530

    SCSI: aacraid: Fix for arrays are going offline in the system. System hangs

    commit c5bebd829dd95602c15f8da8cc50fa938b5e0254 upstream.

    One of the customer had reported that the set of raid logical arrays will
    become unavailable (I/O offline) after a long hours of IO stress test.  The OS
    wouldn`t be accessible afterwards and require a hard reset.

    This driver patch has a fix for race condition between the doorbell and the
    circular buffer. The driver is modified to do an extra read after clearing the
    doorbell in case there had been a completion posted during the small timing
    window.

    With this fix, we ran IO stress for ~13 days. There were no IO failures.

    Signed-off-by: Mahesh Rajashekhara 
    Signed-off-by: James Bottomley 
    Signed-off-by: Greg Kroah-Hartman 

Then it's hard to find what's going wrong ?

Should the firmware being downgraded to previous rev 19076 as it was before the mainboard change and openSUSE 12.3 installation ?

On this Raid6 we have a LVM group which contain virtuals machines who were running with a consequent load (oracle dumps) 
running from 1.5 to 2 hours.

Thanks for any pointer, or feedback 
I can provide more information if needed.

-- 

Bruno Friedmann 
Ioda-Net Sàrl www.ioda-net.ch

openSUSE Member
GPG KEY : D5C9B751C4653227
irc: tigerfoot