[opensuse-kernel] ATA errors
http://fm.no-ip.com/Tmp/Linux/messages-gx280-esata.txt (32bit 12.1 kernel-desktop-3.1.9) http://fm.no-ip.com/Tmp/Linux/messages-gx620-esata.txt (32bit 12.1 kernel-desktop-3.1.9) Is there anything in these /var/log/messages excerpts from two different hosts that can identify which (external) sata device is connected with the reported errors? I don't see any connection between that information and the content of /dev/disk/[by-path,by-id]. Is there a way to tell if a kernel bug or a device is the problem? These errors show up in conjunction with various degrees of I/O delay, sometimes as bad as several minutes of inability to get bash to respond to keyboard input. The most recent time some commands would result in a simple segfaulted. Others would produce bash: input/output error. Ctrl-Alt-Del failed to reboot, as did init 6 (input/output error), forcing me to use power switch and wait a seeming eternity for fsck on 1TB+ partitions to complete. I've run the Seagate thorough diagnostics on the drives, 100% error free. The suspect device is http://www.newegg.com/Product/Product.aspx?Item=N82E16817173042 but I want to provide Rosewill tech support hard evidence of the incompetence of the device if I can be sure it is exclusively responsible for the errors. I bought 5 of them and never had trouble. Only with #6, the only version 2 of the model I have, has there been a problem. The #6 I have now is actually the 3rd replacement of the original #6 purchased last September. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/14/2012 02:00 AM, Felix Miata wrote:
http://fm.no-ip.com/Tmp/Linux/messages-gx280-esata.txt (32bit 12.1 kernel-desktop-3.1.9) http://fm.no-ip.com/Tmp/Linux/messages-gx620-esata.txt (32bit 12.1 kernel-desktop-3.1.9)
These are both coming up with 403s. - -Jeff
Is there anything in these /var/log/messages excerpts from two different hosts that can identify which (external) sata device is connected with the reported errors? I don't see any connection between that information and the content of /dev/disk/[by-path,by-id]. Is there a way to tell if a kernel bug or a device is the problem? These errors show up in conjunction with various degrees of I/O delay, sometimes as bad as several minutes of inability to get bash to respond to keyboard input. The most recent time some commands would result in a simple segfaulted. Others would produce bash: input/output error. Ctrl-Alt-Del failed to reboot, as did init 6 (input/output error), forcing me to use power switch and wait a seeming eternity for fsck on 1TB+ partitions to complete.
I've run the Seagate thorough diagnostics on the drives, 100% error free.
The suspect device is http://www.newegg.com/Product/Product.aspx?Item=N82E16817173042 but I want to provide Rosewill tech support hard evidence of the incompetence of the device if I can be sure it is exclusively responsible for the errors. I bought 5 of them and never had trouble. Only with #6, the only version 2 of the model I have, has there been a problem. The #6 I have now is actually the 3rd replacement of the original #6 purchased last September.
- -- Jeff Mahoney SUSE Labs -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJPiZowAAoJEB57S2MheeWyEBUP/A0ZEQezBakc3E/O24a4ZxSV YhctOH1NKl7+kIng3u6uLVnKA7j5f1D9BQa/O0uDlDC6qrmuHouMIwt9p8GHHpR9 UgTq/Qfm6F4hUu3QnG43tnmOVNUCGyVUNpsF8e2TAVGWWw+JTol23Yfu28OIiEu7 g0KpBCT/Lw/cyAbJgYXlKbC5vsDt5WNTDelHHcptKCvgyOxE1uI8GffEIW3+0I/V btLhKBuAQ/RovGKtqT9TwgoIBIlBglePQzFMbEbLCShRo4ppmYVrWHkqbl0zcOBY tyUd5alkwMk7zXqydp+oGkP5qcwZTIjiPnpD70coGCkUB3+x7QB2d4fsw/IiUmaN RWwGirjOX5+CRxZahwyLHav1AkIkwK3PZ2lTZVWz2P0o+b8C2aSm6oTisNQU3+I7 FdCA21AWqQpbH/Mr0O3yjxJI/AshnWmxtnd03ruoyIKD6B+6mLI6c2/LvNeoUkX9 aDmYnH8/NJLSbCUdd4S8dqt5K3Mz5y1pbroJxThJoMOeYaQsIjPmJtPSN1iffvma vrfmAq6rqDifE4lYSh8KhsI4Gy4wzClKBQdZtOG3zEzdQeo8VOiZr4NbBkrm1tbu BKocfNzlXqevl1wHS+1KArokWMK3Vxgy/7vqThgrI6c87K2eGEFqlhO5neI448xP MQOz/1MyfRN52qEHe389 =rWla -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 2012/04/14 11:39 (GMT-0400) Jeff Mahoney composed:
Felix Miata wrote:
http://fm.no-ip.com/Tmp/Linux/messages-gx280-esata.txt (32bit 12.1 kernel-desktop-3.1.9) http://fm.no-ip.com/Tmp/Linux/messages-gx620-esata.txt (32bit 12.1 kernel-desktop-3.1.9)
These are both coming up with 403s.
Fixed. Thanks! -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/14/2012 02:00 AM, Felix Miata wrote:
http://fm.no-ip.com/Tmp/Linux/messages-gx280-esata.txt (32bit 12.1 kernel-desktop-3.1.9) http://fm.no-ip.com/Tmp/Linux/messages-gx620-esata.txt (32bit 12.1 kernel-desktop-3.1.9)
Is there anything in these /var/log/messages excerpts from two different hosts that can identify which (external) sata device is connected with the reported errors? I don't see any connection between that information and the content of /dev/disk/[by-path,by-id]. Is there a way to tell if a kernel bug or a device is the problem? These errors show up in conjunction with various degrees of I/O delay, sometimes as bad as several minutes of inability to get bash to respond to keyboard input. The most recent time some commands would result in a simple segfaulted. Others would produce bash: input/output error. Ctrl-Alt-Del failed to reboot, as did init 6 (input/output error), forcing me to use power switch and wait a seeming eternity for fsck on 1TB+ partitions to complete.
You'll find that info earlier in the log. e.g. on my system: [ 1.290219] ata1.00: ATA-8: OCZ-VERTEX2, 1.35, max UDMA/133 Alternatively, lsscsi -v will show: [0:0:0:0] disk ATA OCZ-VERTEX2 1.35 /dev/sda dir: /sys/bus/scsi/devices/0:0:0:0 [/sys/devices/pci0000:00/0000:00:11.0/ata1/host0/target0:0:0/0:0:0:0] ... where you can see the 'ata1' in the sysfs path. The problem with that type of error is that it's tough to isolate what the problem is without detailed hardware knowledge (the type of knowledge I don't personally have). Joey's team has more experience with ATA issues. Perhaps they can assist. - -Jeff
I've run the Seagate thorough diagnostics on the drives, 100% error free.
The suspect device is http://www.newegg.com/Product/Product.aspx?Item=N82E16817173042 but I want to provide Rosewill tech support hard evidence of the incompetence of the device if I can be sure it is exclusively responsible for the errors. I bought 5 of them and never had trouble. Only with #6, the only version 2 of the model I have, has there been a problem. The #6 I have now is actually the 3rd replacement of the original #6 purchased last September.
- -- Jeff Mahoney SUSE Labs -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJPiboTAAoJEB57S2MheeWyz7UP/3dU1aCuJu/hs+d0W9dmCojN /1455tvjXYCKIX5GurvJXgt6pHUlheWgpbWVHgORUfB6BOWOnkWyBwY7vXsdF+TR QNxI0Q/IcvcicrFL2kQe66+GOGZoatttb7KhAb0XXiVrKfyzAiHgl7LCGem7clXQ yWJ2Nm0os9yZ+iYMelVA0SIDRB/1fSMvAy+L8UdwpIv6c4mvwqj+Vo/oi0GB8xvm LQZFQ5i9sAj00V/hBQxb12WnfNDIZvrW+FXR4uufSYtzaVzRoq7Ui6cjKW4mREnI o6p7td+wbEaL397mO9zHKGmkURWRW3qwfoLtq4DJyhcmvzsWhn2KsqNeXrHG5KAD MpaQLE+xuKmE/egxCE9TSm+Eks+pNhocugKd3uJSI4P1164oHFBlKycQAOpQvWcH 31v6LRCX08vfRHyL/4diAqw249fPWM1ONR+mNnbzuZffmDz5m0GsVdK2oY1b93Hd Qa4QMIFvYDfGamlH8frlc/9sIVXBSyBQDHrLeRbuGz2PAxmYJQPAbP6+eV1Jv82/ 9z0UMCLlK0gr7PD4p3v+kVEXpwTz8sPEPaA4zejrfP5H1+egVDA+v6SknLpmEdVy qfU4D0ru/9AaJckpA7KtgyXexkDcb0znHWwsy71LxSQPeSuV9arEj7VDpcZw2mJ+ QWOfKrzGhUiLxjU3oauP =sak/ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hi Felix, 於 六，2012-04-14 於 13:55 -0400，Jeff Mahoney 提到：
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/14/2012 02:00 AM, Felix Miata wrote:
http://fm.no-ip.com/Tmp/Linux/messages-gx280-esata.txt (32bit 12.1 kernel-desktop-3.1.9)
There have write DMA timeout: Apr 13 13:54:50 gx280 smartd[1613]: Device: /dev/sdc [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 25 to 30 Apr 13 14:16:48 gx280 kernel: [ 4975.072049] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Apr 13 14:16:48 gx280 kernel: [ 4975.072056] ata5.00: failed command: WRITE DMA EXT Apr 13 14:16:48 gx280 kernel: [ 4975.072065] ata5.00: cmd 35/00:08:48:68:a8/00:00:ae:00:00/e0 tag 0 dma 4096 out Apr 13 14:16:48 gx280 kernel: [ 4975.072067] res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Apr 13 14:16:48 gx280 kernel: [ 4975.072071] ata5.00: status: { DRDY } Apr 13 14:16:48 gx280 kernel: [ 4975.072082] ata5: hard resetting link Apr 13 14:16:49 gx280 kernel: [ 4975.377042] ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Apr 13 14:16:49 gx280 kernel: [ 4975.405487] ata5.00: configured for UDMA/100 Apr 13 14:16:49 gx280 kernel: [ 4975.405502] ata5: EH complete It causes machine pending until timeout.
http://fm.no-ip.com/Tmp/Linux/messages-gx620-esata.txt (32bit 12.1 kernel-desktop-3.1.9)
Have READ DMA timeout: Apr 13 15:18:38 vizio kernel: [ 265.732111] EXT4-fs (sdb2): mounted filesystem without journal. Opts: (null) Apr 13 15:29:41 vizio kernel: [ 929.056055] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Apr 13 15:29:41 vizio kernel: [ 929.056063] ata5.00: failed command: READ DMA Apr 13 15:29:41 vizio kernel: [ 929.056071] ata5.00: cmd c8/00:08:18:38:00/00:00:00:00:00/e0 tag 0 dma 4096 in Apr 13 15:29:41 vizio kernel: [ 929.056073] res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Apr 13 15:29:41 vizio kernel: [ 929.056077] ata5.00: status: { DRDY } Apr 13 15:29:41 vizio kernel: [ 929.056089] ata5: hard resetting link Apr 13 15:29:42 vizio kernel: [ 929.361042] ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Apr 13 15:29:42 vizio kernel: [ 929.405441] ata5.00: configured for UDMA/100 Apr 13 15:29:42 vizio kernel: [ 929.405456] ata5: EH complete
Is there anything in these /var/log/messages excerpts from two different hosts that can identify which (external) sata device is connected with the reported errors? I don't see any connection between that information and the content of /dev/disk/[by-path,by-id]. Is there a way to tell if a kernel bug or a device is the problem? These errors show up in conjunction with various degrees of I/O delay, sometimes as bad as several minutes of inability to get bash to respond to keyboard input. The most recent time some commands would result in a simple segfaulted. Others would produce bash: input/output error. Ctrl-Alt-Del failed to reboot, as did init 6 (input/output error), forcing me to use power switch and wait a seeming eternity for fsck on 1TB+ partitions to complete.
You'll find that info earlier in the log. e.g. on my system: [ 1.290219] ata1.00: ATA-8: OCZ-VERTEX2, 1.35, max UDMA/133
Alternatively, lsscsi -v will show: [0:0:0:0] disk ATA OCZ-VERTEX2 1.35 /dev/sda dir: /sys/bus/scsi/devices/0:0:0:0 [/sys/devices/pci0000:00/0000:00:11.0/ata1/host0/target0:0:0/0:0:0:0]
... where you can see the 'ata1' in the sysfs path.
Yes, please use lsscsi to check the ata5:00 (5 is ata port id) is your external eSATA box.
The problem with that type of error is that it's tough to isolate what the problem is without detailed hardware knowledge (the type of knowledge I don't personally have).
Joey's team has more experience with ATA issues. Perhaps they can assist.
- -Jeff
Currently, no idea why the command timeout when host try to write/read device through DMA. Did you try to use this eSATA box on other OS? And, Please kindly file a bug on https://bugzilla.novell.com and attached on your whole /var/log/messages, /var/log/boot.msg and lsmod result. I will investigate it. Thanks a lot! Joey Lee
I've run the Seagate thorough diagnostics on the drives, 100% error free.
The suspect device is http://www.newegg.com/Product/Product.aspx?Item=N82E16817173042 but I want to provide Rosewill tech support hard evidence of the incompetence of the device if I can be sure it is exclusively responsible for the errors. I bought 5 of them and never had trouble. Only with #6, the only version 2 of the model I have, has there been a problem. The #6 I have now is actually the 3rd replacement of the original #6 purchased last September.
- -- Jeff Mahoney SUSE Labs -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBAgAGBQJPiboTAAoJEB57S2MheeWyz7UP/3dU1aCuJu/hs+d0W9dmCojN /1455tvjXYCKIX5GurvJXgt6pHUlheWgpbWVHgORUfB6BOWOnkWyBwY7vXsdF+TR QNxI0Q/IcvcicrFL2kQe66+GOGZoatttb7KhAb0XXiVrKfyzAiHgl7LCGem7clXQ yWJ2Nm0os9yZ+iYMelVA0SIDRB/1fSMvAy+L8UdwpIv6c4mvwqj+Vo/oi0GB8xvm LQZFQ5i9sAj00V/hBQxb12WnfNDIZvrW+FXR4uufSYtzaVzRoq7Ui6cjKW4mREnI o6p7td+wbEaL397mO9zHKGmkURWRW3qwfoLtq4DJyhcmvzsWhn2KsqNeXrHG5KAD MpaQLE+xuKmE/egxCE9TSm+Eks+pNhocugKd3uJSI4P1164oHFBlKycQAOpQvWcH 31v6LRCX08vfRHyL/4diAqw249fPWM1ONR+mNnbzuZffmDz5m0GsVdK2oY1b93Hd Qa4QMIFvYDfGamlH8frlc/9sIVXBSyBQDHrLeRbuGz2PAxmYJQPAbP6+eV1Jv82/ 9z0UMCLlK0gr7PD4p3v+kVEXpwTz8sPEPaA4zejrfP5H1+egVDA+v6SknLpmEdVy qfU4D0ru/9AaJckpA7KtgyXexkDcb0znHWwsy71LxSQPeSuV9arEj7VDpcZw2mJ+ QWOfKrzGhUiLxjU3oauP =sak/ -----END PGP SIGNATURE-----
-- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 2012/04/16 18:40 (GMT+0800) joeyli composed:
On 04/14/2012 02:00 AM, Felix Miata wrote:
http://fm.no-ip.com/Tmp/Linux/messages-gx280-esata.txt (32bit 12.1 kernel-desktop-3.1.9) ... http://fm.no-ip.com/Tmp/Linux/messages-gx620-esata.txt (32bit 12.1 kernel-desktop-3.1.9) ... Is there anything in these /var/log/messages excerpts from two different hosts that can identify which (external) sata device is connected with the reported errors? I don't see any connection between that information and the content of /dev/disk/[by-path,by-id]. Is there a way to tell if a kernel bug or a device is the problem? These errors show up in conjunction with various degrees of I/O delay, sometimes as bad as several minutes of inability to get bash to respond to keyboard input. The most recent time some commands would result in a simple segfaulted. Others would produce bash: input/output error. Ctrl-Alt-Del failed to reboot, as did init 6 (input/output error), forcing me to use power switch and wait a seeming eternity for fsck on 1TB+ partitions to complete.
Currently, no idea why the command timeout when host try to write/read device through DMA.
Did you try to use this eSATA box on other OS?
This eSATA box is a third replacement of an original (total of 4 IIRC) bought last September from Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16817173042 This device is a v2 model purchased only after buying 5 of the v1 model and never having had any trouble with the v1 models. I've not yet had errors with this 4th one when connected to WinXP, but I did with the others. I rarely have reason to connect it to Windows, and have really only done it when problems using the device with a STB cause NTFS corruption that the STB could not fix and required CHKDSK to repair. Until today I had only connected the device to the STB, WinXP, and multiple machines running OS 11.4, 12.1 & 12.2, with various kernels on each oS. They are all multiboot hosts. Today I got Mandriva Cooker running on one machine with eSATA (in SATA mode only) available via both the motherboard ICH6 controller and PCI add-in card SiL 3512. Now I can verify the problem is not exclusive to openSUSE, but is exclusive to the v2 device, regardless which HD is installed in it. It also manifests regardless whether attached to ICH6 or SiL 3512. None of my v1 devices give this trouble.
And, Please kindly file a bug on https://bugzilla.novell.com and attached on your whole /var/log/messages, /var/log/boot.msg and lsmod result.
I will investigate it.
Doesn't this look more like upstream? Shouldn't I file there, if at all? Maybe upstream and you can combine to provide some workaround if I file there, since the manufacturer doesn't seem able to supply a usable replacement? I was expecting this to be a device problem to be corrected by the manufacturer, and hoping for information from this thread to present to the manufacturer to confirm its faulty replacement design and/or firmware, but certainly I'll file a bug it you think it's the right way to proceed. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
於 一，2012-04-16 於 11:47 -0400，Felix Miata 提到：
On 2012/04/16 18:40 (GMT+0800) joeyli composed:
On 04/14/2012 02:00 AM, Felix Miata wrote:
http://fm.no-ip.com/Tmp/Linux/messages-gx280-esata.txt (32bit 12.1 kernel-desktop-3.1.9) ... http://fm.no-ip.com/Tmp/Linux/messages-gx620-esata.txt (32bit 12.1 kernel-desktop-3.1.9) ... Is there anything in these /var/log/messages excerpts from two different hosts that can identify which (external) sata device is connected with the reported errors? I don't see any connection between that information and the content of /dev/disk/[by-path,by-id]. Is there a way to tell if a kernel bug or a device is the problem? These errors show up in conjunction with various degrees of I/O delay, sometimes as bad as several minutes of inability to get bash to respond to keyboard input. The most recent time some commands would result in a simple segfaulted. Others would produce bash: input/output error. Ctrl-Alt-Del failed to reboot, as did init 6 (input/output error), forcing me to use power switch and wait a seeming eternity for fsck on 1TB+ partitions to complete.
Currently, no idea why the command timeout when host try to write/read device through DMA.
Did you try to use this eSATA box on other OS?
This eSATA box is a third replacement of an original (total of 4 IIRC) bought last September from Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16817173042
This device is a v2 model purchased only after buying 5 of the v1 model and never having had any trouble with the v1 models.
I've not yet had errors with this 4th one when connected to WinXP, but I did with the others. I rarely have reason to connect it to Windows, and have really only done it when problems using the device with a STB cause NTFS corruption that the STB could not fix and required CHKDSK to repair.
Until today I had only connected the device to the STB, WinXP, and multiple machines running OS 11.4, 12.1 & 12.2, with various kernels on each oS. They are all multiboot hosts. Today I got Mandriva Cooker running on one machine with eSATA (in SATA mode only) available via both the motherboard ICH6 controller and PCI add-in card SiL 3512.
Now I can verify the problem is not exclusive to openSUSE, but is exclusive to the v2 device, regardless which HD is installed in it. It also manifests regardless whether attached to ICH6 or SiL 3512. None of my v1 devices give this trouble.
Thanks for your investigation. Per the DMA timeout error in your messages, it happen when host try to read/write data through DMA from device 0 on host4: ata5.00: cmd 35/00:08:38:58:d0/00:00:74:00:00/e0 tag 0 dma 4096 out The e0 is 11100000, bit 4 is 0 means device 0, bit 5 is 1 means LBA enabled. And, per your lsscsi result, device 0 is WDC WD800JD-75MS, device 1 is ST32000542AS, I think ST32000542AS is your external disk in eSATA box. Simply say, the problem is host failed to access device 0 through DMA when you plug-in RX-358 v2 model. I am investigating and will put information on bugzilla.
And, Please kindly file a bug on https://bugzilla.novell.com and attached on your whole /var/log/messages, /var/log/boot.msg and lsmod result.
I will investigate it.
Doesn't this look more like upstream? Shouldn't I file there, if at all? Maybe upstream and you can combine to provide some workaround if I file there, since the manufacturer doesn't seem able to supply a usable replacement?
I was expecting this to be a device problem to be corrected by the manufacturer, and hoping for information from this thread to present to the manufacturer to confirm its faulty replacement design and/or firmware, but certainly I'll file a bug it you think it's the right way to proceed.
You can try the 3.4-rc2 kernel rpm here: https://build.opensuse.org/package/show?package=kernel-default&project=Kernel%3AHEAD You can file bug to kernel upstream (bugzilla or mail) if can reproduce issue on 3.4-rc2. Please kindly add Cc. to me if you file it to kernel upstream. Thanks a lot! Joey Lee -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
participants (3)
-
Felix Miata -
Jeff Mahoney -
joeyli