[Bug 366492] New: file system corruption and scsi errors with open-iscsi and Infortrend iSCSI arrays - openSUSE Bugs - openSUSE Mailing Lists

newer
[Bug 359880] New: nautilus / D&D...

[Bug 366492] New: file system corruption and scsi errors with open-iscsi and Infortrend iSCSI arrays

older
[Bug 359306] New: Clock...

bugzilla_noreply＠novell.com

3 Mar 2008 3 Mar '08

03:10

https://bugzilla.novell.com/show_bug.cgi?id=366492 Summary: file system corruption and scsi errors with open-iscsi and Infortrend iSCSI arrays Product: SUSE Linux 10.1 Version: Final Platform: x86-64 OS/Version: SLES 10 Status: NEW Severity: Critical Priority: P5 - None Component: Network AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: s.pigneguy@niwa.co.nz QAContact: qa@suse.de Found By: Customer Starting with a fresh installation of SLES 10, and upgrading to SP1 I can reliably get open-iscsi to corrupt a filesystem and return scsi errors with an Infortrend iSCSI array. The problem goes away when I use the latest stable release of open-iscsi from the open-iscsi website. Previous to using the latest version of open-iscsi I had replaced all our network switches to quality cisco equipment, reinstalled & reloaded servers, reformatted the iSCSI arrays (we have two identical arrays). The problem version of open-iscsi is open-iscsi-2.0.707-0.24 on both i386 & x86_64 The version of open-iscsi that works is open-iscsi-2.0-865.15 The scsi errors I see are like : (summary of dmesg) sd 14:0:0:1: SCSI error: return code = 0x00020000 end_request: I/O error, dev sdj, sector 3379268680 (summary of /var/log/messages) Feb 26 12:59:19 thor kernel: Buffer I/O error on device dm-9, logical block 67363007 Feb 26 12:59:19 thor kernel: lost page write due to I/O error on dm-9 Feb 27 12:46:43 thor kernel: end_request: I/O error, dev sdj, sector 2041855152 Feb 27 12:46:43 thor kernel: I/O error in filesystem ("dm-9") meta-data dev dm-9 block 0x79b43b30 ("xfs_trans_read_buf") error 5 buf count 8192 Feb 27 12:46:44 thor iscsid: connection1:0 is operational after recovery (1 attempts) The Infortrend iSCSI Array model is : A16E-G2130-4 The network switch model is : Cisco 2960G The server models are : HP DL380, HP DL360 Thanks, Stuart. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

Reply

Sign in to reply online Use email software

Show replies by date

bugzilla_noreply＠novell.com

3 Mar 3 Mar

16:38

New subject: [Bug 366492] file system corruption and scsi errors with open-iscsi and Infortrend iSCSI arrays

https://bugzilla.novell.com/show_bug.cgi?id=366492 Cyril Hrubis changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team-screening@forge.provo.novell.com |hare@novell.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

Reply

Sign in to reply online Use email software

bugzilla_noreply＠novell.com

4 Mar 4 Mar

13:44

New subject: [Bug 366492] file system corruption and scsi errors with open-iscsi and Infortrend iSCSI arrays

https://bugzilla.novell.com/show_bug.cgi?id=366492 User hare@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=366492#c1 Hannes Reinecke changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |s.pigneguy@niwa.co.nz --- Comment #1 from Hannes Reinecke 2008-03-04 06:44:26 MST --- Somehow we incurred connection problems on the iscsi side and it took longer for the daemon to recover. Are you running multipathing? If not, please increase the number of retries under /sys/block/sdX/device/retries to eg 10 and see if the error still occurs. If it does, please get a tcpdump with tcpdump -i ethX -s 0 -w /tmp/tcpdump.log and attach it here. THX. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

Reply

Sign in to reply online Use email software

bugzilla_noreply＠novell.com

20:13

New subject: [Bug 366492] file system corruption and scsi errors with open-iscsi and Infortrend iSCSI arrays

https://bugzilla.novell.com/show_bug.cgi?id=366492 User s.pigneguy@niwa.co.nz added comment https://bugzilla.novell.com/show_bug.cgi?id=366492#c2 --- Comment #2 from Stuart Pigneguy 2008-03-04 13:13:46 MST --- FYI: I am not running multipathing. I will check and increase retries to see if that makes a change on the SuSE provided drivers. The tcpdump will be gigabytes in size, and the contents of the dump will be inaccurate as tcpdump cannot keep up with packets coming in at 1gb/s. I have a snapshot from earlier: 192.168.50.8 is the iSCSI target, 192.168.50.11 is the initiator 11:14:45.391670 IP 192.168.50.8.3260 > 192.168.50.11.58035: P 3143284:3144176(892) ack 193 win 61440 11:14:45.391674 IP 192.168.50.8.3260 > 192.168.50.11.58035: . 3144176:3145636(1460) ack 193 win 61440 11:14:45.391676 IP 192.168.50.11.58035 > 192.168.50.8.3260: . ack 3145636 win 32767 11:14:45.391681 IP 192.168.50.8.3260 > 192.168.50.11.58035: . 3145636:3147096(1460) ack 193 win 61440 11:14:45.391686 IP 192.168.50.8.3260 > 192.168.50.11.58035: P 3147096:3148320(1224) ack 193 win 61440 11:14:45.391689 IP 192.168.50.11.58035 > 192.168.50.8.3260: . ack 3148320 win 32767 11:14:45.391694 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391699 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391702 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391706 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391710 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391713 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391716 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391719 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391722 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391725 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391763 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391766 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.391769 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 193 win 61440 11:14:45.398114 IP 192.168.50.11.58035 > 192.168.50.8.3260: P 193:241(48) ack 3148320 win 32767 11:14:45.398211 IP 192.168.50.8.3260 > 192.168.50.11.58035: . ack 241 win 61440 11:14:45.398636 IP 192.168.50.8.3260 > 192.168.50.11.58035: . 3148320:3149780(1460) ack 241 win 61440 11:14:45.398645 IP 192.168.50.8.3260 > 192.168.50.11.58035: . 3149780:3151240(1460) ack 241 win 61440 11:14:45.398649 IP 192.168.50.11.58035 > 192.168.50.8.3260: . ack 3151240 win 32767 11:14:45.398654 IP 192.168.50.8.3260 > 192.168.50.11.58035: . 3151240:3152464(1224) ack 241 win 61440 11:14:45.398658 IP 192.168.50.8.3260 > 192.168.50.11.58035: . 3152464:3153924(1460) ack 241 win 61440 11:14:45.398662 IP 192.168.50.11.58035 > 192.168.50.8.3260: . ack 3153924 win 32767 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

Reply

Sign in to reply online Use email software

bugzilla_noreply＠novell.com

5 Mar 5 Mar

00:38

New subject: [Bug 366492] file system corruption and scsi errors with open-iscsi and Infortrend iSCSI arrays

https://bugzilla.novell.com/show_bug.cgi?id=366492 User s.pigneguy@niwa.co.nz added comment https://bugzilla.novell.com/show_bug.cgi?id=366492#c3 Stuart Pigneguy changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|s.pigneguy@niwa.co.nz | --- Comment #3 from Stuart Pigneguy 2008-03-04 17:38:06 MST --- Hi, Increasing the retries value didn't help. Should I increase the retries value above 20? The first run of bonnie++ is with the drivers from the open-iscsi website, there were no kernel messages and no error messages in /var/log/messages. The rest of the runs are done with the latest drivers from SuSE using Yast2 online update. They all had kernel errors and they all left error messages in /var/log/messages. How would you like me to proceed? (open-iscsi-2.0-865.15 - 2008-03-05 11:15am - retries=5, no errors) ecweb1:/iscsi/xfs/nobody # bonnie++ -u nobody Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ecweb1 8G 45343 99 94899 27 45229 10 42284 94 93601 9 682.0 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 772 5 +++++ +++ 1023 6 1347 10 +++++ +++ 943 6 ecweb1,8G,45343,99,94899,27,45229,10,42284,94,93601,9,682.0,1,16,772,5,+++++,+++,1023,6,1347,10,+++++,+++,943,6 (open-iscsi-2.0.707-0.32 - 2008-03-05 11:55am - retries=5, kernel errors) ecweb1:/iscsi/xfs/nobody # bonnie++ -u nobody Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ecweb1 8G 44948 98 98479 29 45630 11 8265 18 93404 9 708.0 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 766 5 +++++ +++ 981 6 1261 9 +++++ +++ 915 6 ecweb1,8G,44948,98,98479,29,45630,11,8265,18,93404,9,708.0,1,16,766,5,+++++,+++,981,6,1261,9,+++++,+++,915,6 (open-iscsi-2.0.707-0.32 - 2008-03-05 12:46pm - retries=10, kernel errors) ecweb1:/iscsi/xfs/nobody # bonnie++ -u nobody Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ecweb1 8G 45016 98 116249 33 42722 10 5079 11 93191 8 716.9 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 733 5 +++++ +++ 1029 6 1254 8 +++++ +++ 950 6 ecweb1,8G,45016,98,116249,33,42722,10,5079,11,93191,8,716.9,1,16,733,5,+++++,+++,1029,6,1254,8,+++++,+++,950,6 (open-iscsi-2.0.707-0.32 - 2008-03-05 13:31pm - retries=20, kernel errors) ecweb1:/iscsi/xfs/nobody # bonnie++ -u nobody Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ecweb1 8G 45329 99 98236 28 44994 11 6485 14 94117 9 703.6 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 755 5 +++++ +++ 1018 6 1290 9 +++++ +++ 905 6 ecweb1,8G,45329,99,98236,28,44994,11,6485,14,94117,9,703.6,1,16,755,5,+++++,+++,1018,6,1290,9,+++++,+++,905,6 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

Reply

Sign in to reply online Use email software

bugzilla_noreply＠novell.com

00:46

New subject: [Bug 366492] file system corruption and scsi errors with open-iscsi and Infortrend iSCSI arrays

https://bugzilla.novell.com/show_bug.cgi?id=366492 User s.pigneguy@niwa.co.nz added comment https://bugzilla.novell.com/show_bug.cgi?id=366492#c4 --- Comment #4 from Stuart Pigneguy 2008-03-04 17:46:15 MST --- Created an attachment (id=198646) --> (https://bugzilla.novell.com/attachment.cgi?id=198646) output from dmesg kernel debug messages that occurred during previous run of bonnie++ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

Reply

Sign in to reply online Use email software

bugzilla_noreply＠novell.com

00:46

New subject: [Bug 366492] file system corruption and scsi errors with open-iscsi and Infortrend iSCSI arrays

https://bugzilla.novell.com/show_bug.cgi?id=366492 User s.pigneguy@niwa.co.nz added comment https://bugzilla.novell.com/show_bug.cgi?id=366492#c5 --- Comment #5 from Stuart Pigneguy 2008-03-04 17:46:56 MST --- Created an attachment (id=198648) --> (https://bugzilla.novell.com/attachment.cgi?id=198648) output from /var/log/messages error messages found in /var/log/messages from the previous run of bonnie++ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

Reply

Sign in to reply online Use email software

5906

Age (days ago)

5908

Last active (days ago)

Download

6 comments

1 participants

tags

participants (1)

bugzilla_noreply＠novell.com