https://bugzilla.novell.com/show_bug.cgi?id=462269 User jnelson-suse@jamponi.net added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c1 --- Comment #1 from Jon Nelson 2008-12-24 07:03:09 MST --- The following is an strace taken from a machine running 2.6.25.18, executing the 'dd if=/dev/nbd0 of=/dev/null' command. The server was rebooted with 'reboot -fn' very shortly before this portion of the strace: 07:57:56.568319 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512 <0.000180> 07:57:56.568648 read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512 <0.000185> 07:57:56.568986 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512 <0.000144> 07:57:56.569268 read(0, ^^^ this is where it hangs. This is expected. 0x8059000, 512) = -1 EIO (Input/output error) <161.315346> ^^ it hung for approx. 161 seconds. I was expecting something more like 30 seconds (I used timeout=30 in the setup command) but 161 seconds is better than "forever" ;-) 08:00:37.884925 open("/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000044> 08:00:37.885143 open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000034> 08:00:37.885292 open("/usr/share/locale-bundle/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000034> 08:00:37.885440 open("/usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000032> -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 Jiri Kosina changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|kernel-maintainers@forge.provo.novell.com |pavel@novell.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 User jnelson-suse@jamponi.net added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c4 --- Comment #4 from Jon Nelson 2009-01-17 16:53:56 MST --- Awesome. i've seen some of the patches already. Would you prefer to deal with it here or via the kernel bugzilla entry? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 User devzero@web.de added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c6 roland kletzing changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |devzero@web.de --- Comment #6 from roland kletzing 2009-01-31 03:15:07 MST --- just adding this as a reference : http://bugzilla.kernel.org/show_bug.cgi?id=12277 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 User paul.clements@steeleye.com added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c8 Paul Clements changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High CC| |paul.clements@steeleye.com --- Comment #8 from Paul Clements 2009-02-09 12:00:02 MST --- I too am seeing this on SLES 11 RC3 (2.6.27.13-1.1). I believe the patch I just posted to LKML will fix this. Jon, can you verify? Any chance we can get this into SLES 11 before GA? Patch (with description) attached. The patch has been tested against SLES 11 RC3. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 User jnelson-suse@jamponi.net added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c10 --- Comment #10 from Jon Nelson 2009-02-09 14:28:41 MST --- Correction - the patch only half works: the device is *not* seen as unusable (I/O to the device hangs, forever) and is *not* interruptible. If the server becomes available again, *then* things work out OK but during that time all I/O to the device appears to hang. The 'timeout' parameter does not seem to help. Mine is set at 30 and it's been way more than 30 seconds since the machine with the nbd-server went away. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 User jnelson-suse@jamponi.net added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c12 --- Comment #12 from Jon Nelson 2009-02-09 16:58:48 MST --- I used those flags but got no additional logging. Reads do appear to un-hang if *enough* writes are issued. I had to write quite a bit of data before nbd "noticed" that the server had gone away. I would estimate it at 120-240 MB (some multiple of 40MB), perhaps substantially more. At which point I got this: Feb 9 17:53:51 turnip kernel: nbd: killing hung xmit (nbd0, pid: 8477) Feb 9 17:53:51 turnip kernel: nbd (pid 8477: nbd0) got signal 9 Feb 9 17:53:51 turnip kernel: nbd0: shutting down socket Feb 9 17:53:51 turnip kernel: nbd0: Send data failed (result -4) Feb 9 17:53:51 turnip kernel: nbd0: Request send failed Feb 9 17:53:51 turnip kernel: end_request: I/O error, dev nbd0, sector 12392 Feb 9 17:53:51 turnip kernel: __ratelimit: 250 callbacks suppressed Feb 9 17:53:51 turnip kernel: Buffer I/O error on device nbd0, logical block 6196 Feb 9 17:53:51 turnip kernel: lost page write due to I/O error on nbd0 Feb 9 17:53:51 turnip kernel: Buffer I/O error on device nbd0, logical block 6197 Feb 9 17:53:51 turnip kernel: lost page write due to I/O error on nbd0 Feb 9 17:53:51 turnip kernel: Buffer I/O error on device nbd0, logical block 6198 Feb 9 17:53:51 turnip kernel: lost page write due to I/O error on nbd0 Feb 9 17:53:51 turnip kernel: Buffer I/O error on device nbd0, logical block 6199 Feb 9 17:53:51 turnip kernel: lost page write due to I/O error on nbd0 Feb 9 17:53:51 turnip kernel: Buffer I/O error on device nbd0, logical block 6200 Feb 9 17:53:51 turnip kernel: lost page write due to I/O error on nbd0 Feb 9 17:53:51 turnip kernel: Buffer I/O error on device nbd0, logical block 6201 Feb 9 17:53:51 turnip kernel: lost page write due to I/O error on nbd0 Feb 9 17:53:51 turnip kernel: Buffer I/O error on device nbd0, logical block 6202 and so on, eventually finishing with: Feb 9 17:53:51 turnip kernel: end_request: I/O error, dev nbd0, sector 9082 Feb 9 17:53:51 turnip kernel: end_request: I/O error, dev nbd0, sector 8828 Feb 9 17:53:51 turnip kernel: end_request: I/O error, dev nbd0, sector 8826 Feb 9 17:53:51 turnip kernel: end_request: I/O error, dev nbd0, sector 8572 Feb 9 17:53:51 turnip kernel: end_request: I/O error, dev nbd0, sector 8318 Feb 9 17:53:51 turnip kernel: end_request: I/O error, dev nbd0, sector 8064 Feb 9 17:53:51 turnip kernel: nbd0: queue cleared -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 User jnelson-suse@jamponi.net added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c14 --- Comment #14 from Jon Nelson 2009-02-09 20:20:45 MST --- You bet. Yeah, I did the /etc/modules.d/nbd stuff correctly, but (stupidly) used insmod instead of modprobe (after rmmod, of course). D'oh! The timeout does indeed occur after 15 minutes. Sweet. I knew about synchronous writes triggering it, but I guess I expected a much smaller 'buffer' from nbd. It clearly appears to work, however. I know Pavel Machek is working on-and-off on improving the locking in nbd as well. I have a few performance tweaks for the userland stuff (on the order of about 15-20% under heavy load). Interested? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 User pavel@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c16 --- Comment #16 from Pavel Machek 2009-02-10 02:52:23 MST --- Can you just submit fix to Linus, and cc greg so that it gets into stable? If Linus & Greg take that, it is likely to go into suse trees, too... (For the record, patch looks ok to me.) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 User gregkh@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c17 Greg Kroah-Hartman changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gregkh@novell.com --- Comment #17 from Greg Kroah-Hartman 2009-02-12 20:13:16 MST --- Pavel, are you going to check this patch into our kernel tree? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=462269 User paul.clements@steeleye.com added comment https://bugzilla.novell.com/show_bug.cgi?id=462269#c19 --- Comment #19 from Paul Clements 2009-02-13 06:32:37 MST --- Greg, it's very important for us (SteelEye) to get this fix into the upcoming SLES 11 release. Please let me know if there's anything additional you need from me. Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.