https://bugzilla.novell.com/show_bug.cgi?id=390384 User carnold@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c1 Charles Arnold changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |carnold@novell.com, lbendixs@novell.com, | |jbeulich@novell.com Status|NEW |NEEDINFO Info Provider| |kwolf@novell.com QAContact|qa@suse.de |jdouglas@novell.com --- Comment #1 from Charles Arnold 2008-05-21 09:19:12 MST --- Kevin, Can you reproduce this on the latest kernel for openSUSE 11.0 and provide the same kind of trace for Jan. It still may not be xen specific. If we can't reproduce this on a current kernel, we will probably close this bug. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c3 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |kwolf@novell.com --- Comment #3 from Jan Beulich 2008-05-28 04:42:27 MDT --- Where do the "TAPDISK[4223]: EIO. Repeating op = 1, 0x602e5a/8/5" messages originate from? As they in both cases appeared right before the crash, I would suspect that they might be key to understanding what's going on here. As I wasn't able to locate a message like this anywhere, I wonder whether it is from a local patch of yours... Also, what build does kernel 2.6.25.4-6 come from - all I can find are builds with 2.6.25.4-2, and in order to analyze the back trace I need the matching kernel RPM. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User jbeulich@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c5 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|cgriffin@novell.com |kernel-maintainers@forge.provo.novell.com --- Comment #5 from Jan Beulich 2008-05-28 07:15:07 MDT ---

From the little analysis we were able to do this looks like a problem not specific to the Xen kernel. And even if it is, we'd need someone with much better AIO/direct-io knowledge to assist here.

-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User jack@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c6 Jan Kara changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |kwolf@novell.com --- Comment #6 from Jan Kara 2008-05-28 15:03:50 MDT --- Hmm, it looks the oops happened in do_direct_IO() somewhere in zero_user(). Just to verify - your program was doing direct IO on block device with block size 512 bytes. Somehow the page user provided got freed before we got to it (that could possibly be a fault of Xen's tap:aio if it plays some dirty tricks ordinary userspace process couldn't do). So is tap:aio just an ordinary process using AIO/dio or does it do anything special? Regarding those EIO errors before - what filesystem do you use? There's a problem in ext3 which could cause EIO return from direct IO writes when they are combined with buffered writes. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User jack@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c8 --- Comment #8 from Jan Kara 2008-05-29 02:04:01 MDT --- I see. So it's indeed a bug in AIO/DIO code in kernel (userspace process shouldn't be able to oops the kernel). I'll have a look into that. Anyway, for the EIO problem, I'll attach a fix which just went into -mm tree. Can you try it please, whether it fixes the problem for you? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

From what I've understood from reading tap:aio sources do_cow_read() issues a

https://bugzilla.novell.com/show_bug.cgi?id=390384 User jack@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c10 --- Comment #10 from Jan Kara 2008-05-29 04:01:25 MDT --- To decrease some uncertainty I have a few questions: The XEN device tap:aio has been accessing was backed by a regular file, wasn't it? direct IO read and as a buffer provides mmapped region of the same file, correct? I'm trying to reproduce the problem here (by some simple program) and I'm unsuccessful so far... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User kwolf@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c12 Kevin Wolf changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|kwolf@novell.com | --- Comment #12 from Kevin Wolf 2008-05-29 12:24:52 MDT --- Yes, I use a regular file as image. I think the buffer is a page from the ring buffer shared by Dom0 backend and DomU frontend blktap drivers. If I understand correctly the mmap is not on the image file but on /dev/xen/blktapN. I will try if your patch helps with the EIO returns. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User jack@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c14 Jan Kara changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |kwolf@novell.com --- Comment #14 from Jan Kara 2008-06-02 07:37:03 MDT --- OK, thanks for testing. Would it be possible to create a test program that wouldn't use XEN's kernel blktap driver (only regular files) and still trigger the bug (either EIO or Oops)? At least the Oops seems like it could well be caused by the blktap kernel driver because direct-io oopses because the pages it had been provided to do IO to suddently disappear and these are the pages from mmaped blktap driver if I understand the situation correctly. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User jack@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c16 --- Comment #16 from Jan Kara 2008-06-02 08:09:50 MDT --- How have you found out the page is RO (I always like to learn how to find more info from the oops ;)? Anyway yes, what you write would cause the oops as well and it would be a bug of blktap driver. Thanks for pointing that out. Regarding the EIO: EIO from direct-io can be caused by the fact that invalidate_inode_pages2_range() we do before the actual direct io failed for some reason (this call evicts pages of page cache in the area where direct write is going to happen). This is usually caused by the fact that someone still holds references to buffers where we want to write with direct-IO. Also the fact that simply retrying the IO usually helps suggests that this could be the case here. But all the cases we were aware of (and were able to trigger by our testing) should be fixed by the patch I've attached so I'm currently not sure what we can be missing (and therefore I'd like to have a program to reproduce it without the blktap interference). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User kwolf@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c18 --- Comment #18 from Kevin Wolf 2008-06-04 11:25:39 MDT --- I agree that a small test program independent of blktap would be helpful. I haven't succeeded yet in creating one, though. Jan, I don't think blktap would intentionally free the page, after all it's a ring buffer and should be reused. It seems to map DomU blkfront pages read-only for write operations, though. As I'm not too familiar with the kernel code, could you take a look at the blktap kernel module source? And while we're at it, just out of curiosity with respect to oops reading... Is the 0003 in "Oops: 0003 [1] SMP" the error code of the page fault? That's the way I interpreted it and then it would be another hint that the page is read-only. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User kwolf@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c20 --- Comment #20 from Kevin Wolf 2008-06-06 11:26:18 MDT ---

Ah, indeed, I wasn't aware of them mapping pages for write requests readonly. But that shouldn't cause any problems - nothing should try to access these pages other than for reading in that path.

Thinking about it again, this _must_ cause problems in combination with my patch. As I wrote above it issues a read when the write fails - and then of course we are in the wrong code path. So I think we can forget about the oops... Anyway, there is still the question why I'm getting EIO in the first place. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User jack@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c22 --- Comment #22 from Jan Kara 2008-06-18 11:09:48 MDT --- Oo, sorry. I've just looked at the patch I've posted here originaly and it was a patch for ext4! Now wonder it didn't help to ext3... I'll attach the right patch. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User kwolf@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c24 Kevin Wolf changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|kwolf@novell.com | --- Comment #24 from Kevin Wolf 2008-06-19 06:21:56 MDT --- Looks good. I did three test installations and didn't get a single EIO. So chances are high that your patch is a fix for this problem. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=390384 User kwolf@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=390384#c26 Kevin Wolf changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mmeeks@novell.com --- Comment #26 from Kevin Wolf 2008-06-19 06:45:37 MDT --- *** Bug 380514 has been marked as a duplicate of this bug. *** https://bugzilla.novell.com/show_bug.cgi?id=380514 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.