https://bugzilla.novell.com/show_bug.cgi?id=880599 https://bugzilla.novell.com/show_bug.cgi?id=880599#c1 --- Comment #1 from James Carter 2014-05-29 19:06:10 UTC --- Created an attachment (id=592619) --> (http://bugzilla.novell.com/attachment.cgi?id=592619) Syslog messages when the filesystem remounted readonly Note that the filesystem really is ext3 even though EXT4-fs is reporting the errors and is named in the backtrace. The remount was noticed and the host was rebooted an hour later. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=880599 https://bugzilla.novell.com/show_bug.cgi?id=880599#c2 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nfbrown@suse.com AssignedTo|nfbrown@suse.com |jack@suse.com --- Comment #2 from Neil Brown 2014-06-05 01:38:39 UTC --- Re-assigning to Jack as this seems to be an ext[34]/quotas problem more than an NFS problem. With NFSv3, the client doesn't find out about quota problems until the data is actually sent to the server, and that can be delayed quite a while as writes are cached on the client and sent lazily. If you were to mount with "-o sync" you would get quota errors immediately, but performance would be terrible, so that isn't a good idea. If you reduce /proc/sys/vm/dirty_bytes to a few hundred meg, that would limit the overage to a few hundred meg without hurting performance too much. However that may still be more overage than you would like. How much memory does your client have? I cannot really see why NFSv4 would make a difference. It does have some knowledge of quotas, were NFSv3 has none, but the Linux code doesn't appear to use that at all. I might try some experiments myself to see what happens when you exceed quotas via NFS. However I think we need to start by resolving the ext3/4 issue as that seems to be the main problem. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=880599 https://bugzilla.novell.com/show_bug.cgi?id=880599#c4 --- Comment #4 from Jan Kara 2014-06-05 09:44:14 UTC --- Created an attachment (id=593464) --> (http://bugzilla.novell.com/attachment.cgi?id=593464) [PATCH] ext4: Fix buffer double free in ext4_alloc_branch() This patch should fix the problem with filesystem getting remounted read-only. I'll publish a test kernel for you in a while. And thanks again for your report because I've been scratching my head over another manifestation of this bug for quite some time already :). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=880599 https://bugzilla.novell.com/show_bug.cgi?id=880599#c6 James Carter changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|jimc@math.ucla.edu | --- Comment #6 from James Carter 2014-06-09 20:11:16 UTC --- Sorry to be slow testing the patched kernel, but we had to respond to CVE-2014-0224, and then the issues came up described in bug 881912, which may or may not be related to this one. The production and test servers are set up as i686 (vs. x86_64). I installed the patched iernel (kernel-default-3.11.10-0.x86_64.rpm) on a x86_64 test machine and got different results, plus I re-did my tests on the i686 server and got similar but not identical results. By the way, I'm not concerned if the user succeeds in writing significantly beyond his quota, but it seems like a useful piece of data for analysing the problem, so I report it when it happens. I'm still using "dd if=/dev/zero of=$dir/oinkage.dat bs=1M", executing on the client as the user (mathtest), to try to set off the bug(s). In this table, Kernel = what the server uses (11-0 = Jack's test kernel, 11-10 = distro's updated standard kernel), and NFSv = the NFS version that the client is using. For the first 8 tests, both the servers and the clients were x86_64. For the last 2 the server was i686, clients were x86_64. Kernel NFSv Fsys Outcome 10-0 4 ext4 EDQUOT (good), cut at 750Mb, very little overage 10-11 4 ext4 No signal (bad), had to kill writer 10-0 3 ext4 EDQUOT (good), "quota -v -l mathtest" claims to be using 750000 blocks, but the file is 1180467200 bytes or 1.18Gb. After I ran quotacheck, says 750004 blocks. 10-11 3 ext4 EDQUOT (good), "quota -v -l mathtest" claims to be using 750000 blocks, but the file is 888827904 bytes or 889Mb. After quotacheck, still says 750000 blocks. 10-0 3 ext3 EDQUOT (good), cut off at 1.4Gb, quota says 750Mb 10-11 3 ext3 EDQUOT (good), cut off at 750Mb, very little over Repetitions: ***** 5 all the same exc. overage 10-0 4 ext3 EDQUOT (good), cut off at 750Mb, very little over Repetitions: ***** 5 all the same exc. overage 10-11 3 i686 ext3 EDQUOT (good), cut at 1.1Gb (from 10-11 x86_64) More repetitions: EDQUOT EDQUOT 10-11 3 i686 ext3 EDQUOT (good), cut at 750Mb+ (from 10-0 x86_64) More repetitions: EIO EIO The client that gets EDQUOT is about 1.3x faster than the one getting EIO. All 3 machines are connected by a gigabit hub with little competing traffic, so network speed likely does not affect the results. Conclusion: The originally reported scenario on i686 has not self-healed, but instead of failing "every" time, it fails "most of the time" on the original client but never on a different client. When testing on x86_64 and ext3 I got no failures at all, either on the patched kernel or the standard kernel, but I got one failure (no EDQUOT, no EIO, no signal at all) with the combination of ext4, NFSv4, standard kernel, neither of which had failed before. Odysseus successfully squeezed information out of the god Proteus, and I think we need to learn from him to solve this bug. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=880599 https://bugzilla.novell.com/show_bug.cgi?id=880599#c8 --- Comment #8 from James Carter 2014-06-10 05:49:57 UTC --- Thanks for that explanation; I didn't think of writing blocks out of order. Next time I run the test I'll check the file for sparseness. No fault was seen with Jack's patched kernel (3.11.10-0). But that doesn't answer your question -- I was unhappy to find that my test procedure does not reliably set off the bug on command, but rather has contingencies suggesting a race condition. Next time let's test on kernel-desktop for i686, to use the machine most likely to fail on the standard kernel. And I was also unhappy to discover that it could still mess up even though NFSv4 was used. I thought that would be an intervention that would bypass the bug -- evidently not. Better to discover that in testing rather than in production. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=880599 https://bugzilla.novell.com/show_bug.cgi?id=880599#c10 James Carter changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|jimc@math.ucla.edu | --- Comment #10 from James Carter 2014-06-11 17:15:53 UTC --- Thanks for your work on this. It looks like the patch should solve the problem. But just for paranoia's sake, could you build us a test kernel for i686, because the particular client and i686 server fails almost all the time. If the problem disappears with that combination, I would have more confidence that we had caught "THE shark" rather than just "a shark". I don't like problems which require unknown and uncontrollable timing contingencies to be reproduced. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=880599 https://bugzilla.novell.com/show_bug.cgi?id=880599#c12 --- Comment #12 from Bernhard Wiedemann 2014-06-13 11:01:18 CEST --- This is an autogenerated message for OBS integration: This bug (880599) was mentioned in https://build.opensuse.org/request/show/237091 13.1 / kernel-source -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=880599 https://bugzilla.novell.com/show_bug.cgi?id=880599#c13 James Carter changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED InfoProvider|jimc@math.ucla.edu | --- Comment #13 from James Carter 2014-06-15 01:11:28 UTC --- Curse, curse, my i686 test machine crashed and nobody is on site due to parents attending graduation ceremonies taking up all the parking.... You don't want to know. Anyway, I attempted to do tests on other i686 machines, similar but not identical. I did a lot of trials (dd if=/dev/zero of=/net/$HOST/mathtest/oink.dat bs=1M) with both kernel 3.11.10-11.1 and 3.11.10-0, The NFS client had 3.11.10-11.1 and mounted with Nfsvers=3 in /etc/nfsmount.conf. The target filesystem was ext3 but was actually a 1Gb file attached to a loop device, on both servers, whereas on Ocelot the bare metal was ext3. Hiss, boo. On 3.11.10-11.1, 77 trials. Most got EDQUOT (quota exceeded). Two got EIO (I/O error). 14 got Stale NFS Filehandle, but those shouldn't count because I removed the target file on the server before the client had closed it (even though the server was refusing to write anything because the user was over quota). (I messed up a test script.) I was hoping to get consistent misbehavior, which could then be shown to be cured by the test kernel, but that's not how it worked out. On 3.11.10-0, 12 trials, most got EDQUOT, one got EIO (I/O error). In all EDQUOT outcomes, writing was cut off exactly at the quota limit, although the files were sparse, i.e. the last written block was further along, sometimes quite a bit, i.e. over 1Gb when the quota was 750Mb. So the conclusion has to be, kernel 3.11.10-0 is no worse than 3.11.10-11.1, and given that you fixed a bug, we should move forward with this patch. But I'm afraid we haven't seen the last of the NFS problems. Next week when I can get at the original test machine (Ocelot) I'll re-test, and I'll post something if the results are significantly different from the above. If I hit something in the future and can come up with a procedure to reproduce it, I'll open a new bug. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=880599 https://bugzilla.novell.com/show_bug.cgi?id=880599#c15 --- Comment #15 from Swamp Workflow Management 2014-06-25 07:09:36 UTC --- openSUSE-SU-2014:0840-1: An update that solves 9 vulnerabilities and has 15 fixes is now available. Category: security (important) Bug References: 851338,858067,868315,869563,870173,870576,871561,872715,873374,876102,876981,877257,877713,877721,878115,878274,879258,879792,880599,880613,880892,881697,881727,882648 CVE References: CVE-2013-7339,CVE-2014-0055,CVE-2014-0077,CVE-2014-2678,CVE-2014-2851,CVE-2014-3122,CVE-2014-3144,CVE-2014-3145,CVE-2014-3153 Sources used: openSUSE 13.1 (src): cloop-2.639-11.10.1, crash-7.0.2-2.10.9, hdjmod-1.28-16.10.1, ipset-6.21.1-2.14.1, iscsitarget-1.4.20.3-13.10.1, kernel-docs-3.11.10-17.6, kernel-source-3.11.10-17.2, kernel-syms-3.11.10-17.1, ndiswrapper-1.58-10.1, pcfclock-0.44-258.10.1, vhba-kmp-20130607-2.11.1, virtualbox-4.2.18-2.15.2, xen-4.3.2_01-18.2, xtables-addons-2.3-2.10.1 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.