https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c2 --- Comment #2 from Petr Baudis 2011-03-09 14:35:16 UTC --- It seems unlikely so far, especially if getdents*() hangs in the kernel - then there will likely lie the trouble. There have been couple of getdents-related fixes in glibc since 11.3, but they don't seem to be likely to break anything. Let's wait for straces and what the kernel people say. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c11 --- Comment #11 from Neil Brown 2011-03-16 07:22:59 UTC --- Looking at Stefan Dirsch's problem.... The directory contains 87 entries. The NFS client sends READDIRPLUS requests and receives replies containing 31,30, and 26 entries each. This all appears correct. The cookie numbers are all unique. These get stored in the page cache which can hold 102 entries per page, so they all fit in one page. Then the getdents64 call returns all those 87 entries - confirming that the data stored in the page cache looks mostly correct. glibc makes another getdents64 call (to see if there are any more) and NFS returns the last 37 entries again. And every time another getdents64 call is made, the same 37 entries are returned. It never returns 0 entries, so glibc never sees 'eof'. When nfs finishes with a readdir request it stores the current 'cookie' in a data structure attached to the 'struct file', and on the next request it gets that cookie and searches through its table in the page cache to find out where it was up to. So this seems to imply that either the wrong cookie is being stored at the end of one readdir, the wrong cookie is being used to search for the next readdir, or it finds the wrong place when searching for a cookie - or there are duplicate cookies (which I already know there are not). I cannot see how any of these can be happening, and I cannot imagine any source of error that would cause this directory to respond like this very repeatably, yet would allow most other directories to still work perfectly. I'm not sure where to go next with this one... I might have to try to insert some tracing to see what is really happening.. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c13 --- Comment #13 from Neil Brown 2011-03-16 10:37:33 UTC --- Thanks - that turned out to be a very valuable hint. I think I know what the problem is. I've posted a note about it upstream and will look at committing a fix tomorrow. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c15 --- Comment #15 from Neil Brown 2011-03-16 10:53:51 UTC --- They are two different bugs. Stephan's is a subtle decoding error when decoding readdir replies. Yours is that the default has changed to perform READDIRPLUS on large directories, which is significantly slower when you don't want to 'stat' each name (as somewhat faster when you do). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c17 --- Comment #17 from Neil Brown 2011-03-17 05:06:51 UTC --- Created an attachment (id=419854) --> (http://bugzilla.novell.com/attachment.cgi?id=419854) Patch to avoid excessive cost of readdirplus This patch introduces an adaptive choice of READDIRPLUS. Previous kernels only used READDIRPLUS on relatively small directories. 2.6.38 always uses READDIRPLUS which is good for 'ls -l' and bad for 'ls'. With this patch, nfs tries to detect if the extra stat information of READDIRPLUS is being used and if it isn't it doesn't bother fetching it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c19 --- Comment #19 from Andreas Jaeger 2011-03-17 08:43:21 UTC --- Neil, those are in the openSUSE 11.4 kotd as far as I can see. Please submit to master as well once testing was successful. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c21 --- Comment #21 from Stefan Dirsch 2011-03-17 16:32:49 UTC --- (In reply to comment #20)

(In reply to comment #19)

...

Neil, those are in the openSUSE 11.4 kotd as far as I can see.

I don't see these in Kernel:HEAD. The latest RPM changelog entry is:

* Di Mär 15 2011 jeffm@suse.com - Update to 2.6.38. - commit f2ec6ad

Neil, how do the RPM changelog entries will look like?

But with that kernel I can't reproduce the issue any longer! Maybe the patches are already applied? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c23 --- Comment #23 from Andreas Jaeger 2011-03-17 19:56:05 UTC --- Stefan, you need the openSUSE 11.4 branch, see comment #22. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c25 --- Comment #25 from Neil Brown 2011-03-18 22:26:21 UTC --- Thanks for testing. Stefan's issue is fixing 2.6.38 (and hence 'HEAD') largely because the code was rearranged a bit and the rearrangement happened to remove the bug. Reinhard's issue is not fixed in mainline and is still under discussion upstream. If nothing useful comes of the upstream discussion within a week or so I'll commit my patch to HEAD, but I'd rather a more "official" patch if possible. So I'll leave this for a week or so, then commit something to HEAD and close it. Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c27 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #27 from Neil Brown 2011-04-11 07:44:58 UTC --- I have committed the adaptive readdir fix to 'head' so both these bugs should be fixed in suse kernels now. Update is seriously considering this fix, but things move slowly sometimes... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c29 Ruediger Meier changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED CC| |sweet_f_a@gmx.de Resolution|FIXED | --- Comment #29 from Ruediger Meier 2011-05-26 11:36:03 UTC --- I still run into this bug. Actually Ive got a directory where I can reproduce it on client using different kernels >=2.6.37 taken from suse or kernel.org. Server is always 2.6.37.6-0.5-desktop exporting ext4/nfs. In opposite to earlier tests it doesn't matter whether using nfs3/4 tcp/udp/ ipv4/ipv6 rdirplus/nordirplus. rudi@server:/exports/data/gen/fwo> ls |wc -l 261974 # using suse kernel on client: rudi@client:/data/gen/fwo> uname -r 2.6.37.6-0.5-desktop rudi@client:/data/gen/fwo> ls |wc -l ls: memory exhausted 0 strace show these infinite getdents64() calls # Using kernel 2.6.39 vanilla results in: rudi@client:/data/gen/fwo> ls |wc -l ls: reading directory .: Too many levels of symbolic links 3466 BTW1 I also have a directory where suse kernel works but vanila kernel does not. BTW2 If I "cp -a" these dirs within the same underlying filesystem then the copy behaves the same way like the original. But it's not easy to create such "broken" dir on another exported filesystem. I could give more debug info if wanted. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=678123 https://bugzilla.novell.com/show_bug.cgi?id=678123#c31 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution| |UPSTREAM --- Comment #31 from Neil Brown 2011-05-31 01:09:36 UTC --- You bug is quite a different bug to the other two here. The NFS server is not providing unique readdir cookies, so NFS cannot do much useful. And the NFS server doesn't provide them because ext4 doesn't provide them. It might be possible to work around this in the NFS server code but it really needs to get fixed upstream before we can do anything useful in the openSUSE kernel. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.