http://bugzilla.suse.com/show_bug.cgi?id=934202 --- Comment #2 from Neil Brown --- Thanks for the trace. Can you tell me how it corresponds to the symptom? Did you start the trace before there was a problem or only after? The trace seems to run for about 4 hours (14585 seconds). from 07:27:30 to 11:30:35, though I'm not sure what timezone. When in that time did the problem occur? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=934202 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CONFIRMED --- Comment #4 from Neil Brown --- No, capturing much after the error (more than a few minutes) wouldn't be likely to help. I just need to be sure that I know what I'm looking at - I don't like to assume. I didn't realise at first, but this seems to be very different symptoms than the original problem description. The error message is certainly different and I strong suspect the NN.NN.NN.NN-man process didn't go into a spin - is that correct? There are a few odd things in the trace, but I think the first odd thing comes from the client. In frame 67293 the client tries to close a file which it opened in frame 28511- the file is called "places.sqlite". Then immediately after a successful response from the server (well... 600usec later), the client sends another close request for the same file. The server accepts this request (which is a little odd, but I'm not certain it is wrong) but the next sequenced request (OPEN CLOSE LOCK LOCKU are all 'sequenced' requests and have sequence numbers) gets an error: BAD_SEQID. This is the CLOSE in frame 67385. This close is correctly sequenced so the server shouldn't complain, but presumably it got confused by the earlier double close. I cannot see how the Linux client could be sending a double close like that. I guess I need more data.... Could you (or anyone who can reproduce this) please collect both the tcpdump trace and nfs debugging with "rpcdebug -m nfs -s all". Then when a problem occurs, provide: - the tcpdump log - the kernel logs - any notes on what you noticed, including whether any processes were spinning. - if processes were spinning, a few copies of /proc/PID/stack for those processes. - exact kernel version. Hopefully I will be able to pull all that together. Thanks. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=934202 http://bugzilla.suse.com/show_bug.cgi?id=934202#c6 --- Comment #6 from Neil Brown --- Yes, it does help - thanks. In frame 15164 the client opens a file called "agent-medusa" for read. The server signal success and returns a "read delegation" as well (15165). The client then reads from the file and closes it, still holding the delegation. The client then checks if "agent-medusa-ssh" exists, which it doesn't. So the client wants to hard-link the agent-medusa file to the agent-medusa-ssh name. It knows that doing that will recall the delegation so it pro-actively starts returning the delegation. This involves looking through it's list of open files to see if one was opened, without telling the server, on the strength of the delegation. It seems that it finds one, because (frame 15172) it opens the file again using the delegation, but opens it write-only!! It shouldn't have a write-only file from a read-only delegation. This causes the error, as it should. The LINK then happens, the server recalls the delegation as expected, and the LINK completes. So now I have to find out why the client thinks it has a write-only open on that file.... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=934202 http://bugzilla.suse.com/show_bug.cgi?id=934202#c9 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(mpluskal@suse.com | |) --- Comment #9 from Neil Brown --- Thanks, there are extra hints in there - particularly the fact that it tries to open with O_WRONLY|_O_CREAT|O_EXCL first. I suspect that is where the strange O_WRONLY open comes from - it shouldn't still be around, but somehow it is. Can you try removing the agent-medusa file, enabling NFS debugging (rpcdebug -m nfs -s all), start a tcpdump capture and an strace capture, and then log in (or whatever triggered that last trace). I want to see exactly how the Netapp responds to the O_CREAT|O_EXCL sequence and whatever follows when it succeeds. I'm guessing that something happens in that case which confuses the client, and then subsequent attempts are handled badly because of that confusion. Thanks. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=934202 http://bugzilla.suse.com/show_bug.cgi?id=934202#c13 --- Comment #13 from Neil Brown --- Created attachment 638635 --> http://bugzilla.suse.com/attachment.cgi?id=638635&action=edit Patch that should be worth testing. I think this patch is worth testing. I'm not sure how best to build a kernel RPM (I hardly ever do that), but I'll try to sort something out tomorrow unless someone jumps in and does that for me :-) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=934202 http://bugzilla.suse.com/show_bug.cgi?id=934202#c15 --- Comment #15 from Neil Brown --- If you haven't had a chance to find a kernel with the fix yet: http://milos.suse.de/kerneltest/suse/3fdbd1e/ has an x86_64 3.16 based kernel, and http://milos.suse.de/kerneltest/suse/bab8328/ has a 4.1 based kernel, both with that fix. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=934202 http://bugzilla.suse.com/show_bug.cgi?id=934202#c17 --- Comment #17 from Nathan Cutler ---

As for "state manager process spinning" I am not sure if I am able to help as I am not using nfs home (instead having it mounted as /mnt/home) and using it mainly as backup/easy way to publish stuff on users.suse.cz - and I have not encountered issues which mcaj described - I mean I would like to help but some reliable way to trigger issue would help.

I am the one who encountered the spinning state manager process and associated lockups while on NFS home. Since this was on my primary workstation and the lockups were troublesome (to say the least), I had to migrate to local home to be able to work. The bug was easily reproducible (in fact, unavoidable) on my setup, which was: * openSUSE 13.2 * NFS home against nfs.suse.cz (Netapp) * KDE4 * Thunderbird, Pidgin, Firefox I never did narrow down the trigger, but I *suspect* it was clicking on URLs in Thunderbird and Pidgin, so they open in Firefox. At some point (1-4 hours of normal usage after reboot) I would click on a link only to find that the system had become unresponsive because writes to /home were not completing. When this happened, a process called 10.100.2.89-man was visible in 'top' and the load average would rise due to the "spinning". The following message would appear in /var/log/messages:

NFS: nfs4_handle_delegation_recall_error: unhandled error -13.

(note, not -10026) Martin Caj also has openSUSE 13.2, NFS home, and KDE4, but uses a different MUA and a different IRC client, and AFAIK he could not reproduce -- hence my suspicion that the issue is related to Thunderbird+Pidging+Firefox interaction. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=934202 http://bugzilla.suse.com/show_bug.cgi?id=934202#c19 --- Comment #19 from Neil Brown --- (In reply to Nathan Cutler from comment #18)

I use VirtualBox, and it is upset that it can't find an appropriate version of the virtualbox-host-kmp-default package to go with the kernel.

That's unfortunate. I don't know how to build kmps. The only think I can think of is to wait for the new kernel to get into factory - howfully that will automatically include the right kmps. Maybe you could find matching kernel in: http://download.opensuse.org/repositories/Kernel:/openSUSE-13.2/standard/x86... and virtualbox kmp in: http://download.opensuse.org/repositories/Virtualization/openSUSE_13.2/x86_6... -- You are receiving this mail because: You are on the CC list for the bug.