http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 Stephan Hemeier changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Sauerlandlinux@gmx.de -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 Sebastian Kuhne changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugzilla.opensuse.o | |rg/show_bug.cgi?id=1060159 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c2 Richard Martin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |richard.martin@flyingpeople | |.de --- Comment #2 from Richard Martin --- Are you still working on it or is it already solved and I didn't noticed it? I can't mount the nfs drive from the synology via the entry in the /etc/fstab. I got the error as decribed. Mounting from the terminal works fine. /etc/fstab: FS-Ritchie.fritz.box:/volume1/Daten /home/NFS-Data nfs auto,rw,sync,soft,user,intr,tcp 0 0 Output from journalctl: Feb 10 21:00:06 richardlx systemd[1]: Condition check resulted in RPC security service for NFS server being skipped. Feb 10 21:00:06 richardlx kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Feb 10 21:00:06 richardlx systemd[1]: Condition check resulted in RPC security service for NFS client and server being skipped. Feb 10 21:00:06 richardlx systemd[1]: Reached target NFS client services. Feb 10 21:00:07 richardlx systemd[1]: Starting Notify NFS peers of a restart... Feb 10 21:00:07 richardlx systemd[1]: Started Notify NFS peers of a restart. Feb 10 21:00:07 richardlx systemd[1]: Mounting /home/NFS-Data... Feb 10 21:00:07 richardlx systemd[1]: home-NFS\x2dData.mount: Mount process exited, code=exited, status=32/n/a Feb 10 21:00:07 richardlx systemd[1]: home-NFS\x2dData.mount: Failed with result 'exit-code'. Feb 10 21:00:07 richardlx systemd[1]: Failed to mount /home/NFS-Data. Feb 10 21:00:11 richardlx.fritz.box kernel: NFS: Registering the id_resolver key type Manual mount works fine: sudo mount -t nfs -o soft fs-ritchie.fritz.box:/volume1 /home/NFS-Data/ Output from journalctl: Feb 10 21:02:08 richardlx.fritz.box sudo[3871]: richard : TTY=pts/0 ; PWD=/home/richard ; USER=root ; COMMAND=/usr/bin/mount -t nfs -o soft fs-ritchie.fritz.box:/volume1 /home/NFS-Data/ -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c4 --- Comment #4 from Sebastian Kuhne --- (In reply to Neil Brown from comment #3)

It isn't clear to me what the problem is.

You say the mount option contain

users,noauto

which means that the filesystem should *not* be mounted automatically, but that any root (including non-root) is allows to mount it manually.

You also say: Manual mount works fine:

So what is the problem?

Neil, the report from Richard is not what I have reported at first. My issue still persists, and I was hoping that my request has been acknowledged. I am happy to answer all your questions. Again, I tried first to solve it with help of the forum without success, unfortunately. Now I am here and hope for a solution. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c6 --- Comment #6 from Neil Brown --- Hi Sebastian, Sorry, I didn't notice that the comments were from different people. Still, you say that the mount options conain "noauto", so the filesystem should not be mounted automatically. Can you mount it manually? If not, what error do you get. So I repeat my question: What exactly is the problem? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c8 --- Comment #8 from Sebastian Kuhne --- (In reply to Neil Brown from comment #6)

Hi Sebastian, Sorry, I didn't notice that the comments were from different people.

Still, you say that the mount options conain "noauto", so the filesystem should not be mounted automatically. Can you mount it manually? If not, what error do you get. So I repeat my question: What exactly is the problem?

Hi Neal, no problem at all. However the noauto issue is not *MY* problem - that is Richard's issue. My problem is described at the very beginning of the thread. I can't mount a TW NFS client to a Leap 15.1 server. I have no problem to mount a Leap 15.1 client to the same Leap 15.1 server. So I assume here is a bug. Please refer to the beginning of the thread, and to the openSUSE forum link (which is in German, but outputs of all commands are there, too). Really need help since I assume we have a race condition - sometimes the connection works for whatever reason! Again, the issue is between TW/Leap - it works perfect between Leap/Leap. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c10 --- Comment #10 from Sebastian Kuhne --- (In reply to Neil Brown from comment #9)

Hi Sebastian. I cannot help you unless you tell me what the problem is - and you haven't.

I cannot read German, and seeing the log output doesn't help me unless I know what you are trying to do.

The /etc/fstab entries in the initial description that you wrote mention "noauto", so I have to assume that you do not expect the filesystems to be mounted at boot.

You write:

...

NFS client II: .... ==> NFS directory NOT mounted

So I assume that the directory is not mounted, but I don't know why you think that it should be mounted. Did you try to mount it? If so, what command did you use, and what error messages did you get?

You also write

...

The NFS-Client II (with Tumbleweed) is configured the same way, but mounting is problematic.

but you don't tell me what the problem is. Please Please Please be explicit.

Hi Neil, really sorry for not being specific enough. The following happens: - NFS server Leap running - NFS client TW running The noauto option is well known to me, and I do not expect an automount. BUT, manual mounting doesn't work either: - in Dolphin: Klick on Externel device. Nothing happens for some minutes. Then the error message appears: "mount.nfs: mount system call failed" - sudo mount -a: The command hangs, no message, nothing else - In Yast --> NFSClient: After setup, the following message appears: "Die NFS-Verzeichnisse in /etc/fstab konnten nicht eingehängt werden." (NFS directories couldn't be mounted). HOWEVER, sometimes it works, but mostly is doesn't. That's why I assume that my basic configuration is all right, and that we have a bug here. Also, another Leap client always gets connected to the same server without any issues. I really hope that I am specific enough now. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c13 --- Comment #13 from Richard Martin --- How can I run these commands during boot? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c15 --- Comment #15 from Sebastian Kuhne --- Created attachment 830021 --> http://bugzilla.opensuse.org/attachment.cgi?id=830021&action=edit Bug1160035_Sebastian.tar.gz -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c17 --- Comment #17 from Sebastian Kuhne --- (In reply to Neil Brown from comment #16)

Richard: It would be quite difficult to run them at the right time during boot. So if Sebastian can get useful results, I plan to focus there first.

Sebastian - the nfs.pcap file is empty (24 bytes of header) so no traffic was captured - which is strange, but not impossible. Also, the dmesg file contains no rpc debuging. The string "RPC:" should appear a lot, but there are only 4 standard messages during boot.

Maybe NFS isn't even trying to send and RPC traffic. Please try again with

rpcdebug -m nfs -s all rpcdebug -m rpc -s all

then try to mount the NFS filesystem, the

rpcdebug -m nfs -c all rpcdebug -m rpc -c all

and collect the output of dmesg.

Also, when you try to mount the filesystem, give the "-v" option to mount, and report the output. You don't that in original opensuse-forum.de which was useful, but I'll like to be sure I see the -v output that matches the rpcdebug output.

Hi Neil, hope this is what you need. See attached. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c19 --- Comment #19 from Sebastian Kuhne --- Created attachment 830086 --> http://bugzilla.opensuse.org/attachment.cgi?id=830086&action=edit mount -v -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c21 --- Comment #21 from Sebastian Kuhne --- (In reply to Neil Brown from comment #20)

That's certainly more useful, thanks. Though now that we have the RPC: messages, I really need the tcpdump trace that goes with it.

The dmesg shows the NFS client attempting to connect to the server and very quickly failing. To find the cause of the failure, it might help to see the ICMP messages. So if you could run the experiment while tcpdump -i INTERFACENAME -s 0 -w /tmp/nfs.pcap port 2049 or icmp &

is running, and then provide /tmp/nfs.pcap, that might help. Also collect the rpcdebug info at the same time.

Also, when I said "give the "-v" option to mount" I meant that when you issue the mount command to mount the filesystem, give the -v option as well. So:

mount -v -o options server:/path /path

Something like that.

Hi Neil, attached the tar with some more outputs as requested. I have two observations, especially in terms of the mount command (see attached). The file "mount-command" contains two sessions. However, here are my thoughts: - you see NFS vers 4.2 but I have forced via Yast NFS 4.0 (see image) - you see two sessions (try 1 and try 2), both start with IPv6 and then with IPv4. Immediately after IPv4 was active the connection was established, and NFS access was possible. So, we may have two issues: NFS version 4.2 vs 4.0, and IPv6 vs. IPv4. On the server and client, IPv6 is activated via Yast. Hope this helps. - -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c23 --- Comment #23 from Neil Brown --- Thanks for the new data. The screenshot of yast shows you setting the NFS version as "nfsvers=4". This does *not* mean 4.0, it means the highest 4.x which is available. If you really want 4.0, you need to us "nfsvers=4.0" As you say, the "mount -v" output does strongly suggest that IPv6 isn't working and IPv4 is. I wonder if this could be a firewall problem. Can you try disabling any filewall on either client or server and try again? It still might help to get a network trace taken during a failed mount attempt. tcpdump -s 0 -w /tmp/nfs.pcap port 2049 or icmp & Richard: are you using IPv6 at all? Is it possible that IPv6 attempts fail for you, but IPv4 attempt work? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c25 --- Comment #25 from Neil Brown ---

Wondering... was there an update?

The last update to nfs-utils would have been Dec 2019. I don't follow exactly when things enter Tumbleweed, but it would have been after 24nov2019, but not very much after. Maybe 2 or 3 weeks I guess. The previous update would have been after 30sep2019. The kernel gets updated quite regularly, but I don't think there have been any interesting changes in kernel NFS recently. It is very surprising that you data would have appeared in /home/NFS-Data/Daten on the client, unless it was in /volume1/Daten/Daten on the server. The new behaviour you describe is definitely the correct behaviour. I cannot think how that earlier behaviour could possible happen. (BTW I notice that you are using the 'soft' mount option. Our standard recommendation is never to use that. It can lead to silent data corruption. A new mount option - softreval - is coming for Linux 5.6, which might be a better choice, depending on what your actual need is) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c27 --- Comment #27 from Sebastian Kuhne --- Sorry, the tcpdump is too large for submitting (131 MiB). Any hint what I should look for in the dump file? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c29 --- Comment #29 from Sebastian Kuhne --- (In reply to Neil Brown from comment #28)

Is that 131MiB compressed? The file should compress quite well.

I really just need to see a single connection attempt. The client should send a SYN on port 2049, the server replies with SYN-ACK then the client sends ACK. Possible failure modes might be: - no reply at all from the server - RST from the server after the SYN, or after the ACK I'd also link to see any ICMP that come back. They might say port unreachable, or host unreachable or maybe somthing else.

I probably only needs to see a few dozen packets, starting when the client sends SYN on port 2049. There is a program 'editcap' in the wireshark package which can be used to select a range of packets from a larger pcap. Maybe that will help.

I did the exercise for 10 seconds by interrupting the mount command. The dump file is now smaller of course. Please find attached. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c31 --- Comment #31 from Neil Brown --- Thanks for the shortened log, it was plenty. The NFS client successfully connects to the server (SYN, ACK, SYN/ACK) then end send an NFSv4 NULL request and gets a TCP ack, confirming that the server received the request. It should reply with NULL reply, but instead it closes the connection. There are very few circumstances where a Leap 15.1 server will close the connection in response to a well formed NULL request. I think I can only find one - memory allocation failure. And that would affect IPv4 the same as IPv6. So I need some trace information from the server. So on the server run rpcdebug -m rpc -s all then try to mount the NFS filesystem from the client using IPv6, so that it fails. Then on the server again: rpcdebug -m rpc -c all dmesg > /tmp/dmesg and attach the (compressed) dmesg. Hopefully that will tell me something. I think we are getting closer, but it is still very strange. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c33 --- Comment #33 from Neil Brown --- Sorry for the delay - I suddenly got busy :-( The log shows the server getting error -95: EOPNOTSUPP This is coming from kernel_sendpage() I don't know what would be causing that. I can think of two things to try. 1/ What does ip6table -L report. I know you said you disabled the firewell, this will help me confirm that it really is disabled. 2/ What does ethtool -k INTERFACE report for your network interface. Give the interface name, not INTERFACE of course. For me, that is "ethtool -k enp4s0" I'm particularly interested in any transmit-offload features that might be on. It would be worth using "ethtool -K INTERFACE FEATURE off" to turn each one off, then see if that makes a difference. If that doesn't get you anywhere, I'll have to ask a networking specialist for help. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c35 --- Comment #35 from Neil Brown ---

Does it mean the rpcdebug information is now useless for you?

Not necessarily. The problem probably isn't being caused by a firewall, but I'm deeply suspicious and like to triple check anything that seems even close to where the problem might be. I wouldn't object to new tpcdebug trace with firewall turned of, but I doubt it will show anything. If you just turn off the firewall and run "ip6tables -L" that is probably sufficient.

Where to run (on server or on client or both)?

Server. The rpcdebug trace point to a problem on the server. This doesn't entirely gel with your report that it started occurring after you upgraded the client, but it is where the evidence is currently pointing, and that is what I like to follow.

ip6table -L is not a known command

That is because I cannot type. There is an 's' at the end. /usr/sbin/ip6tables is part of the "iptables" package. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c37 --- Comment #37 from Sebastian Kuhne --- Created attachment 831642 --> http://bugzilla.opensuse.org/attachment.cgi?id=831642&action=edit ethtool_OnServerLeap151_withoutFirewall_20200301 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c39 --- Comment #39 from Sebastian Kuhne --- Created attachment 831644 --> http://bugzilla.opensuse.org/attachment.cgi?id=831644&action=edit ethtool_OnClientTW_withoutFirewall_20200301 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c41 --- Comment #41 from Sebastian Kuhne --- Hi Neil, I have attached five output files. All commands with firewall off on both, server and client. --> ip6table -L and ethtool -k INTERFACE on both, server and client. --> I also repeated the tpcdebug trace on the server with firewall off. The naming of the files should be self explaining. Hope this all is of further help. Seems to be a tricky situation. Many thanks for helping!!! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c44 --- Comment #44 from Sebastian Kuhne --- Hi Neil and Michael, many thanks for the new ideas. 30 minutes ago, I did an update for both, the Leap 15.1 server (zypper up) and the TW client (zypper dup). Unfortunately the issue still persists. BUT: The message on the client is different now: linux-sebastian:/home/sebastian # mount -v linux-multimedia.local:/home/multimedia /mnt/linux-multimedia/home/multimedia/ mount.nfs: timeout set for Mon Mar 2 20:34:14 2020 mount.nfs: trying text-based options 'vers=4.2,addr=2a02:810a:8340:2dc0:6c7a:9044:c416:1954,clientaddr=2a02:810a:8340:2dc0::5' mount.nfs: mount(2): Connection reset by peer mount.nfs: Connection reset by peer linux-sebastian:/home/sebastian # I never have seen the "reset by peer" before. This is new after the updates. Is this giving any hint to you? Just as information, the server is an Intel processor based system whereas the client is an AMD based system. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1160035 http://bugzilla.opensuse.org/show_bug.cgi?id=1160035#c47 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #47 from Neil Brown --- Good news. It does seem like your problem is fully resolved, and Richard's disappeared a while ago, so I'll close this now. -- You are receiving this mail because: You are on the CC list for the bug.