[Bug 1167079] New: /etc/fstab automonts stop working after suspend
http://bugzilla.suse.com/show_bug.cgi?id=1167079 Bug ID: 1167079 Summary: /etc/fstab automonts stop working after suspend Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem Assignee: systemd-maintainers@suse.de Reporter: jslaby@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- This used to work. So this is regression either of systemd or kernel. I have this in my /etc/fstab: anemoi:/mnt/c/nvr /mnt/nvr nfs4 x-systemd.automount,x-systemd.idle-timeout=60 0 0 After resume: # systemctl status mnt-nvr.automount ● mnt-nvr.automount Loaded: loaded (/etc/fstab; generated) Active: failed (Result: resources) since Thu 2020-03-19 07:41:16 CET; 3h 26min ago Triggers: ● mnt-nvr.mount Where: /mnt/nvr Docs: man:fstab(5) man:systemd-fstab-generator(8) bře 18 09:16:59 anemoi2 systemd[1]: Set up automount mnt-nvr.automount. bře 18 09:17:02 anemoi2 systemd[1]: mnt-nvr.automount: Got automount request for /mnt/nvr, triggered by 22278 (ls) bře 18 09:54:58 anemoi2 systemd[1]: mnt-nvr.automount: Got automount request for /mnt/nvr, triggered by 23308 (ls) bře 18 10:01:39 anemoi2 systemd[1]: mnt-nvr.automount: Got automount request for /mnt/nvr, triggered by 23544 (ls) bře 18 10:21:42 anemoi2 systemd[1]: mnt-nvr.automount: Got automount request for /mnt/nvr, triggered by 23998 (ls) bře 19 07:41:16 anemoi2 systemd[1]: mnt-nvr.automount: Got invalid poll event 16 on pipe (fd=74) bře 19 07:41:16 anemoi2 systemd[1]: mnt-nvr.automount: Failed with result 'resources'. If that 16 is POLL* value, then 16 is POLLHUP. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c1 Franck Bui <fbui@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fbui@suse.com --- Comment #1 from Franck Bui <fbui@suse.com> --- (In reply to Jiri Slaby from comment #0)
This used to work. So this is regression either of systemd or kernel.
Then why not trying to downgrade one of the packages and see which one caused the regression ? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c2 Franck Bui <fbui@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jslaby@suse.com Flags| |needinfo?(jslaby@suse.com) --- Comment #2 from Franck Bui <fbui@suse.com> --- (In reply to Jiri Slaby from comment #0)
This used to work. So this is regression either of systemd or kernel.
Can you please figure out which package caused the regression ? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c3 --- Comment #3 from Jiri Slaby <jslaby@suse.com> --- I am slow at rebooting the machine as I have zillion of windows open. But reverting the kernel didn't help. Where can I take older systemd? BTW I updated the nfs server on: 2020-03-11 08:04:04|install|nfs-kernel-server|2.4.3-12.1| the client: 2020-03-02 07:01:04|install|nfs-client|2.4.3-12.1| Could it be also the nfs server update? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c4 --- Comment #4 from Franck Bui <fbui@suse.com> --- (In reply to Jiri Slaby from comment #3)
I am slow at rebooting the machine as I have zillion of windows open. But reverting the kernel didn't help. Where can I take older systemd?
You can try to find old packages at http://download.opensuse.org/history/
BTW I updated the nfs server on: 2020-03-11 08:04:04|install|nfs-kernel-server|2.4.3-12.1|
the client: 2020-03-02 07:01:04|install|nfs-client|2.4.3-12.1|
Could it be also the nfs server update?
Yes I would definitively give it a try too. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c5 --- Comment #5 from Jiri Slaby <jslaby@suse.com> --- I have also suse's disks in fstab:
pobox.suse.cz:/home/exports /mnt/suse-home-cz nfs4 x-systemd.automount,x-systemd.idle-timeout=60,nolock,nosuid 0 0
And it behaves exactly the same, so it must be on the client side. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c6 --- Comment #6 from Jiri Slaby <jslaby@suse.com> --- If I restart *any* of the automount services, all of them start working again. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c7 --- Comment #7 from Franck Bui <fbui@suse.com> --- So did you try to downgrade the kernel used by the client ? Can you interpret the "POLLHUP" event returned by the kernel and see in which condition the kernel returns this event rather than "POLLIN" ? This value was not expected by systemd hence the failure. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c8 --- Comment #8 from Jiri Slaby <jslaby@suse.com> --- (In reply to Franck Bui from comment #7)
So did you try to downgrade the kernel used by the client ?
Not yet :/.
Can you interpret the "POLLHUP" event returned by the kernel and see in which condition the kernel returns this event rather than "POLLIN" ?
This value was not expected by systemd hence the failure.
Looking into the systemd code, perhaps I am missing something really obvious, or the code is completely broken :). automount_enter_waiting creates a pipe and then closes the write end: if (pipe2(p, O_CLOEXEC) < 0) { ... p[1] = safe_close(p[1]); Then, it uses the read end for polling: r = sd_event_add_io(UNIT(a)->manager->event, &a->pipe_event_source, p[0], EPOLLIN, automount_dispatch_io, a); If I am not mistaken, this is exactly the problem as automount_dispatch_io checks revents and prints the error if it is not POLLIN: log_unit_error(UNIT(a), "Got invalid poll event %"PRIu32" on pipe (fd=%d)", events, fd); Obviously, one cannot poll read end of the pipe and expect to receive POLLIN in this situation. POLLHUP is what it obviously and correctly gets. I am not sure what changed recently so that this started happenning. No matter what, the code looks broken to me. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c9 --- Comment #9 from Jiri Slaby <jslaby@suse.com> --- Also note that there are quite few patches (some very intrusive) to pipe which landed after 5.4: 0dd1e3773ae8 pipe: fix empty pipe check in pipe_write() d1c6a2aa02af pipe: simplify signal handling in pipe_read() and add comments 85190d15f4ea pipe: don't use 'pipe_wait() for basic pipe IO a28c8b9db8a1 pipe: remove 'waiting_writers' merging logic f467a6a66419 pipe: fix and clarify pipe read wakeup logic 1b6b26ae7053 pipe: fix and clarify pipe write wakeup logic ad910e36da4c pipe: fix poll/select race introduced by the pipe rework da73fcd8cfdc Merge branch 'pipe-rework' (patches from David Howells) 8f868d68d335 pipe: Fix missing mask update after pipe_wait() 8c7b8c34ae95 pipe: Remove assertion from pipe_poll() 6a965666b7e7 Merge tag 'notifications-pipe-prep-20191115' of git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs d8e464ecc17b vfs: mark pipes and sockets as stream-like file descriptors 3c0edea9b29f pipe: Remove sync on wake_ups cefa80ced57a pipe: Increase the writer-wakeup threshold to reduce context-switch count 8df441294dd3 pipe: Check for ring full inside of the spinlock in pipe_write() 7e25a73f1a52 pipe: Remove redundant wakeup from pipe_write() a194dfe6e6f6 pipe: Rearrange sequence in pipe_write() to preallocate slot 8446487feba9 pipe: Conditionalise wakeup in pipe_read() b667b8673443 pipe: Advance tail pointer inside of wait spinlock in pipe_read() 6718b6f855a0 pipe: Allow pipes to have kernel-reserved slots 8cefc107ca54 pipe: Use head and tail pointers for the ring, not cursor and length -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c10 --- Comment #10 from Franck Bui <fbui@suse.com> --- (In reply to Jiri Slaby from comment #8)
Then, it uses the read end for polling: r = sd_event_add_io(UNIT(a)->manager->event, &a->pipe_event_source, p[0], EPOLLIN, automount_dispatch_io, a);
It basically adds a new event EPOLLIN associated to the read end of the pipe to the list of events we're interested in. EPOLLHUP is also implicitly added BTW.
If I am not mistaken, this is exactly the problem as automount_dispatch_io checks revents and prints the error if it is not POLLIN: log_unit_error(UNIT(a), "Got invalid poll event %"PRIu32" on pipe (fd=%d)", events, fd);
Obviously, one cannot poll read end of the pipe and expect to receive POLLIN in this situation. POLLHUP is what it obviously and correctly gets.
Why not ? EPOLLIN means that the kernel sent some data in the pipe and the data are ready to be read on the other end of the pipe. That's what is happening when the kernel informs systemd that it should proceed by mounting the fs. OTOH EPOLLHUP is supposed to mean (in my understanding) that the kernel closed the write end of the pipe for some reasons. Apparently it does that in your case only since recently after a resume. This new behavior is not expected by systemd and it would be interesting to understand why the kernel does that now. Or maybe I just missed your point ;) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c11 --- Comment #11 from Franck Bui <fbui@suse.com> --- (In reply to Jiri Slaby from comment #9)
Also note that there are quite few patches (some very intrusive) to pipe which landed after 5.4:
Then I would definitively give an older kernel without these commits a try. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c18 Jiri Slaby <jslaby@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|systemd-maintainers@suse.de |werner@suse.com --- Comment #18 from Jiri Slaby <jslaby@suse.com> --- (In reply to Franck Bui from comment #17)
Can you try to dump the processes list right before/after suspending ?
To do that you can put an executable script in /usr/lib/systemd/system-sleep that will be called by systemd, see man systemd-sleep.
Nothing useful, so I dumped processes from all parents of the current process in autofs_kill_sb and look: [ 176.729788] autofs4:pid:7690:autofs_catatonic_mode: entering catatonic mode [ 176.729789] autofs4:pid:7690:autofs_kill_sb: shutting down [ 176.729790] autofs4:pid:7690:autofs_kill_sb: comm=umount [ 176.729791] autofs4:pid:7690:autofs_kill_sb: comm=nfs [ 176.729792] autofs4:pid:7690:autofs_kill_sb: comm=nfs [ 176.729793] autofs4:pid:7690:autofs_kill_sb: comm=nm-dispatcher [ 176.729794] autofs4:pid:7690:autofs_kill_sb: comm=systemd That would be /etc/NetworkManager/dispatcher.d/nfs script from NetworkManager-1.22.10-2.1.x86_64, I suppose. And I updated NM from 1.22.8-1.1 to 1.22.10-1.1 on 17th of Mar (and reported this on 19th):
2020-02-11 07:36:11|install|NetworkManager|1.22.6-1.2|x86_64||repo-oss|a05d44ae56a7d238d116c7ec7f3df9effc1c068ea73af9aa4e04c0178e7660e1| 2020-03-02 07:01:24|install|NetworkManager|1.22.8-1.1|x86_64||repo-oss|1e3014b7156879be49b1fecc733fbd2d4efa7df99ac1deb655c59ccbcf5d859f| 2020-03-17 11:30:04|install|NetworkManager|1.22.10-1.1|x86_64||repo-oss|67ce82236bcffc7d1ae4cd7d5a2cb7d62e7911f76bdabf4d01a80a52989ebc49| 2020-05-12 07:42:06|install|NetworkManager|1.22.10-2.1|x86_64||repo-oss|06dcdee32b57f7801a84a672e23d33e5f062b8e344175b3765feadf960d85b20|
That would be SR#784407. And it says:
- Modify nfs script (boo#1164642) * Also mount nfs4 shares * Ignore nfs or nfs4 shares in case if the noauto option is set
And it does:
- # Only unmount it when the type is nfs - if [ "$FS_TYPE" == "nfs" ]; then + # Only unmount it when the type is nfs or nfs4 + if [ "$FS_TYPE" == "nfs" -o "$FS_TYPE" == "nfs4" ]; then
Oh man! -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c19 --- Comment #19 from Jiri Slaby <jslaby@suse.com> --- (In reply to Jiri Slaby from comment #18)
Nothing useful, so I dumped processes from all parents of the current process in autofs_kill_sb and look:
It should write: Nothing useful, so I dumped all parents of the current process in autofs_kill_sb and look: -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c20 --- Comment #20 from Franck Bui <fbui@suse.com> --- (In reply to Jiri Slaby from comment #18)
That would be /etc/NetworkManager/dispatcher.d/nfs script from NetworkManager-1.22.10-2.1.x86_64, I suppose.
Arf yeah... the script doesn't even try to check whether the directory is mounted with nfs. So in your case it's not but autofs is mounted instead (because automount option was used). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c21 --- Comment #21 from Franck Bui <fbui@suse.com> --- (In reply to Jiri Slaby from comment #18)
And it does:
- # Only unmount it when the type is nfs - if [ "$FS_TYPE" == "nfs" ]; then + # Only unmount it when the type is nfs or nfs4 + if [ "$FS_TYPE" == "nfs" -o "$FS_TYPE" == "nfs4" ]; then
Well checking for "nfs4" too triggered the issue on your system but I think it's the whole logic which is actually borked. Rather than looking at fstab to get the list of all possible nfs mounts, at least it would seem better to get the list of nfs shares actually mounted before suspending the system. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c22 --- Comment #22 from Franck Bui <fbui@suse.com> --- Also I'm wondering if this script is needed at all in case the automount option is used with all nfs entries in fstab... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 http://bugzilla.suse.com/show_bug.cgi?id=1167079#c26 --- Comment #26 from Jiri Slaby <jslaby@suse.com> --- (In reply to Dr. Werner Fink from comment #23)
Oh man!
What `Oh man!` ?
I didn't mean to offend you or anybody, sorry if I did. Read that as "Heureka!", given I found the root cause after 2 months... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.suse.com/show_bug.cgi?id=1167079 Jiri Slaby <jslaby@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(jslaby@suse.com) | -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1167079 https://bugzilla.suse.com/show_bug.cgi?id=1167079#c28 Franck Bui <fbui@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |DUPLICATE --- Comment #28 from Franck Bui <fbui@suse.com> --- Sadly I just noticed that a similar bug was already opened more than one year and a half ago, see bsc#1116625... The reason was already identified but no one took care to fix it :( So let's close this bug a duplicate. *** This bug has been marked as a duplicate of bug 1116625 *** -- You are receiving this mail because: You are on the CC list for the bug.
participants (2)
-
bugzilla_noreply＠novell.com -
bugzilla_noreply＠suse.com