Two new 15.2 installations - systemd automount seems to have stopped working ?
My fstab: (sorry about the folded lines) :
fileserver.local.net:/home/spamchek /home/spamchek nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 fileserver.local.net:/home/hostsuisse /home/hostsuisse nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 //io64.local.net/photos /home/per/Photos cifs user=per,password=abcdefgh,vers=1.0,x-systemd.automount,x-systemd.idle-timeout=300,_netdev 0 0 //heron.local.net/photos /home/per/Photos2 cifs user=per,password=abcdefgh,x-systemd.automount,x-systemd.idle-timeout=300,_netdev 0 0
systemctl | grep automount - produces nothing.
Per Jessen wrote:
Two new 15.2 installations - systemd automount seems to have stopped working ?
My fstab: (sorry about the folded lines) :
fileserver.local.net:/home/spamchek /home/spamchek nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 fileserver.local.net:/home/hostsuisse /home/hostsuisse nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 //io64.local.net/photos /home/per/Photos cifs
user=per,password=abcdefgh,vers=1.0,x-systemd.automount,x-systemd.idle-timeout=300,_netdev
0 0 //heron.local.net/photos /home/per/Photos2 cifs
user=per,password=abcdefgh,x-systemd.automount,x-systemd.idle-timeout=300,_netdev
0 0
systemctl | grep automount - produces nothing.
FWIW, I have done a daemon-reload, but it doesn't seem to trigger the fstab generator.
04.07.2020 14:56, Per Jessen пишет:
Two new 15.2 installations - systemd automount seems to have stopped working ?
What exactly "stopped working" means? No units are generated from fstab? Units are generated but automount unit is not activated on boot? Automount unit is activated but it does activate corresponding mount unit?
My fstab: (sorry about the folded lines) :
fileserver.local.net:/home/spamchek /home/spamchek nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 fileserver.local.net:/home/hostsuisse /home/hostsuisse nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 //io64.local.net/photos /home/per/Photos cifs user=per,password=abcdefgh,vers=1.0,x-systemd.automount,x-systemd.idle-timeout=300,_netdev 0 0 //heron.local.net/photos /home/per/Photos2 cifs user=per,password=abcdefgh,x-systemd.automount,x-systemd.idle-timeout=300,_netdev 0 0
systemctl | grep automount - produces nothing.
Well, it means unit is not active at the time you issued this command.
Anyway, "journalctl -b" with "systemd.log_level=debug" and without "quiet" on kernel command line would be helpful. printk.devkmsg=on may be useful during initrd phase as well.
Andrei Borzenkov wrote:
04.07.2020 14:56, Per Jessen пишет:
Two new 15.2 installations - systemd automount seems to have stopped working ?
What exactly "stopped working" means?
Yeah, I should know better ...
No units are generated from fstab?
Correct, no units were auto-generated from fstab (following a daemon-reload).
Anyway, "journalctl -b" with "systemd.log_level=debug" and without "quiet" on kernel command line would be helpful. printk.devkmsg=on may be useful during initrd phase as well.
Okay, I'll add those and get back.
04.07.2020 20:17, Per Jessen пишет:
Andrei Borzenkov wrote:
04.07.2020 14:56, Per Jessen пишет:
Two new 15.2 installations - systemd automount seems to have stopped working ?
What exactly "stopped working" means?
Yeah, I should know better ...
No units are generated from fstab?
Correct, no units were auto-generated from fstab (following a daemon-reload).
What is the content of /run/systemd/generator{,.early,.late}?
Anyway, "journalctl -b" with "systemd.log_level=debug" and without "quiet" on kernel command line would be helpful. printk.devkmsg=on may be useful during initrd phase as well.
Okay, I'll add those and get back.
Andrei Borzenkov wrote:
04.07.2020 20:17, Per Jessen пишет:
Andrei Borzenkov wrote:
04.07.2020 14:56, Per Jessen пишет:
Two new 15.2 installations - systemd automount seems to have stopped working ?
What exactly "stopped working" means?
Yeah, I should know better ...
No units are generated from fstab?
Correct, no units were auto-generated from fstab (following a daemon-reload).
What is the content of /run/systemd/generator{,.early,.late}?
From my other system (office38), which was not yet rebooted:
office38:~ # l /run/systemd/generator{,.early,.late} ls: cannot access '/run/systemd/generator.early': No such file or directory /run/systemd/generator: total 44 drwxr-xr-x 12 root root 460 Jul 4 13:46 ./ drwxr-xr-x 18 root root 420 Jul 4 13:46 ../ drwxr-xr-x 2 root root 60 Jul 4 13:46 cifs.service.d/ -rw-r--r-- 1 root root 209 Jul 4 13:46 dev-disk-by\x2duuid-293d8657\x2d5d09\x2d4e8c\x2d8737\x2da85c421515aa.swap drwxr-xr-x 2 root root 60 Jul 4 13:46 dnsmasq.service.d/ -rw-r--r-- 1 root root 221 Jul 4 13:46 home-hostsuisse.automount -rw-r--r-- 1 root root 292 Jul 4 13:46 home-hostsuisse.mount -rw-r--r-- 1 root root 449 Jul 4 13:46 home.mount -rw-r--r-- 1 root root 222 Jul 4 13:46 home-per-Photos2.automount -rw-r--r-- 1 root root 304 Jul 4 13:46 home-per-Photos2.mount -rw-r--r-- 1 root root 221 Jul 4 13:46 home-per-Photos.automount -rw-r--r-- 1 root root 311 Jul 4 13:46 home-per-Photos.mount -rw-r--r-- 1 root root 219 Jul 4 13:46 home-spamchek.automount -rw-r--r-- 1 root root 288 Jul 4 13:46 home-spamchek.mount drwxr-xr-x 2 root root 80 Jul 4 13:46 local-fs.target.requires/ drwxr-xr-x 2 root root 60 Jul 4 13:46 local-fs.target.wants/ drwxr-xr-x 2 root root 60 Jul 4 13:46 lwresd.service.d/ -rw-r--r-- 1 root root 250 Jul 4 13:46 -.mount drwxr-xr-x 2 root root 60 Jul 4 13:46 nfs.service.d/ drwxr-xr-x 2 root root 60 Jul 4 13:46 ntpd.service.d/ drwxr-xr-x 2 root root 60 Jul 4 13:46 remote-fs.target.d/ drwxr-xr-x 2 root root 120 Jul 4 13:46 remote-fs.target.requires/ drwxr-xr-x 2 root root 60 Jul 4 13:46 swap.target.requires/
/run/systemd/generator.late: total 4 drwxr-xr-x 2 root root 60 Jul 4 13:46 ./ drwxr-xr-x 18 root root 420 Jul 4 13:46 ../ -rw-r--r-- 1 root root 551 Jul 4 13:46 xfs.service
"systemctl | grep auto" shows nothing, but "systemctl -all | grep auto":
office38:~ # systemctl -all | grep automount boot.automount loaded inactive dead boot.automount home-hostsuisse.automount loaded inactive dead home-hostsuisse.automount home-per-Photos.automount loaded inactive dead home-per-Photos.automount home-per-Photos2.automount loaded inactive dead home-per-Photos2.automount home-spamchek.automount loaded inactive dead home-spamchek.automount proc-sys-fs-binfmt_misc.automount loaded active waiting Arbitrary Executable ...
Per Jessen wrote:
No units are generated from fstab?
Correct, no units were auto-generated from fstab (following a daemon-reload).
Anyway, "journalctl -b" with "systemd.log_level=debug" and without "quiet" on kernel command line would be helpful. printk.devkmsg=on may be useful during initrd phase as well.
Okay, I'll add those and get back.
First feedback - adding the above and rebooting, and I see the auto-generated automount units:
UNIT LOAD ACTIVE SUB DESCRIPTION ● home-hostsuisse.automount loaded failed failed home-hostsuisse.automount home-per-Photos.automount loaded active running home-per-Photos.automount home-per-Photos2.automount loaded active running home-per-Photos2.automount ● home-spamchek.automount loaded failed failed home-spamchek.automount
The two 'Photos' mounts both work.
I guess one conclusion is that 'daemon-reload' isn't enough to re-run the fstab generator?
Debug output here - https://files.jessen.ch/office68-journalctl-b.txt.gz I expect the below is the interesting bit ?
# zgrep spamchek /tmp/office68-journalctl-b.txt.gz Jul 05 09:51:33 localhost systemd-fstab-generator[447]: Found entry what=fileserver.local.net:/home/spamchek where=/home/spamchek type=nfs nofail=no noauto=no Jul 05 09:51:33 localhost systemd[1]: home-spamchek.automount: Installed new job home-spamchek.automount/start as 215 Jul 05 09:51:34 localhost systemd[1]: home-spamchek.automount: Changed dead -> waiting Jul 05 09:51:34 localhost systemd[1]: home-spamchek.automount: Job home-spamchek.automount/start finished, result=done Jul 05 09:51:34 localhost systemd[1]: Set up automount home-spamchek.automount. Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Got invalid poll event 16 on pipe (fd=40) Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Changed waiting -> failed Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Unit entered failed state.
05.07.2020 11:09, Per Jessen пишет:
Per Jessen wrote:
No units are generated from fstab?
Correct, no units were auto-generated from fstab (following a daemon-reload).
Anyway, "journalctl -b" with "systemd.log_level=debug" and without "quiet" on kernel command line would be helpful. printk.devkmsg=on may be useful during initrd phase as well.
Okay, I'll add those and get back.
First feedback - adding the above and rebooting, and I see the auto-generated automount units:
UNIT LOAD ACTIVE SUB DESCRIPTION ● home-hostsuisse.automount loaded failed failed home-hostsuisse.automount home-per-Photos.automount loaded active running home-per-Photos.automount home-per-Photos2.automount loaded active running home-per-Photos2.automount ● home-spamchek.automount loaded failed failed home-spamchek.automount
The two 'Photos' mounts both work.
I guess one conclusion is that 'daemon-reload' isn't enough to re-run the fstab generator?
Debug output here - https://files.jessen.ch/office68-journalctl-b.txt.gz I expect the below is the interesting bit ?
# zgrep spamchek /tmp/office68-journalctl-b.txt.gz Jul 05 09:51:33 localhost systemd-fstab-generator[447]: Found entry what=fileserver.local.net:/home/spamchek where=/home/spamchek type=nfs nofail=no noauto=no Jul 05 09:51:33 localhost systemd[1]: home-spamchek.automount: Installed new job home-spamchek.automount/start as 215 Jul 05 09:51:34 localhost systemd[1]: home-spamchek.automount: Changed dead -> waiting Jul 05 09:51:34 localhost systemd[1]: home-spamchek.automount: Job home-spamchek.automount/start finished, result=done Jul 05 09:51:34 localhost systemd[1]: Set up automount home-spamchek.automount. Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Got invalid poll event 16 on pipe (fd=40) Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Changed waiting -> failed Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Unit entered failed state.
You are likely hitting https://bugzilla.opensuse.org/show_bug.cgi?id=1116625 where NM nfs dispatcher script unmounts systemd-automount-managed mount points due to lack of NFS connectivity. At least, "invalid poll event" exactly follows running nm-dispatcher and errors from ping.
You may want to enable NM debugging instead to get better picture what happens. Unfortunately, enabling debugging often changes timing and makes race condition go away. But still it is better than nothing. In particular, I am not sure why this code would run at boot, but it fits symptoms exactly.
Try change suggested in https://bugzilla.opensuse.org/show_bug.cgi?id=1116625#c32 whether it helps. There was no feedback from reporter.
Also if you do not need NM, testing with wicked also makes sense.
Andrei Borzenkov wrote:
05.07.2020 11:09, Per Jessen пишет:
Per Jessen wrote:
No units are generated from fstab?
Correct, no units were auto-generated from fstab (following a daemon-reload).
Anyway, "journalctl -b" with "systemd.log_level=debug" and without "quiet" on kernel command line would be helpful. printk.devkmsg=on may be useful during initrd phase as well.
Okay, I'll add those and get back.
First feedback - adding the above and rebooting, and I see the auto-generated automount units:
UNIT LOAD ACTIVE SUB DESCRIPTION ● home-hostsuisse.automount loaded failed failed home-hostsuisse.automount home-per-Photos.automount loaded active running home-per-Photos.automount home-per-Photos2.automount loaded active running home-per-Photos2.automount ● home-spamchek.automount loaded failed failed home-spamchek.automount
The two 'Photos' mounts both work.
I guess one conclusion is that 'daemon-reload' isn't enough to re-run the fstab generator?
Debug output here - https://files.jessen.ch/office68-journalctl-b.txt.gz I expect the below is the interesting bit ?
# zgrep spamchek /tmp/office68-journalctl-b.txt.gz Jul 05 09:51:33 localhost systemd-fstab-generator[447]: Found entry what=fileserver.local.net:/home/spamchek where=/home/spamchek type=nfs nofail=no noauto=no Jul 05 09:51:33 localhost systemd[1]: home-spamchek.automount: Installed new job home-spamchek.automount/start as 215 Jul 05 09:51:34 localhost systemd[1]: home-spamchek.automount: Changed dead -> waiting Jul 05 09:51:34 localhost systemd[1]: home-spamchek.automount: Job home-spamchek.automount/start finished, result=done Jul 05 09:51:34 localhost systemd[1]: Set up automount home-spamchek.automount. Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Got invalid poll event 16 on pipe (fd=40) Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Changed waiting -> failed Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Unit entered failed state.
You are likely hitting https://bugzilla.opensuse.org/show_bug.cgi?id=1116625 where NM nfs dispatcher script unmounts systemd-automount-managed mount points due to lack of NFS connectivity. At least, "invalid poll event" exactly follows running nm-dispatcher and errors from ping.
The above system (office68) does indeed use NM.
You may want to enable NM debugging instead to get better picture what happens. Unfortunately, enabling debugging often changes timing and makes race condition go away. But still it is better than nothing. In particular, I am not sure why this code would run at boot, but it fits symptoms exactly.
Try change suggested in https://bugzilla.opensuse.org/show_bug.cgi?id=1116625#c32 whether it helps. There was no feedback from reporter.
I've amended that script and re/started the automount units, we'll see what happens. The last comment in the report says "backported" to all supported distros.
Also if you do not need NM, testing with wicked also makes sense.
The other system (office38) uses wicked, but although the fstab generation works, the automount units are dead? I'll add the debug setup to that system too.
Thanks for digging out that report.
05.07.2020 17:41, Per Jessen пишет: ...
You are likely hitting https://bugzilla.opensuse.org/show_bug.cgi?id=1116625 where NM nfs dispatcher script unmounts systemd-automount-managed mount points due to lack of NFS connectivity. At least, "invalid poll event" exactly follows running nm-dispatcher and errors from ping.
The above system (office68) does indeed use NM.
You may want to enable NM debugging instead to get better picture what happens. Unfortunately, enabling debugging often changes timing and makes race condition go away. But still it is better than nothing. In particular, I am not sure why this code would run at boot, but it fits symptoms exactly.
Try change suggested in https://bugzilla.opensuse.org/show_bug.cgi?id=1116625#c32 whether it helps. There was no feedback from reporter.
I've amended that script and re/started the automount units, we'll see what happens. The last comment in the report says "backported" to all supported distros.
What was backported was patch for systemd, but that was not root cause anyway. There was no change to nfs dispatcher script.
Also if you do not need NM, testing with wicked also makes sense.
The other system (office38) uses wicked, but although the fstab generation works, the automount units are dead? I'll add the debug setup to that system too.
Thanks for digging out that report.
Per Jessen wrote:
Two new 15.2 installations - systemd automount seems to have stopped working ?
Update: system with wicked (office38) - after rebooting, all four automounts (nfs and cifs) now seem to work. I would have thought doing a "daemon-reload" would have been sufficient, but apparently not.
Update: system with NetworkManager (office68) - after rebooting, the two cifs automounts work, the two nfs automounts are status 'failed'. (with the proposed patch to dispatcher.d/nfs). After re/starting the automount units, I am now waiting to see if they'll fail again.
06.07.2020 11:50, Per Jessen пишет:
Per Jessen wrote:
Two new 15.2 installations - systemd automount seems to have stopped working ?
Update: system with wicked (office38) - after rebooting, all four automounts (nfs and cifs) now seem to work. I would have thought doing a "daemon-reload" would have been sufficient, but apparently not.
"Sufficient" for what? daemon-reload just reloads unit definitions, that's all. It does attempt to restart anything. If some unit failed before daemon-reload, it remains failed after daemon-reload (at least, it should, if not, it is a bug).
Update: system with NetworkManager (office68) - after rebooting, the two cifs automounts work, the two nfs automounts are status 'failed'. (with the proposed patch to dispatcher.d/nfs). After re/starting the automount units, I am now waiting to see if they'll fail again.
If hypothesis is correct, they fail during connectivity changes, so you may not see it again. I would add debug print to nfs dispatcher script (print arguments and environment, also some "interesting" variables, dump it into a file on every invocation) to capture what happens during boot.
Andrei Borzenkov wrote:
06.07.2020 11:50, Per Jessen пишет:
Per Jessen wrote:
Two new 15.2 installations - systemd automount seems to have stopped working ?
Update: system with wicked (office38) - after rebooting, all four automounts (nfs and cifs) now seem to work. I would have thought doing a "daemon-reload" would have been sufficient, but apparently not.
"Sufficient" for what? daemon-reload just reloads unit definitions, that's all. It does attempt to restart anything. If some unit failed before daemon-reload, it remains failed after daemon-reload (at least, it should, if not, it is a bug).
I meant I expected 'daemon-reload' to be sufficient to create those automount units after I have updated fstab.
Update: system with NetworkManager (office68) - after rebooting, the two cifs automounts work, the two nfs automounts are status 'failed'. (with the proposed patch to dispatcher.d/nfs). After re/starting the automount units, I am now waiting to see if they'll fail again.
If hypothesis is correct, they fail during connectivity changes, so you may not see it again. I would add debug print to nfs dispatcher script (print arguments and environment, also some "interesting" variables, dump it into a file on every invocation) to capture what happens during boot.
After a suspend/resume over night, the two nfs automount units are now 'failed' again.
07.07.2020 11:31, Per Jessen пишет:
After a suspend/resume over night, the two nfs automount units are now 'failed' again.
Which makes strong argument that you are still seeing the same problem as in mentioned bug report. May be your editing was not effective.
Andrei Borzenkov wrote:
07.07.2020 11:31, Per Jessen пишет:
After a suspend/resume over night, the two nfs automount units are now 'failed' again.
Which makes strong argument that you are still seeing the same problem as in mentioned bug report. May be your editing was not effective.
'effective' :-)
Looking at it just now, after suspending this laptop overnight, the two NFS automount units are 'failed' again, but that happens on resume. (got invalid poll ....)
Re-reading the bugreport:
The dispatcher.d/nfs script should ignore x-systemd.automount entries in fstab. e.g. change
NET_MOUNTS=$(sed -e '/^.*#/d' -e '/^.*:/!d' -e 's/\t/ /g' /etc/fstab | tr -s " ")$'\n'
by adding a clause -e '/x-systemd.automount/!d' to the sed command.
I am not a sed guru, but afaict, that does not work according to the text.
#sed -e '/^.*#/d' -e '/^.*:/!d' -e 's/\t/ /g' -e '/x-systemd.automount/!d' /etc/fstab fileserver.local.net:/home/spamchek /home/spamchek nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 fileserver.local.net:/home/hostsuisse /home/hostsuisse nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0
I have now added -e '/x-systemd.automount/d' instead, i.e. removed the '!'.
On Wed, Jul 8, 2020 at 9:27 AM Per Jessen per@computer.org wrote:
#sed -e '/^.*#/d' -e '/^.*:/!d' -e 's/\t/ /g' -e '/x-systemd.automount/!d' /etc/fstab fileserver.local.net:/home/spamchek /home/spamchek nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 fileserver.local.net:/home/hostsuisse /home/hostsuisse nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0
I use these:
source:/vol1 /source/vol1 nfs defaults,rsize=8192,wsize=8192,bg,soft,intr,noatime,noauto,x-systemd.automount,x-systemd.idle-timeout=60,nofail,_netdev 0 0
So not so different from yours. I do not explicitly request tcp protocol.
Granted this in Tumbleweed and 15.1. I do not have a 15.2 system yet. I have had no trouble across boots or time.
What do the two generated systemd service files for these mounts look like?
Roger Oberholtzer wrote:
On Wed, Jul 8, 2020 at 9:27 AM Per Jessen per@computer.org wrote:
#sed -e '/^.*#/d' -e '/^.*:/!d' -e 's/\t/ /g' -e #'/x-systemd.automount/!d' /etc/fstab fileserver.local.net:/home/spamchek /home/spamchek nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 fileserver.local.net:/home/hostsuisse /home/hostsuisse nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0
I use these:
source:/vol1 /source/vol1 nfs defaults,rsize=8192,wsize=8192,bg,soft,intr,noatime,noauto,x-systemd.automount,x-systemd.idle-timeout=60,nofail,_netdev 0 0
rsize and wsize are superfluous with nfs4, afaik. I'll have to look up what 'bg' means.
So not so different from yours. I do not explicitly request tcp protocol.
Hmm, I don't know why I have that option either.
What do the two generated systemd service files for these mounts look like?
For instance:
# Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Documentation=man:fstab(5) man:systemd-fstab-generator(8) Before=remote-fs.target
[Automount] Where=/home/spamchek TimeoutIdleSec=5min
On Wed, Jul 8, 2020 at 1:41 PM Per Jessen per@computer.org wrote:
rsize and wsize are superfluous with nfs4, afaik. I'll have to look up what 'bg' means.
"bg" makes little sense with automount. If mount cannot be completed immediately, it forks and continues in the background. I'm not even sure how automount will interpret it (mount command completes, but nothing is mounted ...).
Per Jessen wrote:
Re-reading the bugreport:
The dispatcher.d/nfs script should ignore x-systemd.automount entries in fstab. e.g. change
NET_MOUNTS=$(sed -e '/^.*#/d' -e '/^.*:/!d' -e 's/\t/ /g' /etc/fstab | tr -s " ")$'\n'
by adding a clause -e '/x-systemd.automount/!d' to the sed command. [snip] Instead I have now added -e '/x-systemd.automount/d' instead, i.e. removed the '!'.
That seems to done the job - after a suspend/resume, the automount unit is still working.
-
Andrei Borzenkov -
Per Jessen -
Roger Oberholtzer