04.07.2020 14:56, Per Jessen пишет:

Two new 15.2 installations - systemd automount seems to have stopped working ?

What exactly "stopped working" means? No units are generated from fstab? Units are generated but automount unit is not activated on boot? Automount unit is activated but it does activate corresponding mount unit?

My fstab: (sorry about the folded lines) :

fileserver.local.net:/home/spamchek /home/spamchek nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 fileserver.local.net:/home/hostsuisse /home/hostsuisse nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 //io64.local.net/photos /home/per/Photos cifs user=per,password=abcdefgh,vers=1.0,x-systemd.automount,x-systemd.idle-timeout=300,_netdev 0 0 //heron.local.net/photos /home/per/Photos2 cifs user=per,password=abcdefgh,x-systemd.automount,x-systemd.idle-timeout=300,_netdev 0 0

systemctl | grep automount - produces nothing.

Well, it means unit is not active at the time you issued this command. Anyway, "journalctl -b" with "systemd.log_level=debug" and without "quiet" on kernel command line would be helpful. printk.devkmsg=on may be useful during initrd phase as well. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

04.07.2020 20:17, Per Jessen пишет:

Andrei Borzenkov wrote:

...

04.07.2020 14:56, Per Jessen пишет:

...

Two new 15.2 installations - systemd automount seems to have stopped working ?

What exactly "stopped working" means?

Yeah, I should know better ...

...

No units are generated from fstab?

Correct, no units were auto-generated from fstab (following a daemon-reload).

What is the content of /run/systemd/generator{,.early,.late}?

...

Anyway, "journalctl -b" with "systemd.log_level=debug" and without "quiet" on kernel command line would be helpful. printk.devkmsg=on may be useful during initrd phase as well.

Okay, I'll add those and get back.

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andrei Borzenkov wrote:

04.07.2020 20:17, Per Jessen пишет:

...

Andrei Borzenkov wrote:

...

04.07.2020 14:56, Per Jessen пишет:

...

Two new 15.2 installations - systemd automount seems to have stopped working ?

What exactly "stopped working" means?

Yeah, I should know better ...

...

No units are generated from fstab?

Correct, no units were auto-generated from fstab (following a daemon-reload).

What is the content of /run/systemd/generator{,.early,.late}?

From my other system (office38), which was not yet rebooted:

office38:~ # l /run/systemd/generator{,.early,.late} ls: cannot access '/run/systemd/generator.early': No such file or directory /run/systemd/generator: total 44 drwxr-xr-x 12 root root 460 Jul 4 13:46 ./ drwxr-xr-x 18 root root 420 Jul 4 13:46 ../ drwxr-xr-x 2 root root 60 Jul 4 13:46 cifs.service.d/ -rw-r--r-- 1 root root 209 Jul 4 13:46 dev-disk-by\x2duuid-293d8657\x2d5d09\x2d4e8c\x2d8737\x2da85c421515aa.swap drwxr-xr-x 2 root root 60 Jul 4 13:46 dnsmasq.service.d/ -rw-r--r-- 1 root root 221 Jul 4 13:46 home-hostsuisse.automount -rw-r--r-- 1 root root 292 Jul 4 13:46 home-hostsuisse.mount -rw-r--r-- 1 root root 449 Jul 4 13:46 home.mount -rw-r--r-- 1 root root 222 Jul 4 13:46 home-per-Photos2.automount -rw-r--r-- 1 root root 304 Jul 4 13:46 home-per-Photos2.mount -rw-r--r-- 1 root root 221 Jul 4 13:46 home-per-Photos.automount -rw-r--r-- 1 root root 311 Jul 4 13:46 home-per-Photos.mount -rw-r--r-- 1 root root 219 Jul 4 13:46 home-spamchek.automount -rw-r--r-- 1 root root 288 Jul 4 13:46 home-spamchek.mount drwxr-xr-x 2 root root 80 Jul 4 13:46 local-fs.target.requires/ drwxr-xr-x 2 root root 60 Jul 4 13:46 local-fs.target.wants/ drwxr-xr-x 2 root root 60 Jul 4 13:46 lwresd.service.d/ -rw-r--r-- 1 root root 250 Jul 4 13:46 -.mount drwxr-xr-x 2 root root 60 Jul 4 13:46 nfs.service.d/ drwxr-xr-x 2 root root 60 Jul 4 13:46 ntpd.service.d/ drwxr-xr-x 2 root root 60 Jul 4 13:46 remote-fs.target.d/ drwxr-xr-x 2 root root 120 Jul 4 13:46 remote-fs.target.requires/ drwxr-xr-x 2 root root 60 Jul 4 13:46 swap.target.requires/ /run/systemd/generator.late: total 4 drwxr-xr-x 2 root root 60 Jul 4 13:46 ./ drwxr-xr-x 18 root root 420 Jul 4 13:46 ../ -rw-r--r-- 1 root root 551 Jul 4 13:46 xfs.service "systemctl | grep auto" shows nothing, but "systemctl -all | grep auto": office38:~ # systemctl -all | grep automount boot.automount loaded inactive dead boot.automount home-hostsuisse.automount loaded inactive dead home-hostsuisse.automount home-per-Photos.automount loaded inactive dead home-per-Photos.automount home-per-Photos2.automount loaded inactive dead home-per-Photos2.automount home-spamchek.automount loaded inactive dead home-spamchek.automount proc-sys-fs-binfmt_misc.automount loaded active waiting Arbitrary Executable ... -- Per Jessen, Zürich (21.6°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andrei Borzenkov wrote:

05.07.2020 11:09, Per Jessen пишет:

...

Per Jessen wrote:

...

...

No units are generated from fstab?

Correct, no units were auto-generated from fstab (following a daemon-reload).

...

Anyway, "journalctl -b" with "systemd.log_level=debug" and without "quiet" on kernel command line would be helpful. printk.devkmsg=on may be useful during initrd phase as well.

Okay, I'll add those and get back.

First feedback - adding the above and rebooting, and I see the auto-generated automount units:

UNIT LOAD ACTIVE SUB DESCRIPTION ● home-hostsuisse.automount loaded failed failed home-hostsuisse.automount home-per-Photos.automount loaded active running home-per-Photos.automount home-per-Photos2.automount loaded active running home-per-Photos2.automount ● home-spamchek.automount loaded failed failed home-spamchek.automount

The two 'Photos' mounts both work.

I guess one conclusion is that 'daemon-reload' isn't enough to re-run the fstab generator?

Debug output here - https://files.jessen.ch/office68-journalctl-b.txt.gz I expect the below is the interesting bit ?

# zgrep spamchek /tmp/office68-journalctl-b.txt.gz Jul 05 09:51:33 localhost systemd-fstab-generator[447]: Found entry what=fileserver.local.net:/home/spamchek where=/home/spamchek type=nfs nofail=no noauto=no Jul 05 09:51:33 localhost systemd[1]: home-spamchek.automount: Installed new job home-spamchek.automount/start as 215 Jul 05 09:51:34 localhost systemd[1]: home-spamchek.automount: Changed dead -> waiting Jul 05 09:51:34 localhost systemd[1]: home-spamchek.automount: Job home-spamchek.automount/start finished, result=done Jul 05 09:51:34 localhost systemd[1]: Set up automount home-spamchek.automount. Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Got invalid poll event 16 on pipe (fd=40) Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Changed waiting -> failed Jul 05 09:52:12 localhost.localdomain systemd[1]: home-spamchek.automount: Unit entered failed state.

You are likely hitting https://bugzilla.opensuse.org/show_bug.cgi?id=1116625 where NM nfs dispatcher script unmounts systemd-automount-managed mount points due to lack of NFS connectivity. At least, "invalid poll event" exactly follows running nm-dispatcher and errors from ping.

The above system (office68) does indeed use NM.

You may want to enable NM debugging instead to get better picture what happens. Unfortunately, enabling debugging often changes timing and makes race condition go away. But still it is better than nothing. In particular, I am not sure why this code would run at boot, but it fits symptoms exactly.

Try change suggested in https://bugzilla.opensuse.org/show_bug.cgi?id=1116625#c32 whether it helps. There was no feedback from reporter.

I've amended that script and re/started the automount units, we'll see what happens. The last comment in the report says "backported" to all supported distros.

Also if you do not need NM, testing with wicked also makes sense.

The other system (office38) uses wicked, but although the fstab generation works, the automount units are dead? I'll add the debug setup to that system too. Thanks for digging out that report. -- Per Jessen, Zürich (27.9°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Per Jessen wrote:

Two new 15.2 installations - systemd automount seems to have stopped working ?

Update: system with wicked (office38) - after rebooting, all four automounts (nfs and cifs) now seem to work. I would have thought doing a "daemon-reload" would have been sufficient, but apparently not. Update: system with NetworkManager (office68) - after rebooting, the two cifs automounts work, the two nfs automounts are status 'failed'. (with the proposed patch to dispatcher.d/nfs). After re/starting the automount units, I am now waiting to see if they'll fail again. -- Per Jessen, Zürich (20.8°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andrei Borzenkov wrote:

06.07.2020 11:50, Per Jessen пишет:

...

Per Jessen wrote:

...

Two new 15.2 installations - systemd automount seems to have stopped working ?

Update: system with wicked (office38) - after rebooting, all four automounts (nfs and cifs) now seem to work. I would have thought doing a "daemon-reload" would have been sufficient, but apparently not.

"Sufficient" for what? daemon-reload just reloads unit definitions, that's all. It does attempt to restart anything. If some unit failed before daemon-reload, it remains failed after daemon-reload (at least, it should, if not, it is a bug).

I meant I expected 'daemon-reload' to be sufficient to create those automount units after I have updated fstab.

...

Update: system with NetworkManager (office68) - after rebooting, the two cifs automounts work, the two nfs automounts are status 'failed'. (with the proposed patch to dispatcher.d/nfs). After re/starting the automount units, I am now waiting to see if they'll fail again.

If hypothesis is correct, they fail during connectivity changes, so you may not see it again. I would add debug print to nfs dispatcher script (print arguments and environment, also some "interesting" variables, dump it into a file on every invocation) to capture what happens during boot.

After a suspend/resume over night, the two nfs automount units are now 'failed' again. -- Per Jessen, Zürich (17.1°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andrei Borzenkov wrote:

07.07.2020 11:31, Per Jessen пишет:

...

After a suspend/resume over night, the two nfs automount units are now 'failed' again.

Which makes strong argument that you are still seeing the same problem as in mentioned bug report. May be your editing was not effective.

'effective' :-) Looking at it just now, after suspending this laptop overnight, the two NFS automount units are 'failed' again, but that happens on resume. (got invalid poll ....) Re-reading the bugreport: The dispatcher.d/nfs script should ignore x-systemd.automount entries in fstab. e.g. change NET_MOUNTS=$(sed -e '/^.*#/d' -e '/^.*:/!d' -e 's/\t/ /g' /etc/fstab | tr -s " ")$'\n' by adding a clause -e '/x-systemd.automount/!d' to the sed command. I am not a sed guru, but afaict, that does not work according to the text. #sed -e '/^.*#/d' -e '/^.*:/!d' -e 's/\t/ /g' -e '/x-systemd.automount/!d' /etc/fstab fileserver.local.net:/home/spamchek /home/spamchek nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 fileserver.local.net:/home/hostsuisse /home/hostsuisse nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 I have now added -e '/x-systemd.automount/d' instead, i.e. removed the '!'. -- Per Jessen, Zürich (19.0°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Roger Oberholtzer wrote:

On Wed, Jul 8, 2020 at 9:27 AM Per Jessen wrote:

...

#sed -e '/^.*#/d' -e '/^.*:/!d' -e 's/\t/ /g' -e #'/x-systemd.automount/!d' /etc/fstab fileserver.local.net:/home/spamchek /home/spamchek nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0 fileserver.local.net:/home/hostsuisse /home/hostsuisse nfs x-systemd.automount,x-systemd.idle-timeout=300,tcp,_netdev 0 0

I use these:

source:/vol1 /source/vol1 nfs defaults,rsize=8192,wsize=8192,bg,soft,intr,noatime,noauto,x-systemd.automount,x-systemd.idle-timeout=60,nofail,_netdev 0 0

rsize and wsize are superfluous with nfs4, afaik. I'll have to look up what 'bg' means.

So not so different from yours. I do not explicitly request tcp protocol.

Hmm, I don't know why I have that option either.

What do the two generated systemd service files for these mounts look like?

For instance: # Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Documentation=man:fstab(5) man:systemd-fstab-generator(8) Before=remote-fs.target [Automount] Where=/home/spamchek TimeoutIdleSec=5min -- Per Jessen, Zürich (21.8°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Per Jessen wrote:

Re-reading the bugreport:

The dispatcher.d/nfs script should ignore x-systemd.automount entries in fstab. e.g. change

NET_MOUNTS=$(sed -e '/^.*#/d' -e '/^.*:/!d' -e 's/\t/ /g' /etc/fstab | tr -s " ")$'\n'

by adding a clause -e '/x-systemd.automount/!d' to the sed command. [snip] Instead I have now added -e '/x-systemd.automount/d' instead, i.e. removed the '!'.

That seems to done the job - after a suspend/resume, the automount unit is still working. -- Per Jessen, Zürich (22.4°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org