http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c2

--- Comment #2 from Jochen Breuer jbreuer@suse.com --- (In reply to Jochen Breuer from comment #1)

Why do you think this is a Salt related problem?

The component was set to Salt.

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c5

--- Comment #5 from Walddys Emmanuel Dorrejo C�spedes dev.dorrejo@gmail.com --- (In reply to Jochen Breuer from comment #1)

Why do you think this is a Salt related problem?

sorry didn't notice

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c7

--- Comment #7 from Walddys Emmanuel Dorrejo C�spedes dev.dorrejo@gmail.com --- (In reply to Walddys Emmanuel Dorrejo C�spedes from comment #6)

Hello, after following the logs while trying to unmask var.mount and tmp.mount found that it will generate this:

/usr/lib/systemd/system/plymouth-start.service:15: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed.

so how can i disable playmouth to try if this solve the issue?

thanks.

restarting the whole server is the only solution i found

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c13

--- Comment #13 from Walddys Emmanuel Dorrejo C�spedes dev.dorrejo@gmail.com --- (In reply to Franck Bui from comment #12)

Walddys, could you provide the information requested in comment #11 ?

Sorry for late response., was hit by covid-19, but all good now.

https://pastebin.com/A6G1bJd3

upgrading today my TW server give this too:

systemctl restart php-fpm.service apache2.service snapperd.service postgresql.service sshd.service mariadb.service redis@default.service redis@peer.service polkit.service systemd-logind.service polkit.service irqbalance.service

Failed to restart php-fpm.service: Unit tmp.mount is masked. Failed to restart apache2.service: Unit var.mount is masked. Failed to restart redis@default.service: Unit var.mount is masked. Failed to restart redis@peer.service: Unit var.mount is masked. Failed to restart systemd-logind.service: Unit tmp.mount is masked.

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c15

--- Comment #15 from Walddys Emmanuel Dorrejo C�spedes dev.dorrejo@gmail.com --- Created attachment 850425 --> http://bugzilla.opensuse.org/attachment.cgi?id=850425&action=edit mask-apache2-service

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970

Cliff Zhao qzhao@suse.com changed:

What |Removed |Added ---------------------------------------------------------------------------- CC|qzhao@suse.com |

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c18

--- Comment #18 from Walddys Emmanuel Dorrejo C�spedes dev.dorrejo@gmail.com --- (In reply to Franck Bui from comment #17)

Walddys,

Did you change anything in your setup ?

I'm asking because restarting apache was initially failing like this:

Failed to restart apache2.service: Unit tmp.mount is masked.

But now it fails with:

Failed to restart apache2.service: Unit var.mount is masked.

So now systemd complains about /var being masked.

Also why is var.mount masked ?

Can you share /etc/fstab ?

Thanks.

no i dont change nothing, i said before (up) that complain about that two tmp and var.

/etc/fstab

/dev/system/root / btrfs defaults 0 0 /dev/system/root /.snapshots btrfs subvol=/@/.snapshots 0 0 /dev/system/root /var btrfs subvol=/@/var 0 0 /dev/system/root /usr/local btrfs subvol=/@/usr/local 0 0 /dev/system/root /srv btrfs subvol=/@/srv 0 0 /dev/system/root /root btrfs subvol=/@/root 0 0 /dev/system/root /opt btrfs subvol=/@/opt 0 0 /dev/system/root /home btrfs subvol=/@/home 0 0 /dev/system/root /boot/grub2/x86_64-efi btrfs subvol=/@/boot/grub2/x86_64-efi 0 0 /dev/system/root /boot/grub2/i386-pc btrfs subvol=/@/boot/grub2/i386-pc 0 0 UUID=3510-7E2E /boot/efi vfat utf8 0 2 /dev/system/astian /opt/astian btrfs defaults 0 0 /dev/system/root /snap btrfs subvol=@/snap 0 0 /dev/system/acloud /opt/acloud btrfs defaults 0 0 /dev/system/peertube /opt/peertube btrfs defaults 0 0

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c20

--- Comment #20 from Walddys Emmanuel Dorrejo C��spedes dev.dorrejo@gmail.com --- (In reply to Franck Bui from comment #19)

Can you please answer to this question (already asked twice): why is var.mount masked ?

don't know, appear with tmp.mount at the same time.

Both tmp.mount and var.mount appear masked when tried to restart the service.

i have this server on TW, and start appearing this issue from the moment i create this issue, and since there, when i update the server and have to restart the service commonly apache2.service and php-fpm.service, there both file will said that are mask, but the mask flag on the .service appear after i try restart those both service.

before the restart

mercurio:/home/wdorrejo # systemctl status var.mount ��� var.mount - /var Loaded: loaded (/etc/fstab; generated) Active: active (mounted) since Fri 2021-06-18 12:47:22 -05; 4 days ago Where: /var What: /dev/mapper/system-root Docs: man:fstab(5) man:systemd-fstab-generator(8) Tasks: 0 (limit: 4915) CPU: 1ms CGroup: /system.slice/var.mount

Jun 18 12:47:21 localhost systemd[1]: Mounting /var... Jun 18 12:47:22 localhost systemd[1]: Mounted /var. mercurio:/home/wdorrejo # systemctl status tmp.mount ��� tmp.mount - Temporary Directory (/tmp) Loaded: loaded (/proc/self/mountinfo; static) Active: active (mounted) since Fri 2021-06-18 12:47:21 -05; 4 days ago Where: /tmp What: tmpfs Docs: https://systemd.io/TEMPORARY_DIRECTORIES man:file-hierarchy(7) https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems Tasks: 0 (limit: 4915) CPU: 2ms CGroup: /system.slice/tmp.mount

is will show normally.

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c22

--- Comment #22 from Walddys Emmanuel Dorrejo C�spedes dev.dorrejo@gmail.com --- (In reply to Franck Bui from comment #21)

It sounds like restarting apache2 is the action that would mask var.mount, which sounds strange.

But looking at your logs, it appears that var.mount was masked already before:

...

[40367.178540] mercurio systemd[1]: logrotate.timer: Failed to queue unit startup job: Unit var.mount is masked.

When the problem is present, can you please attach the output of:

# find /run/systemd -name var*.mount -exec ls -l {} ;

?

Ok, maybe take so time because have to wait for an apache upgrade or so packages that alter apache2.

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c24

--- Comment #24 from OBSbugzilla Bot bwiedemann+obsbugzillabot@suse.com --- This is an autogenerated message for OBS integration: This bug (1181970) was mentioned in https://build.opensuse.org/request/show/902866 Factory / systemd

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c26

--- Comment #26 from Franck Bui fbui@suse.com --- Then it would be nice whether you could find out how the mount unit get masked, not sure how to do that though.

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c28

--- Comment #28 from Walddys Emmanuel Dorrejo C�spedes dev.dorrejo@gmail.com --- (In reply to Walddys Emmanuel Dorrejo C�spedes from comment #27)

(In reply to Franck Bui from comment #21)

...

It sounds like restarting apache2 is the action that would mask var.mount, which sounds strange.

But looking at your logs, it appears that var.mount was masked already before:

...

[40367.178540] mercurio systemd[1]: logrotate.timer: Failed to queue unit startup job: Unit var.mount is masked.

When the problem is present, can you please attach the output of:

# find /run/systemd -name var*.mount -exec ls -l {} ;

?

find /run/systemd -name var*.mount -exec ls -l {} ; -rw-r--r-- 1 root root 254 Jul 8 22:40 /run/systemd/generator/var.mount lrwxrwxrwx 1 root root 12 Jul 8 22:40 /run/systemd/generator/local-fs.target.requires/var.mount -> ../var.mount lrwxrwxrwx 1 root root 9 Jul 8 21:06 /run/systemd/system/var.mount -> /dev/null lrwxrwxrwx 1 root root 9 Jul 8 21:06 /run/systemd/system/var-run.mount -> /dev/null lrwxrwxrwx 1 root root 9 Jul 8 21:06 /run/systemd/system/var-lock.mount -> /dev/null lrwxrwxrwx 1 root root 9 Jul 8 21:06 /run/systemd/system/var-lib-machines.mount -> /dev/null

i think is snapd, because i use certbot, but will have to change to pip to install certbot.

but have to test more.

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c30

--- Comment #30 from Franck Bui fbui@suse.com --- To discover the process that masks the mount units, you can try to use auditd.

Please follow the following steps to enable auditd and add the relevant rules to monitor the symlinks:

- make sure that the mount units are no more masked, maybe reboot so you start with a clean state

- make sure auditd is enabled and running with `systemctl status auditd`

- clear all existing rules with `auditctl -D`

- add the rules to monitor the creation of the symlinks:

auditctl -w /run/systemd/system/var.mount -p w -k mask-var-mounts auditctl -w /run/systemd/system/var-run.mount -p w -k mask-var-mounts

- verify that the rule has been added: `auditctl -l`

After that just wait until var.mount or var-run.mount get masked again then attach the output of `ausearch -i -k mask-var-mounts`.

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c31

--- Comment #31 from Franck Bui fbui@suse.com --- Also please enable the systemd debug logs with `systemctl log-level debug` because if the process that masks the mount units calls `systemctl mask` then PID1 will be the one that creates the symlinks but we won't be able to see which process initiates the request.

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c34

--- Comment #34 from Franck Bui fbui@suse.com --- (In reply to Walddys Emmanuel Dorrejo C�spedes from comment #33)

i'm back.

Date of creating *.mount

lrwxrwxrwx 1 root root 9 Aug 25 11:44 tmp.mount -> /dev/null lrwxrwxrwx 1 root root 9 Aug 25 11:44 var-lib-machines.mount -> /dev/null lrwxrwxrwx 1 root root 9 Aug 25 11:44 var-lock.mount -> /dev/null lrwxrwxrwx 1 root root 9 Aug 25 11:44 var-run.mount -> /dev/null lrwxrwxrwx 1 root root 9 Aug 25 11:44 var.mount -> /dev/null

journal > https://pastebin.com/4h34yEcw

This log file starts with:

-- Journal begins at Fri 2021-08-20 19:37:51 -05, ends at Wed 2021-08-25 15:31:11 -05. -- [...]

That's not really helpfull since it shows logs emitted after the mount units were masked.

Can you show the relevant parts of the journal instead (anything that happened around Aug 25 11:44) ?

var-mask > https://pastebin.com/TsxZYkzY

Nothing interesting here meaning probably that PID1 was requested to mask the units.

zypp-history > https://pastebin.com/GU6Q2QmT

This one shows that a `zypper dup` was (manually) issued less than 2 minutes before:

2021-08-25 11:42:43|command|root@venus|'zypper' 'dup'

A small set of packages was updated.

The last thing that happened right before the creation of the masks is the recreation of the initrd which was triggered by the update of btrfsprogs package:

# 2021-08-25 11:43:37 Output of btrfsprogs-5.13.1-1.1.x86_64.rpm %posttrans script:

Dunno, maybe try to recreate your initrd manually with `mkinitrd` and see if it's the culprit ?

Or try to reinstall the packages that were updated with `zypper dup`.

http://bugzilla.opensuse.org/show_bug.cgi?id=1181970 http://bugzilla.opensuse.org/show_bug.cgi?id=1181970#c39

Franck Bui fbui@suse.com changed:

What |Removed |Added ---------------------------------------------------------------------------- Status|IN_PROGRESS |RESOLVED Resolution|--- |WONTFIX

--- Comment #39 from Franck Bui fbui@suse.com --- Ok, closing then.

Sorry for not being able to track down your issue and thanks for your help/feedback you provided.