Thanks for the quick response, Bernhard! On Mi, Jul 31, 2013 at 11:37:45 +0200, Bernhard Voelker wrote:

Your commands above show that /var/tmp is not the mount point (but "/" is"). How does your strace look like?

raven:/ # strace -e rename,stat,lstat mv /var/tmp /var/tmp-orig rename("/var/tmp", "/var/tmp-orig") = -1 EBUSY (Device or resource busy) mv: cannot move ‘/var/tmp’ to ‘/var/tmp-orig’: Device or resource busy +++ exited with 1 +++ raven:/ #

In a plain installation, I can't imagine much other reasons for EBUSY. Did you play with some other restricting methods like SELinux etc.?

No. Everything is as it was on a fresh install.

Are there suspicious entries in /proc/self/mounts?

raven:/ # grep tmp /proc/self/mounts devtmpfs /dev devtmpfs rw,relatime,size=1530924k,nr_inodes=207269,mode=755 0 0 tmpfs /dev/shm tmpfs rw,relatime 0 0 tmpfs /run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,mode=755 0 0 tmpfs /tmp tmpfs rw,nosuid,nodev,noexec,relatime,size=524288k 0 0 tmpfs /var/lock tmpfs rw,nosuid,nodev,relatime,mode=755 0 0 tmpfs /var/run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0 raven:/ #

Does "lsof /var/tmp" or "fuser /var/tmp" show something, e.g. a process chroot-ed to /var/tmp?

Nothing chrooted: raven:/ # ls -ld /proc/*/root|grep -v ' /$' lrwxrwxrwx 1 root root 0 Jul 31 12:09 /proc/3309/root -> /var/lib/ntp lrwxrwxrwx 1 root root 0 Jul 31 12:11 /proc/3892/root -> /proc lsof and fuser show a lot of entries, but none of them are related directly to /var/tmp. In the 20 years I have been using linux, I have never seen that changing into a directory or keeping files open (even for writing) in a direcoty would prevent you from deleting/renaming that directory. Not even chroot would prevent renaming (I just tried with /var/lib/ntp shown above). While deletion could fail if deletion of some file or directory further down fails, I don't see any reason why a rename could possibly fail. And I strongly consider those semantics to be a feauture, not a bug. Just take a look over there to the windows world, where all applications need to be stopped every time you want to install or update something. The only reason I can see would be a mount. But there is nothing mounted on /var/tmp. -- Josef Wolf jw@raven.inka.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Wed, Jul 31, 2013 at 01:27:18PM +0200, Bernhard Voelker wrote:

On 07/31/2013 12:40 PM, Josef Wolf wrote:

...

The only reason I can see would be a mount. But there is nothing mounted on /var/tmp.

Agreed.

Last idea: the kernel returns EBUSY if it holds an internal reference to the directory. Is that a strange or special file system? I.e. something like BTRFS and 'var' being the name of a snapshot? (I've not played much with BTRFS yet).

It's a standard ext3 as the suse-installer defaults to it: raven:/ # cat /proc/mounts | grep " / " rootfs / rootfs rw 0 0 /dev/mapper/cr_sda8 / ext3 rw,relatime,data=ordered 0 0 raven:/ # Although playing with snapshots for backups (on ext4) is on my todo-list, I've not been on that route yet. I have installed 12.3 exactly the same way as I had installed the countless versions before it. I have a hand-written script of instructions that I follow when installing and I only derivate from it when changes in the opensuse installer force me to do so. After installation the machine is configured by a system similar to cfengine. In fact, this configuration system stumbled over this /var/tmp problem. So I am pretty sure that I have _not_ installed or configured anything different than in previous versions. How can I check whether SELinux might be a problem here? I've never been into SELinux yet... What about policykit? Isn't that something similar to SELinux? Please, can _you_ try to rename /var/tmp? -- Josef Wolf jw@raven.inka.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Wed, Jul 31, 2013 at 02:59:58PM +0200, Bernhard Voelker wrote:

On 07/31/2013 01:56 PM, Josef Wolf wrote:

...

Please, can _you_ try to rename /var/tmp?

It works here:

$ strace -e rename mv /var/tmp /var/tmp-orig rename("/var/tmp", "/var/tmp-orig") = 0 +++ exited with 0 +++

Strange

Hmm, your data is encrypted, isn't it?

...

/dev/mapper/cr_sda8 / ext3 rw,relatime,data=ordered 0 0

I've never used it, but possibly there's something wrong on that level or the underlying device.

Yes. I've been using LUKS for many years with different distros. Never had any problems.

Did you do an fsck on "/"?

Not yet. Can't shut down the system right now.

And a silly question: you are not in a virtualized system, are you?

No. I always use real systems. -- Josef Wolf jw@raven.inka.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Do, Aug 01, 2013 at 03:48:55 +0200, Carlos E. R. wrote:

Try restarting 'famd'.

famd is not installed here. Finally, I have found _one_ suspect: haveged keeps /var/tmp from beeing renamed, even though it don't access it: raven:/home/jw # lsof | grep haveged haveged 578 root cwd DIR 253,0 4096 2 / haveged 578 root rtd DIR 253,0 4096 2 / haveged 578 root txt REG 253,0 108320 739481 /usr/sbin/haveged haveged 578 root mem REG 253,0 1741576 171370 /lib/libc-2.17.so haveged 578 root mem REG 253,0 148206 171363 /lib/ld-2.17.so haveged 578 root 0r CHR 1,3 0t0 1034 /dev/null haveged 578 root 1u unix 0xf4450b80 0t0 9217 socket haveged 578 root 2u unix 0xf4450b80 0t0 9217 socket haveged 578 root 3u CHR 1,8 0t0 1038 /dev/random I find this to be a _very_ surprising fact. I'd never expect a daemon that gathers random data to lock _anything_ on _any_ filesystem. But there are some more suspects: rtkit-daemon udisksd polkitd upowerd Those seem to be managed by dbus. How can I disable/enable those services selectively to find out which of them is responsible? -- Josef Wolf jw@raven.inka.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Friday, 2013-08-02 at 13:06 +0200, Josef Wolf wrote:

...

On Do, Aug 01, 2013 at 03:48:55 +0200, Carlos E. R. wrote:

...

Try restarting 'famd'.

famd is not installed here.

How is that possible? :-o I thought it was "mandatory". (the package name is 'fam', not 'famd')

No 'famd?' installed on my 12.3 system here either. -- Per Jessen, Zürich (25.8°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fr, Aug 02, 2013 at 10:27:25 +0400, Andrey Borzenkov wrote:

The only case when rename returns EBUSY is when directory is mount point. systemd creates private tmp directories if service has PrivateTmp=yes:

Ugh, this seems to be hardwired to /var/tmp. No way to move it to /tmp or something?

bor@opensuse:~> grep -rlw PrivateTmp /usr/lib/systemd/system /usr/lib/systemd/system/rsyncd.service /usr/lib/systemd/system/pesign.service /usr/lib/systemd/system/haveged.service /usr/lib/systemd/system/rtkit-daemon.service

Yes, that seems to be the root of the problem.

bor@opensuse:~> systemctl status haveged haveged.service - Haveged Entropy Gathering Daemon Loaded: loaded (/usr/lib/systemd/system/haveged.service; enabled) Active: active (running) since Sat, 2013-07-27 11:13:45 MSK; 6 days ago Main PID: 672 (haveged) CGroup: name=systemd:/system/haveged.service └ 672 /usr/sbin/haveged -w 1024 -v 0 -F

bor@opensuse:~> sudo grep /var/tmp /proc/672/mountinfo 73 44 253:1 /var/tmp/systemd-private-xztGlQ /var/tmp rw,relatime shared:56 master:1 - ext4 /dev/mapper/system-root rw,commit=600,data=ordered bor@opensuse:~>

How comes that I don't see this in /proc/mounts? And lsof don't show it, too.

So /var/tmp cannot be renamed because it mounted - in one of namespaces.

But when I stop haveged, I _still_ can't rename, although grep /var/tmp /proc/*/mountinfo don't show anything. So there must be something else. -- Josef Wolf jw@raven.inka.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Mon, Aug 05, 2013 at 10:09:47PM +0400, Andrey Borzenkov wrote:

В Mon, 5 Aug 2013 15:15:38 +0200 Josef Wolf пишет:

...

On Fr, Aug 02, 2013 at 10:27:25 +0400, Andrey Borzenkov wrote:

...

The only case when rename returns EBUSY is when directory is mount point. systemd creates private tmp directories if service has PrivateTmp=yes:

Ugh, this seems to be hardwired to /var/tmp. No way to move it to /tmp or something?

Actually it creates private mounts for both.

The directory for those private mounts seems to be hardwired to /var/tmp. It don't seem to be possible to move it to /tmp? Do those directories really need to survive reboots? Those cryptic names don't seem to suggest this. Oh, and then there's the bug on cleaning /var/tmp/systemd-private-XXX. Isn't this an indicator that all this should be moved to /tmp?

...

...

bor@opensuse:~> sudo grep /var/tmp /proc/672/mountinfo 73 44 253:1 /var/tmp/systemd-private-xztGlQ /var/tmp rw,relatime shared:56 master:1 - ext4 /dev/mapper/system-root rw,commit=600,data=ordered bor@opensuse:~>

How comes that I don't see this in /proc/mounts? And lsof don't show it, too.

bind mounts are not shown in /proc/mounts.

This is not true: raven:/ # mkdir /tmp/t raven:/ # mount -obind / /tmp/t raven:/ # grep /tmp/t /proc/mounts /dev/mapper/cr_sda8 /tmp/t ext3 rw,relatime,data=ordered 0 0 raven:/ #

Partially because they are not really mounts as such. And nothing has any file open under /var/tmp (or rather /var/tmp/systemd-private-xxxxxxx) so it is of course not shown by lsof.

I still fail to understand why /var/tmp is locked down. Isn't /var/tmp/systemd-private-XXX the mount point? Thus this private directory should be locked down, not the containing directory. Maybe it would be possible, to move those private directories one level deeper: /tmp/systemd.d/private-XXX or something. That would have two benefits: - it would keep the /tmp directory clean. - the /tmp directory would not be locked down .

...

...

So /var/tmp cannot be renamed because it mounted - in one of namespaces.

But when I stop haveged, I _still_ can't rename, although

grep /var/tmp /proc/*/mountinfo

don't show anything. So there must be something else.

I tend to think it is kernel bug. It works if you do it without GUI (i.e. booting in run level 3) and testing it I managed to hard lock my system so I stopped it at this point :)

It is because the other three services are still running. When I stop all of them, I can rename even though the GUI is up. The wired thing here is that /proc/*/mountinfo and /proc/mounts won't show anything. -- Josef Wolf jw@raven.inka.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Tue, Aug 06, 2013 at 08:28:31PM +0400, Andrey Borzenkov wrote:

...

...

...

How comes that I don't see this in /proc/mounts? And lsof don't show it, too.

bind mounts are not shown in /proc/mounts.

This is not true:

raven:/ # mkdir /tmp/t raven:/ # mount -obind / /tmp/t raven:/ # grep /tmp/t /proc/mounts /dev/mapper/cr_sda8 /tmp/t ext3 rw,relatime,data=ordered 0 0 raven:/ #

a) you did not mount /dev/mapper/cr_sda8 on /tmp/t, did you? So you have no information about bind mounts there.

/dev/mapper/cr_sda8 _IS_ my root file system. It is resolved somehow by the mount operation. Nothing on this system would mount anything on /tmp/t. If I umount /tmp/t, this line disappears again.

b) /proc/mounts is alias for /proc/self/mounts which shows mount points in *your* namespace. The whole point of namespaces is to restrict visibility ...

So why is the directory in *my* namespace locked down although the mount is limited to *their* namespace?

...

I still fail to understand why /var/tmp is locked down. Isn't /var/tmp/systemd-private-XXX the mount point?

No, mount point is /var/tmp. Just like in "mount /dev/foo /var/tmp".

But it is the /var/tmp in *their* namespace. -- Josef Wolf jw@raven.inka.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Tue, Aug 06, 2013 at 01:46:37AM -0400, Cristian Rodríguez wrote:

Well, this is the expected behaviour, but anyway, if you get an updated haveged package you will see I removed PrivateTmp=yes from the unit file because it is not needed as this service does not create temporary files.

Thanks! -- Josef Wolf jw@raven.inka.de -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org