Hello, Am Freitag, 16. Februar 2018, 01:43:14 CET schrieb Jim Henderson:

Prior to the issue, I ran a 'zypper up', and now I have deleted files showing as open (owned by gdm):

jhenderson@lamuella:~> sudo zypper ps -s The following running processes use deleted files:

--- snip ---

PID | PPID | UID | User | Command | Service ------+-------+-----+------+----------------------+-------- 12484 | 12470 | 478 | gdm | gnome-session-binary | 12495 | 1 | 478 | gdm | at-spi-bus-launcher | 12646 | 1 | 478 | gdm | goa-daemon | 12664 | 1 | 478 | gdm | mission-control-5 |

--- snip ---

If I switch out of graphical mode, these files go away. If I switch *back* into graphical mode (effectively init 3; init 5), the in-use files *return*. If I completely reboot the system, I *still* see this output from zypper ps -s.

Is it possible the subvolumes are somehow hosed? If so, how do I fix them?

Maybe the listed processes create a tempfile and instantly delete it (while keeping it open and therefore useable for the process that opened it). Please check with zypper ps (without the -s) to see the filenames of the deleted files used by these processes. Regards, Christian Boltz -- Registrierter Linux-Nutzer #239431 Linux is like a wigwam: no gates, no windows, but an apache inside. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, 16 Feb 2018 23:59:02 +0000, Jim Henderson wrote:

On Sat, 17 Feb 2018 00:28:36 +0100, Christian Boltz wrote:

...

Maybe the listed processes create a tempfile and instantly delete it (while keeping it open and therefore useable for the process that opened it).

Please check with zypper ps (without the -s) to see the filenames of the deleted files used by these processes.

Thanks, Christian. The good news is that after rebooting into recovery, I ran a btrfs check; it didn't find any issues, but the issue with zypper ps -s showing in-use files seems to have cleared up. It was really strange that that issue persisted through a reboot earlier this week.

Well, it came back after running another zypper up. The files shown are: jhenderson@lamuella:~> sudo zypper ps The following running processes use deleted files: PID | PPID | UID | User | Command | Service | Files ------+-------+-----+------+----------------------+--------- +-------------------------------- 18849 | 18838 | 478 | gdm | gnome-session-binary | | /var/lib/ gdm/.config/dconf/user 18860 | 1 | 478 | gdm | at-spi-bus-launcher | | /var/lib/ gdm/.config/dconf/user 19023 | 1 | 478 | gdm | goa-daemon | | /var/lib/ gdm/.config/dconf/user 19041 | 1 | 478 | gdm | mission-control-5 | | /var/lib/ gdm/.config/dconf/user Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, 16 Feb 2018 20:04:51 -0500, Patrick Shanahan wrote:

* Jim Henderson [02-16-18 19:28]:

...

On Fri, 16 Feb 2018 23:59:02 +0000, Jim Henderson wrote:

...

On Sat, 17 Feb 2018 00:28:36 +0100, Christian Boltz wrote:

...

Maybe the listed processes create a tempfile and instantly delete it (while keeping it open and therefore useable for the process that opened it).

Please check with zypper ps (without the -s) to see the filenames of the deleted files used by these processes.

Thanks, Christian. The good news is that after rebooting into recovery, I ran a btrfs check; it didn't find any issues, but the issue with zypper ps -s showing in-use files seems to have cleared up. It was really strange that that issue persisted through a reboot earlier this week.

Well, it came back after running another zypper up.

The files shown are:

jhenderson@lamuella:~> sudo zypper ps The following running processes use deleted files:

PID | PPID | UID | User | Command | Service | Files ------+-------+-----+------+----------------------+--------- +-------------------------------- 18849 | 18838 | 478 | gdm | gnome-session-binary | | /var/lib/ gdm/.config/dconf/user 18860 | 1 | 478 | gdm | at-spi-bus-launcher | | /var/lib/ gdm/.config/dconf/user 19023 | 1 | 478 | gdm | goa-daemon | | /var/lib/ gdm/.config/dconf/user 19041 | 1 | 478 | gdm | mission-control-5 | | /var/lib/ gdm/.config/dconf/user

processes using deleted files are processes that continue running after you update those file with zypper. when you reboot, you clear the processes and they restart using the newly installed files rather than the deleted/replaced/upgraded files.

you will only notice running processes using deleted files after and update.

you do not need to reboot to clear the subject processes. many you can kill and restart, also restarting the graphic target will clear some. usually the only time a reboot is necessary is for a kernel update or a dbus (messagebus) update.

some knowledge of the particular processes is necessary to determine the proper procedure to clear it w/o rebooting.

Thanks, Patrick - I am aware of this, but I do appreciate your jumping in to help try to sort out what's going on. This isn't my first Linux rodeo (been running openSUSE and its predecessors since 2003, and Linux since the late 90s - used to build my own kernels back in the day). What I'm saying is that this happened: 1. I ran zypper up. 2. I ran zypper ps -s 3. Processes using deleted files showed up. 4. I stopped the graphical interface. 5. zypper ps -s showed the deleted files were no longer in use 6. I started the graphical interface. 7. zypper ps -s showed the deleted files were once again in use 8. I decided with the other btrfs issue that I reported here, that I would reboot the system. 9. After a full system reboot and starting back into the graphical interface, the deleted files that were in use were still somehow in use. After a reboot. I've been doing Linux for a while - and updating zypper with doing a proper system restart for years now - and ran 42.2 on one system with btrfs for the entire life of 42.2 without any issues. I'm completely stumped as to two things here: 1. Why 'snapper cleanup number' is throwing a quota error after it worked once, even though (as I understand it) quotas are disabled by default (and I have tried turning quotas off and back on again with no effect) 2. Why even after a reboot, various gdm-owned processes open deleted files instead of using the new files the way it's supposed to work. My assumption here is that something is hosed in a btrfs subvolume and these processes are accessing files from an earlier snapshot (or something along those lines). btrfs check doesn't show any problems. Nothing is logged that I can find. The only symptoms I see are the quota error running a snapper cleanup number and after running zypper up, these gdm-owned processes continue to use deleted files when they start up - even after a reboot. I'm totally stumped. I've tried rebuilding the qgroups, I've tried disabling quotas entirely, I've grepped through dmesg and journalctl outputs until I'm blue in the face. Nothing is actually getting me additional info about what's going on in this filesystem. Nothing appears corrupt to the tools, but the behaviour is like nothing I've ever seen in ~20 years of running Linux. By definition, if a file is deleted, a new process can't open the inode. Yet that's exactly what's happening here. I terminate the processes with an 'init 3', and zypper ps says no deleted files are in use. I run 'init 5' to switch back to the GUI, and the deleted files are *reopened*. I reboot the machine, and the deleted files are *reopened*. How does that happen? And does it have anything to do with the other weirdness in the filesystem that's causing snapper to not allow me to run a cleanup command - but only for the 'number' snapshots. I can delete those snapshots individually without a problem. Then I run zypper up, new snapshots are created, and the same behaviour returns - deleted files in use, stop the process, deleted files not in use. Start the process again, deleted files are *again* in use. Reboot the system - deleted files are *again* in use. It doesn't seem to *harm* the system (for now), but I'm tempted to reinstall again. I'd rather not do that - I'd rather understand what happened so I can prevent it from happening again, and if I've run into a defect, I'd like to report it so it can get fixed. Sorry for the long-winded answer. I'm just totally baffled about what's going on here. Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hello, Am Samstag, 17. Februar 2018, 03:43:37 CET schrieb Jim Henderson:

On Fri, 16 Feb 2018 20:04:51 -0500, Patrick Shanahan wrote:

...

* Jim Henderson [02-16-18 19:28]:

[zypper ps shows /var/lib/gdm/.config/dconf/user was deleted, and is still opened by several processes running as user "gdm"]

What I'm saying is that this happened:

1. I ran zypper up. 2. I ran zypper ps -s 3. Processes using deleted files showed up. 4. I stopped the graphical interface. 5. zypper ps -s showed the deleted files were no longer in use 6. I started the graphical interface. 7. zypper ps -s showed the deleted files were once again in use 8. I decided with the other btrfs issue that I reported here, that I would reboot the system. 9. After a full system reboot and starting back into the graphical interface, the deleted files that were in use were still somehow in use. After a reboot.

Without checking the code, I'm quite sure that the gdm processes open /var/lib/gdm/.config/dconf/user and then instantly delete it (while keeping the file open). This means the file becomes "invisible" on the filesystem, but still exists and is usable for the processes that have it open. Actually at this stage the file only gets unlinked (it no longer has a filename, and isn't visible in directory listings), but still exists on the filesystem. When the file gets closed, the disk space finally gets freed. Besides the obvious disadvantage of causing confusion, this has a few advantages that might make it attractive for developers to use this way: - automatic cleanup - as soon as the file gets closed (or the process that opened it crashes), the file gets finally removed / the disk space gets freed on the filesystem. - no other process can read or modify the file (that's not entirely true because it's still available as /proc/$pid/fd/$number - but at least that makes it harder to access a deleted file) I'll give you a short demo: #!/bin/bash tempfile="$(mktemp)" || { echo 'mktemp failure'; exit 1; } echo 'foo bar baz' > "$tempfile" cat "$tempfile" | while read line ; do ls -l "$tempfile" rm -v "$tempfile" echo $line done cat opens $tempfile and keeps it open. In the first run of the loop $tempfile gets deleted, but cat still has an open filehandle on it and can successfully read the file content of the deleted file. The script (intentionally) prints some error messages for rm and ls (when the file no longer exists), but between them, you'll see that it prints out foo, bar and baz.

I've been doing Linux for a while - and updating zypper with doing a proper system restart for years now - and ran 42.2 on one system with btrfs for the entire life of 42.2 without any issues.

I'm completely stumped as to two things here:

1. Why 'snapper cleanup number' is throwing a quota error after it worked once, even though (as I understand it) quotas are disabled by default (and I have tried turning quotas off and back on again with no effect)

2. Why even after a reboot, various gdm-owned processes open deleted files instead of using the new files the way it's supposed to work. My assumption here is that something is hosed in a btrfs subvolume and these processes are accessing files from an earlier snapshot (or something along those lines).

I don't have an answer for your btrfs issues, but I'm sure it's unrelated to the gdm using deleted files.

By definition, if a file is deleted, a new process can't open the inode.

Right, but the new process can create a new file, open it, and then delete it.

It doesn't seem to *harm* the system (for now), but I'm tempted to reinstall again. I'd rather not do that - I'd rather understand what happened so I can prevent it from happening again, and if I've run into a defect, I'd like to report it so it can get fixed.

I hope the above makes it clear what happens - if not, feel free to ask again. Regards, Christian Boltz -- <cboltz> jjohansen: we can just label it "the can't be more broken than 2.8.3 release" ;-) <sbeattie> cboltz: I'm disappointed in your inability to believe that I can't screw things up further. [from #apparmor] -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, 16 Feb 2018 20:04:51 -0500, Patrick Shanahan wrote:

you do not need to reboot to clear the subject processes. many you can kill and restart, also restarting the graphic target will clear some. usually the only time a reboot is necessary is for a kernel update or a dbus (messagebus) update.

Unrelated to my issue - but on 42.2, I never had to reboot for kernel updates, or for dbus updates. It looked like live patching was included there, but that doesn't seem to be the case with 42.3. With dbus, if you shut down the graphical target, restart dbus, and then restart systemd-logind, you can restart the graphical target and things will continue to run. That's been my go-to procedure for a dbus restart, and it's worked for me for a while now (can't remember when I figured out that it was systemd-logind that needed to be restarted afterwards - or even how I made that determination). Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Wed, 21 Feb 2018 13:15:29 +1000, Konstantin Voinov wrote:

On 2018-02-17 12:46, Jim Henderson wrote:

...

Unrelated to my issue - but on 42.2, I never had to reboot for kernel updates, or for dbus updates. It looked like live patching was included there, but that doesn't seem to be the case with 42.3.

No, it was same rpm install/upgrade.

Huh, interesting. I could've sworn that kernel updates in 42.2 didn't require a reboot on my system. Oh well, now I know better. (I suppose I could reinstall the old hard drive that I replaced when I moved to 42.3, if I could be bothered - but it's not that important.) Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Sat, 17 Feb 2018 10:45:46 +0300, Andrei Borzenkov wrote:

zypper ps sumply checks for files marked as "deleted". It is possible that this file is somehow changed since processes opened it. I do not have 42.3 with GDM, on TW I do not see this.

I have 3 42.3 systems using gdm, and only one of them does this.

...

1. I ran zypper up. 2. I ran zypper ps -s 3. Processes using deleted files showed up. 4. I stopped the graphical interface. 5. zypper ps -s showed the deleted files were no longer in use 6. I started the graphical interface. 7. zypper ps -s showed the deleted files were once again in use

OK, so

check file details at step 3 (ls -li, probably using ISO time format - with milliseconds)

356499 -rw-r--r-- 1 gdm gdm 711 Feb 16 16:24 /var/lib/gdm/.config/dconf/ user

Check file details again at step 4 - are they the same?

356499 -rw-r--r-- 1 gdm gdm 711 Feb 16 16:24 /var/lib/gdm/.config/dconf/ user Yep.

Check file details again at step 6 - are they still the same?

357735 -rw-r--r-- 1 gdm gdm 711 Feb 17 12:13 /var/lib/gdm/.config/dconf/ user Nope, it changed. Just checked on my main desktop, which also runs 42.3, and those gdm processes definitely don't show the file as deleted. One of the running processes is gnome-shell, so I ran an rpm -Vv on it, and it doesn't show any changes from the default package contents - just in case it was something that got corrupted. -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Sun, 18 Feb 2018 22:05:44 +0300, Andrei Borzenkov wrote:

17.02.2018 23:16, Jim Henderson wrote:

...

On Sat, 17 Feb 2018 10:45:46 +0300, Andrei Borzenkov wrote:

...

zypper ps sumply checks for files marked as "deleted". It is possible that this file is somehow changed since processes opened it. I do not have 42.3 with GDM, on TW I do not see this.

I have 3 42.3 systems using gdm, and only one of them does this.

...

...

1. I ran zypper up. 2. I ran zypper ps -s 3. Processes using deleted files showed up. 4. I stopped the graphical interface. 5. zypper ps -s showed the deleted files were no longer in use 6. I started the graphical interface. 7. zypper ps -s showed the deleted files were once again in use

OK, so

check file details at step 3 (ls -li, probably using ISO time format - with milliseconds)

356499 -rw-r--r-- 1 gdm gdm 711 Feb 16 16:24 /var/lib/gdm/.config/dconf/ user

...

Check file details again at step 4 - are they the same?

356499 -rw-r--r-- 1 gdm gdm 711 Feb 16 16:24 /var/lib/gdm/.config/dconf/ user

Yep.

...

Check file details again at step 6 - are they still the same?

357735 -rw-r--r-- 1 gdm gdm 711 Feb 17 12:13 /var/lib/gdm/.config/dconf/ user

Nope, it changed.

I actually observed something similar on TW (I intentionally looked for this file). Some processes did hold reference to apparently deleted and recreated /var/ilb/gdm/.config/dconf/user. So it seems normal and does not indicate filesystem corruption.

...

Just checked on my main desktop, which also runs 42.3, and those gdm processes definitely don't show the file as deleted.

This likely requires die hard GNOME developer to dig into details what exactly triggers it.

...

One of the running processes is gnome-shell, so I ran an rpm -Vv on it, and it doesn't show any changes from the default package contents - just in case it was something that got corrupted.

OK, I'll consider this an unrelated issue to the btrfs issue then - thanks. Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Sun, 18 Feb 2018 13:23:33 -0500, James Knott wrote:

Why don't people ever have even problems? ;-)

I suppose right now, you could say I do - one with btrfs and one with gdm- owned processes using deleted files after a restart. :) Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org