Re: [opensuse] back to the 'fix this eternally broken temp file cleanup service' debate
On 19/02/15 16:56, Joe Zappa wrote:
Peter wrote:
Bonjour.
This is now driving me mad. Just over a year ago (I had to go hunt down the thread, here: http://lists.opensuse.org/opensuse/2014-01/msg00988.html) the issue was once again raised of why temp files were not being cleaned at boot (on openSUSE 13.1) if one tried to set the system to do so.
It was established that the cron job configurable from YaST's sysconfig editor was no longer functional since supposedly being replaced by the all-round saviour and tuck-your-sheets-in-would-you-like-a-cup-of-hot-cocoa-with-that princess that is systemd. Except that the replacement doesn't work. At least not on any of my 13.1 installations, whereupon both /tmp and /var/tmp become laden with relics that outright refuse to ever part my system, gradually filling my root partition until there is no more room for maneouvre. Currently my /tmp is sitting at 5GB. Looking at files in there, many are up to a year old. It seems to just accumulate stuff like an eccentric lonely old man with an attic. This was a fresh 13.1 installation made about a year ago. The root partition (ext4) is 17GB, about half of which was filled after installation. Now there's little left.
It was also established that SuSE likes to do its SuSEy thing and go its own way, against systemd defaults, in preserving temp files across reboots, since some SuSErs want their systems configured that way and consider /tmp to be a persistent directory.
I've just given up
Now, when I reboot my machines, I log into root and run a shellscript that I call "start" which I wrote to do any and all initializations which I USED to put in in /etc/init.d/local.initializations and then link to from rc{2,3,5}.d directories as S99local.initalizations.
But since systemd completely and utterly fucking ignores /etc/init.d/*, that's out the window.
So is /tmp filling interminably for all users of 13.1 who haven't otherwise fiddled with something to prevent this, or who don't have /tmp stored in RAM? I don't understand if this is a universal issue/bug or it's only affecting certain users, such as those not using btrfs for example. Google searches aren't throwing up much in the way of openSUSE, only similar issues affecting various other distros. I don't know what more I can do to empty my /tmp and /var/tmp aside from simply rm -rf /tmp. I'm reluctant to do that as a last resort because I had problems long ago on this same system after emptying temp, although admittedly I stupidly did that whilst running as user with x. Doing some more plugging I found that systemctl -l status systemd-tmpfiles-clean.service was throwing up a failed message concerning gvfs. Thinking this might be preventing the service from running, I found a thread on the Arch bug tracker (https://bugs.archlinux.org/task/32715) which suggests a workaround for this being to create a file, /etc/tmpfiles.d/gvfs.conf containing, essentially: x /run/user/* to stop systemd going near /run/user at all. Well that changed nothing on my system. I've tried manually running systemd-tmpfiles --clean --remove but again, though it doesn't return an error, nothing is cleaned up or removed. I also notice when running systemctl -l status systemd-tmpfiles-clean.service the lines: Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.service; static) Active: inactive (dead) It seems to always be inactive. Meanwhile, running systemctl -l status systemd-tmpfiles-setup.service I get this: systemd-tmpfiles-setup.service - Recreate Volatile Files and Directories Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static) Active: active (exited) since Sat 2015-02-21 20:17:27 CET; 2min 4s ago Docs: man:tmpfiles.d(5) man:systemd-tmpfiles(8) Process: 356 ExecStart=/usr/bin/systemd-tmpfiles --create --remove --exclude-prefix=/dev (code=exited, status=0/SUCCESS) Main PID: 356 (code=exited, status=0/SUCCESS) And I've no idea how to go about setting /tmp to store files for 30 days and /var/tmp for 90 like in the good old days, because there's clearly not much point even tinkering with that idea until I can get the service to run at all. The documentation (man page) for systemd-tmpfiles is awful and tells me nothing useful. I still have 2.5GB to fill on my root partition. I'll keep waiting. Before long I'll have to do something. Peter -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 21/02/15 19:55, Peter wrote:
So is /tmp filling interminably for all users of 13.1 who haven't otherwise fiddled with something to prevent this, or who don't have /tmp stored in RAM? I don't understand if this is a universal issue/bug or it's only affecting certain users, such as those not using btrfs for example. Google searches aren't throwing up much in the way of openSUSE, only similar issues affecting various other distros.
I don't know what more I can do to empty my /tmp and /var/tmp aside from simply rm -rf /tmp. I'm reluctant to do that as a last resort because I had problems long ago on this same system after emptying temp, although admittedly I stupidly did that whilst running as user with x.
It affects me and three friends I give tech support to over about 8 machines. I manually empty /tmp and /var/tmp as part of my weekly maintenance round. I've not had any problem deleting the files when logged in as a user in x (KDE). I use midnight commander in an xterm to delete the contents. Dx -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Peter wrote:
So is /tmp filling interminably for all users of 13.1 who haven't otherwise fiddled with something to prevent this, or who don't have /tmp stored in RAM?
Nope, I don't have that problem.
I don't understand if this is a universal issue/bug or it's only affecting certain users, such as those not using btrfs for example. Google searches aren't throwing up much in the way of openSUSE, only similar issues affecting various other distros.
If you look at the contents of /tmp, can't you tell what it is that's filling it up?
And I've no idea how to go about setting /tmp to store files for 30 days and /var/tmp for 90 like in the good old days, because there's clearly not much point even tinkering with that idea until I can get the service to run at all. The documentation (man page) for systemd-tmpfiles is awful and tells me nothing useful.
from man systemd-tmpfiles : "systemd-tmpfiles creates, deletes and cleans up volatile and temporary files and directories, based on the configuration file format and location specified in tmpfiles.d(5)." man tmpfiles.d The configuration format is one line per path containing action, path, mode, ownership, age and argument fields: Type Path Mode UID GID Age Argument d /run/user 0755 root root 10d - L /tmp/foobar - - - - /dev/null LIke I said, I don't have thjis problem, but the man pages don't seem that bad. (my /tmp/tmpfiles.d is empty though).
I still have 2.5GB to fill on my root partition. I'll keep waiting. Before long I'll have to do something.
find <DIRS> -type f -mtime +365 | xargs rm -- Per Jessen, Zürich (0.0°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
Like I said, I don't have this problem, but the man pages don't seem that bad. (my /tmp/tmpfiles.d is empty though).
That should be /etc/tmpfiles.d -- Per Jessen, Zürich (0.0°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-02-22 10:37, Per Jessen wrote:
Peter wrote:
So is /tmp filling interminably for all users of 13.1 who haven't otherwise fiddled with something to prevent this, or who don't have /tmp stored in RAM?
Nope, I don't have that problem.
I do. Not that big, but I have files or directories in /tmp going back to 2010. Well, two files are dated 1970, but those dates are obviously wrong.
If you look at the contents of /tmp, can't you tell what it is that's filling it up?
Irrelevant. Old files should be automatically deleted, no matter what created them. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlTum/UACgkQja8UbcUWM1yy5gD/RNF6TdqKr1yMhu8g1wJGf6CQ xWubAEnM8xsbDV13wxQA/1zzgmSy9pmD4MGPuTt8MfGbc5bsYqEaPaV5GgwNiCeE =XjcU -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-02-22 10:37, Per Jessen wrote:
Peter wrote:
So is /tmp filling interminably for all users of 13.1 who haven't otherwise fiddled with something to prevent this, or who don't have /tmp stored in RAM?
Nope, I don't have that problem.
I do.
Not that big, but I have files or directories in /tmp going back to 2010. Well, two files are dated 1970, but those dates are obviously wrong.
If you look at the contents of /tmp, can't you tell what it is that's filling it up?
Irrelevant. Old files should be automatically deleted, no matter what created them.
I beg to differ. Where they come from is not irrelevant - the tmp files should be removed by whatever created them. Any automatic deletion is really just a band-aid. Also, when some people have the problem and others don't, there is most probably also a difference in usage. Hence it makes sense to look at the contents and who created it. -- Per Jessen, Zürich (0.6°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On February 26, 2015 12:32:01 AM PST, Per Jessen <per@computer.org> wrote:
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-02-22 10:37, Per Jessen wrote:
Peter wrote:
So is /tmp filling interminably for all users of 13.1 who haven't otherwise fiddled with something to prevent this, or who don't have /tmp stored in RAM?
Nope, I don't have that problem.
I do.
Not that big, but I have files or directories in /tmp going back to 2010. Well, two files are dated 1970, but those dates are obviously wrong.
If you look at the contents of /tmp, can't you tell what it is that's filling it up?
Irrelevant. Old files should be automatically deleted, no matter what created them.
I beg to differ. Where they come from is not irrelevant - the tmp files should be removed by whatever created them. Any automatic deletion is really just a band-aid. Also, when some people have the problem and others don't, there is most probably also a difference in usage. Hence it makes sense to look at the contents and who created it.
Maybe in s perfect world, perhaps. But tmp has always been for temporary files, and software was always written with the understanding that the system would clean these spaced, and therefore these things could be abandoned. Its been that way since the Pleistoscene. Tmp historically wsd cleared at boot up. When system we reliable enough to run for months, purges on schedule were added. Opensuse took the unprecedented step of breaking that rule, unilaterally. And the users are now left cleaning up the mess. If it is, as you unilaterally declare, up to the file creator to clean up every file in tmp, then by all means, lets start by having Opensuse clean up THEIR files first. Personally I think it was just laziness on opensuse's part. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-02-26 09:32, Per Jessen wrote:
Carlos E. R. wrote:
If you look at the contents of /tmp, can't you tell what it is that's filling it up?
Irrelevant. Old files should be automatically deleted, no matter what created them.
I beg to differ. Where they come from is not irrelevant - the tmp files should be removed by whatever created them. Any automatic deletion is really just a band-aid. Also, when some people have the problem and others don't, there is most probably also a difference in usage. Hence it makes sense to look at the contents and who created it.
It is beyond the power of the openSUSE people to correct _all_ those pieces of software that don't delete their temporary files after use, which is why we had, since I remember (ie, 1998), a cron job to erase them on boot, or when too old, or in both cases. That cron job has been superseded by a systemd job, which doesn't (yet?) do a good job; perhaps because systemd people recommend using tmpfs — but still, old files are a problem on a system that seldom boots, tmpfs or not. So we need something to do the job properly, either by somehow configuring systemd, or by hacking a script that does it (no, upgrading systemd is not an option for people like me, using 13.1). However, if you are curious about what I have in /tmp, there are several types. Some directories have a fixed name, with content reused or replaced, I suppose: drwx------ 2 pepe users 12288 Jan 1 1970 orbit-pepe drwx------ 2 cer-g users 4096 Jan 1 1970 orbit-cer-g drwx------ 2 cer users 139264 Jan 1 1970 orbit-cer Many have random names: drwx------ 2 cer users 4096 Feb 23 2010 virtual-cer.OVXtjx drwx------ 2 root root 4096 Feb 23 2010 virtual-root.O4e0Gh drwx------ 2 root root 4096 Feb 23 2010 libstorage-5noZrW drwx------ 2 root root 4096 Feb 23 2010 libstorage-3Dkk8U drwx------ 2 cer users 4096 Feb 23 2010 virtual-cer.eMugfA drwx------ 2 cer users 4096 Feb 24 2010 virtual-cer.cod7Od drwx------ 2 cer users 4096 Feb 24 2010 virtual-cer.Sr15pb drwx------ 2 cer users 4096 Feb 24 2010 seahorse-eAtHl6 drwx------ 2 cer users 4096 Feb 24 2010 virtual-cer.NbJt6u drwx------ 2 cer users 4096 Feb 24 2010 seahorse-JCGYv7 drwx------ 2 cer users 4096 Feb 24 2010 keyring-UtN8MA drwx------ 2 cer users 4096 Feb 24 2010 virtual-cer.b66e3r drwx------ 2 cer users 4096 Feb 24 2010 seahorse-RjM80L drwx------ 2 cer users 4096 Feb 24 2010 keyring-vvXqmI drwx------ 2 cer users 4096 Feb 24 2010 virtual-cer.Ng3aYS - -rw------- 1 root root 105 Feb 24 2010 yast2-err-Hj4mgn drwx------ 2 root root 4096 Feb 24 2010 YaST2-07563-Emv6Lw drwx------ 2 root root 4096 Feb 24 2010 YaST2-07563-IXjOtP - -rw------- 1 root root 271 Feb 24 2010 yast2-err-iANErN So, you see they are created by several programs — including YaST. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlT04hoACgkQja8UbcUWM1wMJgD/V8wH5H2W6VegQ3TcPaEpXnv+ V32XlUlC7k7dLrE00cMA/j5IIUji4rrNsPUTsko202g7xNHtEcXP1CYdiVUfTK/e =bjIJ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/21/2015 02:55 PM, Peter wrote:
I still have 2.5GB to fill on my root partition. I'll keep waiting. Before long I'll have to do something.
My /tmp fills and its annoying, but the /etc/tmpfile.d etnry is a misdirtection. You need to look in /usr/lib/tmpfiles.d/tmp.conf which is what is used by default. Yes it has # Clear tmp directories separately, to make them easier to override # SUSE policy: we don't clean those directories d /tmp 1777 root root - d /var/tmp 1777 root root - which isn't what we want and so we try creating lines in /etc/tmpfiles.d But that is compeltely beside the point. The lines that matter are: # Exclude namespace mountpoints created with PrivateTmp=yes x /tmp/systemd-private-* x /var/tmp/systemd-private-* X /tmp/systemd-private-*/tmp X /var/tmp/systemd-private-*/tmp It means file with that pattern are kept eternally! -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 02/21/2015 02:55 PM, Peter wrote:
I still have 2.5GB to fill on my root partition. I'll keep waiting. Before long I'll have to do something.
My /tmp fills and its annoying, but the /etc/tmpfile.d etnry is a misdirtection.
You need to look in /usr/lib/tmpfiles.d/tmp.conf which is what is used by default.
Doesn't sound like the right place for config files? I wonder if /etc/tmpfiles.d is read in addition. -- Per Jessen, Zürich (4.8°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/22/2015 09:39 AM, Per Jessen wrote:
Anton Aylward wrote:
On 02/21/2015 02:55 PM, Peter wrote:
I still have 2.5GB to fill on my root partition. I'll keep waiting. Before long I'll have to do something.
My /tmp fills and its annoying, but the /etc/tmpfile.d etnry is a misdirtection.
You need to look in /usr/lib/tmpfiles.d/tmp.conf which is what is used by default.
Doesn't sound like the right place for config files? I wonder if /etc/tmpfiles.d is read in addition.
Irt seems that the /usr/lib are the installed default and the /etc are the over-ride/localization <quote src=:Man 5 tmpfiles.d:> CONFIGURATION FORMAT Each configuration file shall be named in the style of <package>.conf. Files in /etc/ override files with the same name in /usr/lib/ and /run/. Files in /run/ override files with the same name in /usr/lib/. Packages should install their configuration files in /usr/lib/. Files in /etc/ are reserved for the local administrator, who may use this logic to override the configuration files installed by vendor packages. </quote> -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Anton Aylward <opensuse@antonaylward.com> [02-22-15 08:51]: [...]
# Clear tmp directories separately, to make them easier to override # SUSE policy: we don't clean those directories d /tmp 1777 root root - d /var/tmp 1777 root root -
which isn't what we want and so we try creating lines in /etc/tmpfiles.d But that is compeltely beside the point.
The lines that matter are:
# Exclude namespace mountpoints created with PrivateTmp=yes x /tmp/systemd-private-* x /var/tmp/systemd-private-* X /tmp/systemd-private-*/tmp X /var/tmp/systemd-private-*/tmp
The tmp.conf in /etc/tmpfiles.d/ is for maintaining config over updates and is the last/final word. Some experimenting reveals a possible solution. It appears that there is no default time or it is beyond expectations. The following change helps: d /tmp 1777 root root 2d - d /var/tmp 1777 root root 2d - note the added "2d" and it's position. This is described in the man file but it's usage not clear until it is correctly applied and just works. One thing I still do not understand is the cleaning of files in directories marked "PrivateTmp=yes", nor can I find an effective override. Files below those directories *are* cleared/removed/cleaned, but not the directories. Seems this would eventially cause a problem. ps: I set my "tmp" cleaning to four hours: d /tmp 1777 root root 4d - d /var/tmp 1777 root root 4d - -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/22/2015 07:46 AM, Patrick Shanahan wrote:
This is described in the man file but it's usage not clear
Which man page??? until it is
correctly applied and just works. One thing I still do not understand is the cleaning of files in directories marked "PrivateTmp=yes", nor can I find an effective override. Files below those directories *are* cleared/removed/cleaned, but not the directories. Seems this would
Is that what /etc/tmpdirs.d is for perhaps?
eventially cause a problem.
ps: I set my "tmp" cleaning to four hours: d /tmp 1777 root root 4d - d /var/tmp 1777 root root 4d -
Is that hours or days? -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* John Andersen <jsamyth@gmail.com> [02-22-15 14:34]:
On 02/22/2015 07:46 AM, Patrick Shanahan wrote:
Seems you have been somewhat overly aggressive in trimming. My post is: <quote> The tmp.conf in /etc/tmpfiles.d/ is for maintaining config over updates and is the last/final word. Some experimenting reveals a possible solution. It appears that there is no default time or it is beyond expectations. The following change helps: d /tmp 1777 root root 2d - d /var/tmp 1777 root root 2d - note the added "2d" and it's position. This is described in the man file but it's usage not clear until it is correctly applied and just works. One thing I still do not understand is the cleaning of files in directories marked "PrivateTmp=yes", nor can I find an effective override. Files below those directories *are* cleared/removed/cleaned, but not the directories. Seems this would eventially cause a problem. ps: I set my "tmp" cleaning to four hours: d /tmp 1777 root root 4d - d /var/tmp 1777 root root 4d - </quote>
This is described in the man file but it's usage not clear
Which man page???
man tempfiles.d <quote> Age The date field, when set, is used to decide what files to delete when cleaning. If a file or directory is older than the current time minus the age field, it is deleted. The field format is a series of integers each followed by one of the following postfixes for the respective time units: s, min, h, d, w, ms, m, us If multiple integers and units are specified, the time values are summed up. If an integer is given without a unit, s is assumed. When the age is set to zero, the files are cleaned unconditionally. The age field only applies to lines starting with d, D, and x. If omitted or set to "-", no automatic clean-up is done. </quote> Effectively aiui, the default, "-" precludes automatic cleaning.
until it is correctly applied and just works. One thing I still do not understand is the cleaning of files in directories marked "PrivateTmp=yes", nor can I find an effective override. Files below those directories *are* cleared/removed/cleaned, but not the directories. Seems this would
Is that what /etc/tmpdirs.d is for perhaps?
Not from my reading and understanding, or not. /etc/tmpdirs.d/tmp.conf is for persistance and overriding default supplied and to preclude changing your desires when an updated package is installed, similar to using ignore.local in /etc/logdigest/.
eventially cause a problem.
ps: I set my "tmp" cleaning to four hours: d /tmp 1777 root root 4d - d /var/tmp 1777 root root 4d -
Is that hours or days?
That is days "d", but was intended hours "h". Typo here, too much hurry. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/22/2015 11:57 AM, Patrick Shanahan wrote:
The age field only applies to lines starting with d, D, and x. If omitted or set to "-", no automatic clean-up is done.
I noticed you added a days field, but left the trailing "-" Does that still work? -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* John Andersen <jsamyth@gmail.com> [02-22-15 15:04]:
On 02/22/2015 11:57 AM, Patrick Shanahan wrote:
The age field only applies to lines starting with d, D, and x. If omitted or set to "-", no automatic clean-up is done.
I noticed you added a days field, but left the trailing "-" Does that still work?
Yes, memory serves the "-" was a placeholder but I see from the docs it is effectively a null. It works-for-me with it, didn't try w/o. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/22/2015 07:46 AM, Patrick Shanahan wrote:
One thing I still do not understand is the cleaning of files in directories marked "PrivateTmp=yes", nor can I find an effective override. Files below those directories *are* cleared/removed/cleaned, but not the directories. Seems this would eventially cause a problem.
It seems that you will need the r line type for that.
r Remove a file or directory if it exists. This may not be used to remove non-empty directories, use R for that. Lines of this type accept shell-style globs in place of normal path names.
Since most of the temporary directories are created with reasonably descriptive names, ( gpg-xxxxxxxx ssh-xxxxxxxxxx) you can glob the deletion of these, and since it won't delete non empty directories it should be safe as long as you glob it intelligently. Its also possible that the cleaning of the directories is tripping you up, because deleting files within them changes the date on the directory. It might be a self solving problem given a few days or (hours in your case. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* John Andersen <jsamyth@gmail.com> [02-22-15 16:03]:
On 02/22/2015 07:46 AM, Patrick Shanahan wrote:
One thing I still do not understand is the cleaning of files in directories marked "PrivateTmp=yes", nor can I find an effective override. Files below those directories *are* cleared/removed/cleaned, but not the directories. Seems this would eventially cause a problem.
It seems that you will need the r line type for that.
r Remove a file or directory if it exists. This may not be used to remove non-empty directories, use R for that. Lines of this type accept shell-style globs in place of normal path names.
Since most of the temporary directories are created with reasonably descriptive names, ( gpg-xxxxxxxx ssh-xxxxxxxxxx) you can glob the deletion of these, and since it won't delete non empty directories it should be safe as long as you glob it intelligently.
Its also possible that the cleaning of the directories is tripping you up, because deleting files within them changes the date on the directory. It might be a self solving problem given a few days or (hours in your case.
Will reserve judgement until observation offers an intelligent direction. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/22/2015 09:46 AM, Patrick Shanahan wrote:
* Anton Aylward<opensuse@antonaylward.com> [02-22-15 08:51]: [...]
# Clear tmp directories separately, to make them easier to override # SUSE policy: we don't clean those directories d /tmp 1777 root root - d /var/tmp 1777 root root -
which isn't what we want and so we try creating lines in /etc/tmpfiles.d But that is compeltely beside the point.
The lines that matter are:
# Exclude namespace mountpoints created with PrivateTmp=yes x /tmp/systemd-private-* x /var/tmp/systemd-private-* X /tmp/systemd-private-*/tmp X /var/tmp/systemd-private-*/tmp The tmp.conf in/etc/tmpfiles.d/ is for maintaining config over updates and is the last/final word. Some experimenting reveals a possible solution. It appears that there is no default time or it is beyond expectations. The following change helps:
d /tmp 1777 root root 2d - d /var/tmp 1777 root root 2d -
note the added "2d" and it's position. This is described in the man file but it's usage not clear until it is correctly applied and just works. One thing I still do not understand is the cleaning of files in directories marked "PrivateTmp=yes", nor can I find an effective override. Files below those directories*are* cleared/removed/cleaned, but not the directories. Seems this would eventially cause a problem.
Intrigued by the issue, I thought I would offer another, albeit non-authoritative, data point on this matter. My x86-64 13.1 install has been running 24/7 for the better part of a year with heavy use made of the /tmp dir. Currently it contains (virtually nothing): [08:52 alchemy:.../scr/tmp/stack] # du -hcs /tmp 2.3M /tmp 2.3M total Both the following are empty: /etc/tmpdirs.d/ /etc/tmpfiles.d/ The only relevant config found is: /usr/lib/tmpfiles.d/tmp.conf with content (without comment): d /tmp 1777 root root - d /var/tmp 1777 root root - x /tmp/systemd-private-* x /var/tmp/systemd-private-* X /tmp/systemd-private-*/tmp X /var/tmp/systemd-private-*/tmp Checking the service shows the last run of "Cleanup of Temporary Directories" occurred Feb 22 22:07:46 (yes, I get the gvfs failure as well, it is irrelevant to whether /tmp is cleaned, and simply indicates gvfs was NOT mounted at the time the cleanup was run [check /var/run/user/UID]): # systemctl status systemd-tmpfiles-clean.service systemd-tmpfiles-clean.service - Cleanup of Temporary Directories Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.service; static) Active: inactive (dead) since Sun 2015-02-22 22:07:46 CST; 11h ago Docs: man:tmpfiles.d(5) man:systemd-tmpfiles(8) Process: 13040 ExecStart=/usr/bin/systemd-tmpfiles --clean (code=exited, status=0/SUCCESS) Main PID: 13040 (code=exited, status=0/SUCCESS) Feb 22 22:07:46 alchemy systemd-tmpfiles[13040]: stat(/run/user/1000/gvfs) failed: Permission denied Feb 22 22:07:46 alchemy systemd[1]: Started Cleanup of Temporary Directories. A quick check of the journal from last boot shows, at least on my system, the cleanup of /tmp is called daily: # journalctl -u systemd-tmpfiles-clean.service -- Logs begin at Sun 2015-02-15 21:52:53 CST, end at Mon 2015-02-23 09:14:09 CST. -- Feb 15 22:07:46 alchemy systemd[1]: Starting Cleanup of Temporary Directories... Feb 15 22:07:46 alchemy systemd-tmpfiles[1876]: stat(/run/user/1000/gvfs) failed: Permission denied Feb 15 22:07:47 alchemy systemd[1]: Started Cleanup of Temporary Directories. Feb 16 22:07:46 alchemy systemd[1]: Starting Cleanup of Temporary Directories... Feb 16 22:07:46 alchemy systemd-tmpfiles[14082]: stat(/run/user/1000/gvfs) failed: Permission denied Feb 16 22:07:46 alchemy systemd[1]: Started Cleanup of Temporary Directories. Feb 17 22:07:46 alchemy systemd[1]: Starting Cleanup of Temporary Directories... Feb 17 22:07:46 alchemy systemd-tmpfiles[18255]: stat(/run/user/1000/gvfs) failed: Permission denied Feb 17 22:07:46 alchemy systemd[1]: Started Cleanup of Temporary Directories. <snip...> So for all practical purposes, the unit systemd-tmpfiles-clean.service is running on my box daily at 22:07 (which is the time of last boot after return from San Antonio on the 15th). The service file looks for which files to clean in: ConditionDirectoryNotEmpty=|/usr/lib/tmpfiles.d ConditionDirectoryNotEmpty=|/usr/local/lib/tmpfiles.d ConditionDirectoryNotEmpty=|/etc/tmpfiles.d ConditionDirectoryNotEmpty=|/run/tmpfiles.d of which the only non-empty directory on my system is: /usr/lib/tmpfiles.d and the only relevant config is /usr/lib/tmpfiles.d/tmp.conf (contents posted above) So my only guess is that these are the pieces of the puzzle that are supposed to insure tmpfiles are cleaned. Checking the journal will tell you if they are running. If the only entry in the journal is the gvfs failure without the Starting/Started Cleanup of Temporary Directories..., then figuring out why they are not running will likely solve the issue. I don't recall doing anything strange/special on install regarding tmpfiles, although I do go though each config setting and I DO set a traditional 'root' account and password. I've always been leery about not using a traditional root account just for the reason that some hooked process that relies on being run by root -- won't -- in the absence of a traditional root account. I doubt it is related. Regardless, here is one data-point among others where the cleaning of tmpfiles seems to be working as it should. From the original post, it appears Peter has all parts of the cleanup configuration discussed above. The only additional checks I see are to check the journal and find out just what is being run (and when) and go from there. Further, there should be a way to manually kick-off the cleaning of tmpfiles. I would figure that out and attempt to force a cleaning and check the results/errors there. That will provide a further bit of info. Good luck. (SysVinit was so much cleaner...) -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
В Mon, 23 Feb 2015 09:48:57 -0600 "David C. Rankin" <drankinatty@suddenlinkmail.com> пишет:
The only relevant config found is:
/usr/lib/tmpfiles.d/tmp.conf
with content (without comment):
d /tmp 1777 root root - d /var/tmp 1777 root root - x /tmp/systemd-private-* x /var/tmp/systemd-private-* X /tmp/systemd-private-*/tmp X /var/tmp/systemd-private-*/tmp
Checking the service shows the last run of "Cleanup of Temporary Directories" occurred Feb 22 22:07:46
(yes, I get the gvfs failure as well, it is irrelevant to whether /tmp is cleaned, and simply indicates gvfs was NOT mounted at the time the cleanup was run [check /var/run/user/UID]):
# systemctl status systemd-tmpfiles-clean.service systemd-tmpfiles-clean.service - Cleanup of Temporary Directories Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.service; static) Active: inactive (dead) since Sun 2015-02-22 22:07:46 CST; 11h ago Docs: man:tmpfiles.d(5) man:systemd-tmpfiles(8) Process: 13040 ExecStart=/usr/bin/systemd-tmpfiles --clean (code=exited, status=0/SUCCESS) Main PID: 13040 (code=exited, status=0/SUCCESS)
This simply starts program that reads and interprets tmpfiles.conf snippets (systemd-tmpfiles). What it really does is determined entirely by content of various */tmpfiles.d/*.conf files. The configuration you have shown does not delete any file or directory at all. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 11:05]:
В Mon, 23 Feb 2015 09:48:57 -0600 "David C. Rankin" <drankinatty@suddenlinkmail.com> пишет:
The only relevant config found is:
/usr/lib/tmpfiles.d/tmp.conf
with content (without comment):
d /tmp 1777 root root - d /var/tmp 1777 root root - x /tmp/systemd-private-* x /var/tmp/systemd-private-* X /tmp/systemd-private-*/tmp X /var/tmp/systemd-private-*/tmp
Checking the service shows the last run of "Cleanup of Temporary Directories" occurred Feb 22 22:07:46
(yes, I get the gvfs failure as well, it is irrelevant to whether /tmp is cleaned, and simply indicates gvfs was NOT mounted at the time the cleanup was run [check /var/run/user/UID]):
# systemctl status systemd-tmpfiles-clean.service systemd-tmpfiles-clean.service - Cleanup of Temporary Directories Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.service; static) Active: inactive (dead) since Sun 2015-02-22 22:07:46 CST; 11h ago Docs: man:tmpfiles.d(5) man:systemd-tmpfiles(8) Process: 13040 ExecStart=/usr/bin/systemd-tmpfiles --clean (code=exited, status=0/SUCCESS) Main PID: 13040 (code=exited, status=0/SUCCESS)
This simply starts program that reads and interprets tmpfiles.conf snippets (systemd-tmpfiles). What it really does is determined entirely by content of various */tmpfiles.d/*.conf files. The configuration you have shown does not delete any file or directory at all.
The files below /usr/lib/tmpfiles.d/ indicate a 30d default for the most part, but do not show what I was looking for, time for removing stale systemd-private* directories -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 23 Feb 2015 17:22, Patrick Shanahan wrote:
* Andrei Borzenkov [02-23-15 11:05]:
On Mon, 23 Feb 2015 09:48:57 -0600 "David C. Rankin" wrote:
[snip]
The files below /usr/lib/tmpfiles.d/ indicate a 30d default for the most part, but do not show what I was looking for, time for removing stale systemd-private* directories
Have you given a tought on removing these directories on system shutdown? see: /usr/lib/systemd/system/halt-local.service which starts: /etc/init.d/halt.local into the "/etc/init.d/halt.local" script add this line: [code] rm -rf /tmp/systemd-private-* /var/tmp/systemd-private-* [/code] And, yes, I consider the failure of systemd to remove these directories after the service is stopped a bug. - Yamaban. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
В Mon, 23 Feb 2015 11:22:20 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
The files below /usr/lib/tmpfiles.d/ indicate a 30d default for the most part, but do not show what I was looking for, time for removing stale systemd-private* directories
I think it was discussed on this list already. It is not safe to delete them because it is not known whether they belong to a running service. Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2). This leaves us with directories left as result of system crash which can be (unless already done) removed on startup. I believe among packages in default installation there are probably couple of those using systemd-private-* directories and I'm not sure whether they are even enabled by default. So unless you permanently restart them it hardly can be considered an issue. On startup I delete them with bor@opensuse:~> cat /etc/tmpfiles.d/remove-systemd-private.conf R /tmp/systemd-private-* R /var/tmp/systemd-private-* bor@opensuse:~> -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 12:06]:
В Mon, 23 Feb 2015 11:22:20 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
The files below /usr/lib/tmpfiles.d/ indicate a 30d default for the most part, but do not show what I was looking for, time for removing stale systemd-private* directories
I think it was discussed on this list already. It is not safe to delete them because it is not known whether they belong to a running service. Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2). This leaves us with directories left as result of system crash which can be (unless already done) removed on startup.
I believe among packages in default installation there are probably couple of those using systemd-private-* directories and I'm not sure whether they are even enabled by default. So unless you permanently restart them it hardly can be considered an issue. On startup I delete them with
bor@opensuse:~> cat /etc/tmpfiles.d/remove-systemd-private.conf R /tmp/systemd-private-* R /var/tmp/systemd-private-* bor@opensuse:~>
Tks, I will institute that on my desk/laptops but what do you do on a server? -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
В Mon, 23 Feb 2015 12:18:18 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 12:06]:
В Mon, 23 Feb 2015 11:22:20 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
The files below /usr/lib/tmpfiles.d/ indicate a 30d default for the most part, but do not show what I was looking for, time for removing stale systemd-private* directories
I think it was discussed on this list already. It is not safe to delete them because it is not known whether they belong to a running service. Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2). This leaves us with directories left as result of system crash which can be (unless already done) removed on startup.
I believe among packages in default installation there are probably couple of those using systemd-private-* directories and I'm not sure whether they are even enabled by default. So unless you permanently restart them it hardly can be considered an issue. On startup I delete them with
bor@opensuse:~> cat /etc/tmpfiles.d/remove-systemd-private.conf R /tmp/systemd-private-* R /var/tmp/systemd-private-* bor@opensuse:~>
Tks, I will institute that on my desk/laptops but what do you do on a server?
I'm not sure I understand the question. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 12:23]:
В Mon, 23 Feb 2015 12:18:18 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 12:06]:
В Mon, 23 Feb 2015 11:22:20 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
The files below /usr/lib/tmpfiles.d/ indicate a 30d default for the most part, but do not show what I was looking for, time for removing stale systemd-private* directories
I think it was discussed on this list already. It is not safe to delete them because it is not known whether they belong to a running service. Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2). This leaves us with directories left as result of system crash which can be (unless already done) removed on startup.
I believe among packages in default installation there are probably couple of those using systemd-private-* directories and I'm not sure whether they are even enabled by default. So unless you permanently restart them it hardly can be considered an issue. On startup I delete them with
bor@opensuse:~> cat /etc/tmpfiles.d/remove-systemd-private.conf R /tmp/systemd-private-* R /var/tmp/systemd-private-* bor@opensuse:~>
Tks, I will institute that on my desk/laptops but what do you do on a server?
I'm not sure I understand the question.
I make every effort to not "reboot" servers. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
В Mon, 23 Feb 2015 12:27:16 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 12:23]:
В Mon, 23 Feb 2015 12:18:18 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 12:06]:
В Mon, 23 Feb 2015 11:22:20 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
The files below /usr/lib/tmpfiles.d/ indicate a 30d default for the most part, but do not show what I was looking for, time for removing stale systemd-private* directories
I think it was discussed on this list already. It is not safe to delete them because it is not known whether they belong to a running service. Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2). This leaves us with directories left as result of system crash which can be (unless already done) removed on startup.
I believe among packages in default installation there are probably couple of those using systemd-private-* directories and I'm not sure whether they are even enabled by default. So unless you permanently restart them it hardly can be considered an issue. On startup I delete them with
bor@opensuse:~> cat /etc/tmpfiles.d/remove-systemd-private.conf R /tmp/systemd-private-* R /var/tmp/systemd-private-* bor@opensuse:~>
Tks, I will institute that on my desk/laptops but what do you do on a server?
I'm not sure I understand the question.
I make every effort to not "reboot" servers.
How often services that are using private tmp directories are restarted on your servers? If this happens so often as to become a problem, you can install systemd from factory. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 12:39]:
В Mon, 23 Feb 2015 12:27:16 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 12:23]:
В Mon, 23 Feb 2015 12:18:18 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
* Andrei Borzenkov <arvidjaar@gmail.com> [02-23-15 12:06]:
В Mon, 23 Feb 2015 11:22:20 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
The files below /usr/lib/tmpfiles.d/ indicate a 30d default for the most part, but do not show what I was looking for, time for removing stale systemd-private* directories
I think it was discussed on this list already. It is not safe to delete them because it is not known whether they belong to a running service. Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2). This leaves us with directories left as result of system crash which can be (unless already done) removed on startup.
I believe among packages in default installation there are probably couple of those using systemd-private-* directories and I'm not sure whether they are even enabled by default. So unless you permanently restart them it hardly can be considered an issue. On startup I delete them with
bor@opensuse:~> cat /etc/tmpfiles.d/remove-systemd-private.conf R /tmp/systemd-private-* R /var/tmp/systemd-private-* bor@opensuse:~>
Tks, I will institute that on my desk/laptops but what do you do on a server?
I'm not sure I understand the question.
I make every effort to not "reboot" servers.
How often services that are using private tmp directories are restarted on your servers? If this happens so often as to become a problem, you can install systemd from factory.
Not an intelligent or viable option on a server. Being a server espouses stability. factory != stability -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/23/2015 12:20 PM, Andrei Borzenkov wrote:
Tks, I will institute that on my desk/laptops but what do you do on a server?
I'm not sure I understand the question.
I *think* he means that he's one of the people who shuts down and restarts his desktop every day or perhaps more than once a day, but the server (under the desk in my case, in the "computer room" for many others) is not shut down/restarted with any regularly; in fact the whole point of the server is uninterrupted operation.. And it is on the server that the accumulation of /tmp "debris" is likely to be at its most pernicious. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 02/23/2015 12:20 PM, Andrei Borzenkov wrote:
Tks, I will institute that on my desk/laptops but what do you do on a server?
I'm not sure I understand the question.
I *think* he means that he's one of the people who shuts down and restarts his desktop every day or perhaps more than once a day, but the server (under the desk in my case, in the "computer room" for many others) is not shut down/restarted with any regularly; in fact the whole point of the server is uninterrupted operation..
And it is on the server that the accumulation of /tmp "debris" is likely to be at its most pernicious.
I have a few servers running 13.1, even one or two on 13.2 - I don't see /tmp filling up. Has anyone actually looked at what it is _what_ is filling up {/var}/tmp ? -- Per Jessen, Zürich (2.6°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 23/02/15 18:31, Per Jessen wrote:
Has anyone actually looked at what it is _what_ is filling up {/var}/tmp ?
I've not had time since starting this topic to come back to it and things might remain that way for another couple of days, but what I have just discovered is what is responsible for 98% of all the data consumption in my /tmp: it's Dolphin. My /tmp contains all sorts, anything dating from the system installation over a year ago: gpg-xxxxxx folders, empty plugtmp-x folders, some SpiderOak stuff, about a dozen systemd-private-xxxxxx folders but those only consuming 112kb in total, lots of Clementine art jpg files, clipboard cache, attachments from Thunderbird and a few downloaded items from Firefox opened with Okular or perhaps other programs. But, the vast majority is contained within 184 folders named dolphinxxxxxx inside the kde-<username> subfolder. About two thirds of these are empty, whilst of the remainder, it's just the entire cached contents of various directories, both those permanently on my system and others from removable devices such as my dumbphone when connected via USB Mass Storage. Of the 5.0GB /tmp currently consumes, 4.9GB is this. /var/tmp is more moderate, currently at 756MB, most of the bytes being within the kdecache-<username> folders. A couple of plasma_theme_xxx.kcache files take up the majority. So there are two issues from my own perspective: 1. Why is Dolphin caching all this stuff forever, and is there a setting that governs this or a misconfiguration somewhere? (Note: Though Nepomuk Semantic Desktop is enabled, I have file indexing turned off.) 2. Whilst Dolphin accounts for 98% of the actual data size, there's still thousands of files and folders from elsewhere being held in temp and never being cleaned. Well, perhaps some stuff is being cleaned, that I've yet to find out. I'll have to look into the previous suggestions for cleaning temp when I get a bit more time. Thanks for the replies thus far. Peter -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2/25/2015 4:12 PM, Peter wrote:
On 23/02/15 18:31, Per Jessen wrote:
Has anyone actually looked at what it is _what_ is filling up {/var}/tmp ?
I've not had time since starting this topic to come back to it and things might remain that way for another couple of days, but what I have just discovered is what is responsible for 98% of all the data consumption in my /tmp: it's Dolphin.
My /tmp contains all sorts, anything dating from the system installation over a year ago: gpg-xxxxxx folders, empty plugtmp-x folders, some SpiderOak stuff, about a dozen systemd-private-xxxxxx folders but those only consuming 112kb in total, lots of Clementine art jpg files, clipboard cache, attachments from Thunderbird and a few downloaded items from Firefox opened with Okular or perhaps other programs. But, the vast majority is contained within 184 folders named dolphinxxxxxx inside the kde-<username> subfolder. About two thirds of these are empty, whilst of the remainder, it's just the entire cached contents of various directories, both those permanently on my system and others from removable devices such as my dumbphone when connected via USB Mass Storage. Of the 5.0GB /tmp currently consumes, 4.9GB is this.
/var/tmp is more moderate, currently at 756MB, most of the bytes being within the kdecache-<username> folders. A couple of plasma_theme_xxx.kcache files take up the majority.
So there are two issues from my own perspective: 1. Why is Dolphin caching all this stuff forever, and is there a setting that governs this or a misconfiguration somewhere? (Note: Though Nepomuk Semantic Desktop is enabled, I have file indexing turned off.) 2. Whilst Dolphin accounts for 98% of the actual data size, there's still thousands of files and folders from elsewhere being held in temp and never being cleaned. Well, perhaps some stuff is being cleaned, that I've yet to find out.
I'll have to look into the previous suggestions for cleaning temp when I get a bit more time.
Thanks for the replies thus far.
Peter
I had questions about the Spideraok files as well. I asked them about them and they responded that anything in /tmp can be deleted upon boot:
This is Dana from SpiderOak Support.
Yes, you can delete the spideroak_inotify_db.xxxxxx files as long as SpiderOak isn't running. As long as SpiderOak isn't running during the time you're deleting the files then you should be fine.
So I put in a delete "r" with an exclamation per the man page to delete these only at boot. Its good to know you've tracked down the bulk of this to caches of dolphinxxxxxx, I can make some custom entries for this. I occasionally surf over to my NAS where I have a large collection of photos and such, and if these are being cached, or even thumbnails, it would be huge. I don't know why the current rules don't take care of those, as these rules should descend to those directories. I'll do some digging and post my settings. I don't want to mess up anything that is actually used, and I don't believe that anything in /tmp has anything to do with either nepomuk or Baloo (the 13.2 replacement there-of). That is in your personal directory, -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/25/2015 09:22 PM, John M Andersen wrote:
I don't know why the current rules don't take care of those, as these rules should descend to those directories. I'll do some digging and post my settings.
Having given my system enough time for the scheduled deletions to take effect, and having rebooted it, I find that my usage in /tmp is way down from what it was. du -h /tmp shows only only 540k total, down from 4gig. Specifically I have nothing in my /tmp/kde-username directory upon boot. If I mount a data cd or music cd I will get some zero byte temp files /tmp/kde-username. But apparently these are not long lived Note I'm on OS 13.2 Here is a listing of my tmp.conf:
poulsbo:/tmp # cat /etc/tmpfiles.d/tmp.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version.
# See tmpfiles.d(5) for details
# Clear tmp directories separately, to make them easier to override # SUSE policy: we don't clean those directories JSA Policy: Yes you do!!! d /tmp 1777 root root 2d - d /var/tmp 1777 root root 2d - # # here we delete some directories that tend to accumulate, but only if they are empty r /tmp/gpg-??????/ r /tmp/ssh-????????????/ r /tmp/sni-qt_SpiderOak_* r! /tmp/spideroak_inotify_db.*
# Exclude namespace mountpoints created with PrivateTmp=yes x /tmp/systemd-private-%b-* X /tmp/systemd-private-%b-*/tmp x /var/tmp/systemd-private-%b-* X /var/tmp/systemd-private-%b-*/tmp
Notice, I haven't messed with the Systemd exclusions at all. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 26/02/15 22:49, John Andersen wrote:
Having given my system enough time for the scheduled deletions to take effect, and having rebooted it, I find that my usage in /tmp is way down from what it was. du -h /tmp shows only only 540k total, down from 4gig.
Specifically I have nothing in my /tmp/kde-username directory upon boot. If I mount a data cd or music cd I will get some zero byte temp files /tmp/kde-username. But apparently these are not long lived
Note I'm on OS 13.2
Here is a listing of my tmp.conf:
poulsbo:/tmp # cat /etc/tmpfiles.d/tmp.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version.
# See tmpfiles.d(5) for details
# Clear tmp directories separately, to make them easier to override # SUSE policy: we don't clean those directories JSA Policy: Yes you do!!! d /tmp 1777 root root 2d - d /var/tmp 1777 root root 2d - # # here we delete some directories that tend to accumulate, but only if they are empty r /tmp/gpg-??????/ r /tmp/ssh-????????????/ r /tmp/sni-qt_SpiderOak_* r! /tmp/spideroak_inotify_db.*
# Exclude namespace mountpoints created with PrivateTmp=yes x /tmp/systemd-private-%b-* X /tmp/systemd-private-%b-*/tmp x /var/tmp/systemd-private-%b-* X /var/tmp/systemd-private-%b-*/tmp
Notice, I haven't messed with the Systemd exclusions at all.
I finally got around to looking at this again. I added the tmp.conf file in /etc/tmpfiles.d and set a 30-day limit on /tmp, and 90 days on /var/tmp. I had to reboot since KDED had crashed on me as it often does, though the temp clearance didn't take place at or after boot. I don't know how long afterwards since I wasn't monitoring constantly, but within a couple of hours or so I noticed I'd suddenly regained several gigabytes of space on the root partition. Whereas previously, /tmp was occupying 5GB it's now down to about 0.9GB. /var/tmp was at 750MB and now it's down to 440MB. I haven't yet toyed with the removal of empty directories but I note that many still remain. Indeed, there are still quite a lot of files inside /tmp dated from months ago, so I assume they have had their date touched, perhaps even by my listing of the directory contents in Dolphin Super User Mode. I might avoid looking in there for 31 days and then see if they vanish. So, it works, with a simple adjustment to the tmp.conf file by adding the time limit. I realise it is SuSE policy to not clean these directories, but with the new work on YaST3 it would be nice if an easy config option were put back in somewhere to manage this, because for an average user, or indeed any user coming from most other distros that don't have this policy, expecting them to dive into an obscure config file in usr/lib to save their root partitions getting filled up with potentially undesirable consequences is a rather bad setup. (Or if they haven't set a separate /home and have a small drive, especially on a device like a Raspberry Pi or a tablet with storage on a low-capacity card, it'll be their entire drive getting full up; not good). Thanks to each of you who suggested these solutions. Peter -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/28/2015 03:48 PM, Peter wrote:
I realise it is SuSE policy to not clean these directories,
That policy really has to go. Seriously, how could you expect a server to operate unattended with that kind of policy in place? This default policy is just ludicrous. I've put most of my /tmp file duration at 2 days. I've found there is just nothing that I will use after 2 days. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Am Sonntag, 01. März 2015 06:19 CET, John Andersen <jsamyth@gmail.com> schrieb:
On 02/28/2015 03:48 PM, Peter wrote:
I realise it is SuSE policy to not clean these directories,
That policy really has to go. Seriously, how could you expect a server to operate unattended with that kind of policy in place? This default policy is just ludicrous.
I don't think openSUSE is meant to be used on servers. There is an enterprise edition for that. If you still want to use openSUSE on a server, then I suggest you start using things like Puppet or Chef to harden the system which is geared more towards everyday use.
I've put most of my /tmp file duration at 2 days. I've found there is just nothing that I will use after 2 days.
I have 60GB of /tmp and I don't care how long the files lie in there. How is my argument more or less worth than your's? A small script (which you could share) should be enough to configure a standard openSUSE system to retain files for only N days. Then everyone can decide whether they want to run it or not. Regards, -- Aaron "Optimizer" Digulla a.k.a. Philmann Dark "It's not the universe that's limited, it's our imagination. Follow me and I'll show you something beyond the limits." http://blog.pdark.de/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/02/2015 01:11 AM, Aaron Digulla wrote:
Am Sonntag, 01. März 2015 06:19 CET, John Andersen <jsamyth@gmail.com> schrieb:
On 02/28/2015 03:48 PM, Peter wrote:
I realise it is SuSE policy to not clean these directories,
That policy really has to go. Seriously, how could you expect a server to operate unattended with that kind of policy in place? This default policy is just ludicrous.
I don't think openSUSE is meant to be used on servers. There is an enterprise edition for that.
If you still want to use openSUSE on a server, then I suggest you start using things like Puppet or Chef to harden the system which is geared more towards everyday use.
What is this "Meant to" of which you speak? Its a linux distribution. It has an install option for no graphic desktop. When I say Server that doesn't necessarily mean Fortune 500 company. That said, those machines used as servers don't tend to accumulate much in /tmp But even a workstation used every day will build /tmp detritus, and one day come crashing down on you, likely compromising something. Because you "think" you have allocated enough disk, is no excuse to save useless garbage forever. You will never need it again, why save it. One reason I run Opensuse is because it installs very securely by default. Every red-hat machine I've ever installed required me to run around and close a mountain of loop-holes in the default setup before I ever plugged in the cat-5 cable.
I've put most of my /tmp file duration at 2 days. I've found there is just nothing that I will use after 2 days.
I have 60GB of /tmp and I don't care how long the files lie in there. How is my argument more or less worth than your's?
Well you may start caring, especially if you burn dvds do other large things which will fill up your 60 gig in short order because not every program cleans up after itself, because, well, its called "/tmp" for a reason. The fact that /tmp grows is proof that a lot of software still abandons files there. Its full of files that have last access dates many months old which will never again be used. Its your machine, and you are free to do with it as you like, but to me spinning useless storage is a waste of resources. Opensuse deviated from the long established standard here. Extraordinary policies require extraordinary reasons. Not just "buy a bigger disk".
A small script (which you could share) should be enough to configure a standard openSUSE system to retain files for only N days. Then everyone can decide whether they want to run it or not.
Had you been following the thread, you would have noticed that I already posted that upthread. It works perfectly, and its driven by the system normal cron jobs, without any manual scheduling or error prone custom script writing. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/01/15 06:19, John Andersen wrote:
On 02/28/2015 03:48 PM, Peter wrote:
I realise it is SuSE policy to not clean these directories,
That policy really has to go. Seriously, how could you expect a server to operate unattended with that kind of policy in place? This default policy is just ludicrous.
I've put most of my /tmp file duration at 2 days. I've found there is just nothing that I will use after 2 days.
pussy:~ # uptime 02:12am up 391 days 10:56, 4 users, load average: 0.09, 0.14, 0.21 pussy:~ # ll /tmp/ total 84 drwxr-xr-x 3 root root 4096 May 5 2014 db2.tmp.30515 drwxr-xr-x 3 root root 4096 May 5 2014 db2.tmp.30997 drwx------ 2 detig npc 4096 Jan 2 14:58 gpg-hyNluI drwx------ 2 detig npc 4096 Jan 2 14:58 gsrvdir1001 drwxr-xr-x 2 tomcat tomcat 4096 Sep 3 2014 hsperfdata_tomcat drwx------ 2 detig npc 4096 Jan 24 14:23 ksocket-detig drwx------ 2 root root 16384 Feb 11 2014 lost+found drwx------ 2 root root 4096 Mar 10 02:12 ssh-2J9qGPReW0 drwx------ 2 detig npc 4096 Jan 2 14:58 ssh-u6RdTLL4QK7Z drwx------ 2 root root 32768 Mar 10 01:15 vmware-root The DB2 dirs are in use. The ksocket dir is in use. The ssh dirs are in use; the ssh-agent socket is there. (One of them is more than a year old and still works, just FTR.) The VMware dir is in use. Actually, gpg*, gsrv*, and hsperf* might also been in use, haven't checked them. Please don't expect that your use case is representative of all openSUSE users. If you can delete all files older than 2 days, fine for you. (Do you restart your openSUSE system each day?) I have a different situation. I need lots of files and dirs in /tmp that are older than 2 days, and need to retain them. I don't need them across reboots -- then /tmp is cleared, but look at the uptime cited above: it's not as easy as you want to make it sound. Joachim -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Joachim Schrod, Roedermark, Germany Email: jschrod@acm.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 3/9/2015 6:19 PM, Joachim Schrod wrote:
Please don't expect that your use case is representative of all openSUSE users. If you can delete all files older than 2 days,
Nowhere did I say that my use case was representative, Also the system cleanup tasks (as well as any manual method you might care to try) will not delete any files that are in use, (and you've been around here long enough to know that). So YES, it is just as easy as I made it sound! Just make the changes to the /etc/tmpfiles.d/tmp.conf (which I posted elsewhere in this thread) and the system will take care of these for you automatically. There is simply no reason for opensuse to not clean these directories. -- _____________________________________ ---This space for rent--- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/10/15 03:27, John Andersen wrote:
On 3/9/2015 6:19 PM, Joachim Schrod wrote:
Please don't expect that your use case is representative of all openSUSE users. If you can delete all files older than 2 days,
Nowhere did I say that my use case was representative, Also the system cleanup tasks (as well as any manual method you might care to try) will not delete any files that are in use, (and you've been around here long enough to know that).
I beg to differ. The (changed) system cleanup tasks deleted files that were needed by daemons. In particular, they deleted some DB2 files. (That's on 13.1, btw; no 13.2 in production yet.) We're talking about a development system here; DB2 is started there by default, but may be used seldomly, only when a customer ticket arrives. I.e., files may be accessed months after a daemon get started.
Just make the changes to the /etc/tmpfiles.d/tmp.conf (which I posted elsewhere in this thread)
Been there, done that. Your changes are quiet obvious, and were also the first thing that I tried. Didn't help in my case -- here's to you they worked in your use case. I don't want to prevent or hinder other folks from experimenting with your tmp.conf changes. If they help, that's OK. I wanted to add a caveat that it is *NOT* always as easy as you make it sound. Joachim -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Joachim Schrod, Roedermark, Germany Email: jschrod@acm.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/23/2015 12:05 PM, Andrei Borzenkov wrote:
Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2).
Don't you just hate hangers like that? Andrei, could you explain please. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
В Mon, 23 Feb 2015 12:24:55 -0500 Anton Aylward <opensuse@antonaylward.com> пишет:
On 02/23/2015 12:05 PM, Andrei Borzenkov wrote:
Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2).
Don't you just hate hangers like that?
Andrei, could you explain please.
I mean that amount of changes needed to implement it exceeds what is usually considered acceptable for a stable version. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/23/2015 12:42 PM, Andrei Borzenkov wrote:
В Mon, 23 Feb 2015 12:24:55 -0500 Anton Aylward <opensuse@antonaylward.com> пишет:
On 02/23/2015 12:05 PM, Andrei Borzenkov wrote:
Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2).
Don't you just hate hangers like that?
Andrei, could you explain please.
I mean that amount of changes needed to implement it exceeds what is usually considered acceptable for a stable version.
And are all those changes functional changes or are some of the what amounts to 'refactoring'? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, 23 Feb 2015 18:05, Andrei Borzenkov <arvidjaar@...> wrote:
В Mon, 23 Feb 2015 11:22:20 -0500 Patrick Shanahan <paka@opensuse.org> пишет:
The files below /usr/lib/tmpfiles.d/ indicate a 30d default for the most part, but do not show what I was looking for, time for removing stale systemd-private* directories
I think it was discussed on this list already. It is not safe to delete them because it is not known whether they belong to a running service. Current upstream version now removes these directories when service stops (change is rather intrusive to integrate in version used by 13.2). This leaves us with directories left as result of system crash which can be (unless already done) removed on startup.
I believe among packages in default installation there are probably couple of those using systemd-private-* directories and I'm not sure whether they are even enabled by default. So unless you permanently restart them it hardly can be considered an issue. On startup I delete them with
bor@opensuse:~> cat /etc/tmpfiles.d/remove-systemd-private.conf R /tmp/systemd-private-* R /var/tmp/systemd-private-* bor@opensuse:~>
Two services spring to mind: haveged.service rtkit-daemon.service Both of these have the "PrivateTmp=yes" setting, but I have never seen these two services use any tmpfiles at all. Additional, haveged is installed on all systems, rtkit is 'required' by pulseaudio and thus installed on most desktops. Could the PrivateTmp setting be removed of these two? That would reduce the impact for most users. I can accept the "PrivateTmp=yes" setting for rsyncd, there it makes sense in terms of privacy and security. - Yamaban.
On 02/23/2015 12:27 PM, Yamaban wrote:
Both of these have the "PrivateTmp=yes" setting, but I have never seen these two services use any tmpfiles at all.
I haven't looked at the code but ... It is a well established technique dating all the way back in my experience to UNIX V6 of 1970s vintage to have a process open a file (often in /tmp) and then delete it. So long as the process or its children has an open file handle for that file it exists and is usable by the process or its children, even for interprocess communication & signalling. When the last processing holding a file handle for this file closes the handle or exits the file vanishes. Perhaps those processes use this technique. As I say, I haven't looked at the source. Either way, this raises the question about systemd leaving such debris in /tmp. Hmm. It seems a few other things like acroread and Mozilla do too. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* David C. Rankin <drankinatty@suddenlinkmail.com> [02-23-15 10:49]: [...]
So my only guess is that these are the pieces of the puzzle that are supposed to insure tmpfiles are cleaned. Checking the journal will tell you if they are running. If the only entry in the journal is the gvfs failure without the Starting/Started Cleanup of Temporary Directories..., then figuring out why they are not running will likely solve the issue.
I checked my server which ran out of tmp space recently and see that indeed the cleanup service was started "twice" each day (appears, didn't check that closely except around the failure period). But left over text files from mutt were not cleared/cleaned nor the systemd-private* directories which may have presented an ?? inode problem. Since specifying the time period, the text files *are* clearned but the directories are not. I do not know and cannot find a specified time period for the "persistant/protected" directories.
I don't recall doing anything strange/special on install regarding tmpfiles, although I do go though each config setting and I DO set a traditional 'root' account and password. I've always been leery about not using a traditional root account just for the reason that some hooked process that relies on being run by root -- won't -- in the absence of a traditional root account. I doubt it is related.
I do the same.
Regardless, here is one data-point among others where the cleaning of tmpfiles seems to be working as it should. From the original post, it appears Peter has all parts of the cleanup configuration discussed above. The only additional checks I see are to check the journal and find out just what is being run (and when) and go from there. Further, there should be a way to manually kick-off the cleaning of tmpfiles. I would figure that out and attempt to force a cleaning and check the results/errors there. That will provide a further bit of info.
I also check the journal on a desktop but haven't observed enough time since specifying the time period to make a judgement past the systemd-private* directories remaining. I do see periods of 3-8 days where no cleanup was initiated and single rather than double entries in the journal (one second apart). But subject system is < 2 months old.
Good luck. (SysVinit was so much cleaner...)
In some ways, but I like *much* about systemd. SysVinit was like a manila folder crammed full of random papers and systemd more like an ordered digest. Problem is learning the particular paths vs the remaining rote. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (14)
-
Aaron Digulla -
Andrei Borzenkov -
Anton Aylward -
Carlos E. R. -
Carlos E. R. -
David C. Rankin -
Dylan -
Joachim Schrod -
John Andersen -
John M Andersen
-
Patrick Shanahan -
Per Jessen -
Peter -
Yamaban