On Mon, Sep 19, 2016 at 2:01 PM, sdm
On 09/19/2016 12:54 PM, Chris Murphy wrote:
On Mon, Sep 19, 2016 at 1:47 PM, sdm
wrote: On 09/19/2016 08:44 AM, Linda Walsh wrote:
From the docs George quoted, the journal should already be set to autotrim -- where it should auto clean itself when the disk is getting near full. It sorta sounds like it didn't do that intelligently -- i.e. he was having disk-full conditions and the log wasn't autotrimming.
The defaults are volatile logs on a clean 42.1 Leap install, because "auto" is should be commented out (can someone verify, on a clean 42.1 install?) and the storage limits commented out by default, which make the journal behave practically like "volatile" as I wrote in the above email. So, George, did you do an upgrade from 13.1 or 13.2?
All systemd configuration files show the defaults commented out, including /etc/systemd/journald.conf.
# Entries in this file show the compile time defaults.
So if you have an entry: #Storage=auto
That means it looks for /var/log/journal. If it exists already, then the journal will be stored persistently in that location. If it doesn't exist, it will be stored in /run/log/journal/ which is volatile.
Not on Leap 422. *If* /var/log/journal exists, those journal files are wiped with each reboot, and Leap 42.1 behaved the same exact way for me, although I don't rememember right now if /var/log/journal existed on my past Leap 42.1 box. I wouldn't be surprised if after installing Leap 42.1, and the /var/log/journal directory existed, the behaviour would be exactly the same as Leap 422.
I just did a defaultinstallation of openSUSE-Leap-42.2-DVD-x86_64-Build0164-Media.iso in a virt-manager VM. And I did several reboots. - /var/log/journal exists, I assume it's created by something that sets up the fs structure, could be the installer or a package that just creates a bunch of directories to make sure they exist $ rpm -qa | grep logger systemd-logger-228-8.1.x86_64 $ sudo journalctl --list-boots -2 cb919764f6034901a5ca588992aecafe Mon 2016-09-19 17:20:48 EDT—Mon 2016-09-19 17:26:15 EDT -1 ce9c562b531a428c825cbd40d7656ccf Mon 2016-09-19 17:26:31 EDT—Mon 2016-09-19 17:27:42 EDT 0 fb40d5665c4b42e1abbf46106e37f606 Mon 2016-09-19 17:27:52 EDT—Mon 2016-09-19 17:29:22 EDT $ sudo journalctl -b-1 shows the log of the previous boot, so clearly it's persistent $ sudo journalctl -b | grep journald Sep 19 17:27:52 linux-ae34 systemd-journald[105]: Runtime journal (/run/log/journal/) is currently using 8.0M. Sep 19 17:27:52 linux-ae34 systemd-journald[105]: Journal started Sep 19 17:27:54 linux-ae34 systemd-journald[105]: Journal stopped Sep 19 17:27:54 linux-ae34 systemd-journald[444]: Runtime journal (/run/log/journal/) is currently using 8.0M. Sep 19 17:27:54 linux-ae34 systemd-journald[105]: Received SIGTERM from PID 1 (systemd). Sep 19 17:27:54 linux-ae34 systemd-journald[444]: Journal started Sep 19 17:27:55 linux-ae34 systemd-journald[444]: System journal (/var/log/journal/) is currently using 16.0M. Sep 19 17:27:55 linux-ae34 systemd-journald[444]: Time spent on flushing to /var is 19.289ms for 763 entries. It's clearly being flushed to disk, the journal is persistent. $ sudo journalctl -b -- Logs begin at Mon 2016-09-19 17:20:48 EDT, end at Mon 2016-09-19 17:34:31 EDT. -- Sep 19 17:27:52 linux-ae34 systemd-journald[105]: Runtime journal (/run/log/journal/) is currently using 8.0M. Maximum allowed usage is set to 150.3M. Leaving at least 225.5M free (of currently available 1.4G of space). Enforced usage limit is thus 150.3M, of which 142.3M are still available. [...snip...] Sep 19 17:27:55 linux-ae34 systemd-journald[444]: System journal (/var/log/journal/) is currently using 16.0M. Maximum allowed usage is set to 4.0G. Leaving at least 4.0G free (of currently available 35.6G of space). Enforced usage limit is thus 4.0G, of which 3.9G are still available. Sep 19 17:27:55 linux-ae34 systemd-journald[444]: Time spent on flushing to /var is 19.289ms for 763 entries. Sep 19 17:27:55 linux-ae34 systemd[1]: Started Flush Journal to Persistent Storage. Again there are two journals, the initial volatile one during startup, as soon as rootfs is rw the journal gets flushed to persistent storage on /var/log/journal. $ sudo journalctl --disk-usage Archived and active journals take up 16.2M on disk So the gotcha here might be this from 'man journald.conf' SystemMaxUse= and RuntimeMaxUse= control how much disk space the journal may use up at most. SystemKeepFree= and RuntimeKeepFree= control how much disk space systemd-journald shall leave free for other uses. systemd-journald will respect both limits and use the smaller of the two values. The first pair defaults to 10% and the second to 15% of the size of the respective file system, but each value is capped to 4G. If the file system is nearly full and either SystemKeepFree= or RuntimeKeepFree= are violated when systemd-journald is started, the limit will be raised to the percentage that is actually free. This means that if there was enough free space before and journal files were created, and subsequently something else causes the file system to fill up, journald will stop using more space, but it will not be removing existing files to reduce the footprint again, either. And as we know the something else can cause it to fill up, which are snapshots. Hypothetically this should get better with quotas better informing snapper what to free up, but this needs testing to see what the interactions are. -- Chris Murphy -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org