Bruno Friedmann writes:

Take into account, that some people will use journald without the rsyslog emulation on (no /var/log/message text flushing) And the default doesn't look that bad

systemd-journal[419]: Runtime journal is using 2.7M (max 796.6M, leaving 1.1G of free 7.7G, current limit 796.6M).

None of this is even remotely related to the problem at hand AFAICS.

du total of /var/log/journal = 1.8GB

How many files do you have in that directory and what's the result of time journalctl --verify (if you have lots of rotated backups, you can do the verify only on the "open" system and user journals, which seems to be -- at least runtime-wise -- what happens at shutdown). Regards, Achim. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bruno Friedmann writes:

...

How many files do you have in that directory and what's the result of 70 files since January 28th 2013.

So these files are 25MByte on average?

...

time journalctl --verify

real 2m20.742s user 2m6.646s sys 0m12.043s

You obviously have a much speedier system than mine (which doesn't really surprise me as my computer is 11 years old). Is this an SSD? Out of curiosity, how long does it take to just verify the active journal file? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Waldorf MIDI Implementation & additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Friday 04 April 2014 09.31:38 Claudio Freire wrote:

On Fri, Apr 4, 2014 at 8:38 AM, Bruno Friedmann wrote:

...

...

...

...

time journalctl --verify

real 2m20.742s user 2m6.646s sys 0m12.043s

You obviously have a much speedier system than mine (which doesn't really surprise me as my computer is 11 years old). Is this an SSD? Out of curiosity, how long does it take to just verify the active journal file?

It's already a old computer 3 years old i7-2nd generation with 16GB Ram and a 480GB Corsair SSD GT-Force3 (6gbs sata)

2 minutes on an SSD?

That's completely unworkable for rotating media (they're *orders of magnitude* slower for random access). For 2.9GB of logs ?

Can you test that on rotating media please? A great deal of us are still using spinning rust for our storage needs, testing on SSD and calling it representative is probably as wrong as you can get.

Well if only I've still rotating rust for system. I've chosen journald exclusively cause I know some report of being perhaps slow on rust and all system here are now powered on ssd. (Tired of sector defects & co) Now the concerns raised about it are to take into account http://seife.kernalert.de/blog/2013/05/31/the-systemd-journal-is-a-broken-pi... http://www.happyassassin.net/2013/06/14/fedora-1920-logfile-explosions/ (chapter four) I also have decided to export them to external storage in plain text mode for archiving purpose. Rotating the journal is also a nice way to get perf back. Afterall we also rotate text log files and compress them. But when it work, honestly I became quickly addict to the color/bold , filtering etc :-) for example I've rotated my log this morning and now du -sh /var/log/journal/32aed2336d02ef306e8c3ca25103d7ee 3.7M /var/log/journal/32aed2336d02ef306e8c3ca25103d7ee time journalctl --verify --system PASS: /var/log/journal/32aed2336d02ef306e8c3ca25103d7ee/system.journal real 0m0.041s user 0m0.035s sys 0m0.006s Look like there's some good practices to be determined. -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch openSUSE Member & Board GPG KEY : D5C9B751C4653227 irc: tigerfoot ~~~Don't take Life too serious. Nobody gets out alive anyway!~~~ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Am 04.04.2014 19:42, schrieb Claudio Freire:

Can you not copy into rotating rust and verify the copy?

That's not a real test, since the files will be defragmented by the process. In a real setup, they are fragmented like hell, or at least they were before Werner backported the fix into systemd-208 on Feb.14 :-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Friday 04 April 2014 23.25:09 Stefan Seyfried wrote:

Am 04.04.2014 19:12, schrieb Bruno Friedmann:

...

Well if only I've still rotating rust for system. I've chosen journald exclusively

I really would not do that, I'd always add syslog-ng in addition.

I cannot count the number of times the journald crapped itself and I had files that could not be read. Even better: when the broken file was present I could not read any journal anymore. Basically journalctl was going in circles:

2013-12-21 00:00:01 .... 2013-12-21 12:34:45 .... 2013-12-21 19:20:31 .... --- reboot --- 2013-12-21 00:00:01 ....

etc. And just with trial and error (removing one file after another) I was finally able to delete 3 weeks of journal logs to at least be able to read the rest.

This happened so often that I finally got tired of filing bugs :-)

With syslog-ng, i sometimes after a crash get 4K of \000 in the file, but that's it. And I can just skip over it.

...

cause I know some report of being perhaps slow on rust and all system here are now powered on ssd. (Tired of sector defects & co)

Slowness is one problem, but being unreliable is really a PITA.

...

for example I've rotated my log this morning and now

How do you do that? How do you determine which of the files are no longer needed?

While I'm still experiencing the impact of the different value in journalctl.conf on running system. And also due to some of the same kind of corruption (on old journal in 2013) (I've successfully reduce the lost by playing with the --since=DATE --until=DATE to bisect the faulty. Upstream have always promise that the journal has not problem, just the parser. So I keep them ;-) I'm using a poor dirty man hack rotate way. I've no formal code but my hack follow the idea of Find journal storage D=$(journalctl --header --system | grep "Machine ID" | awk '{print $3}') R=${D}_$(date --utc "+%F_%H%M%S") $B="/var/log/journal" systemctl stop systemd-journald.service mv ${B}/${D} ${B}/${R} systemctl start systemd-journald.service Journal is recreated and old one are still accessible with journalctl --directory=PATH could be exported to another host etc... Not that ideal as you can see, but I know that I don't have 100 lines of incoming logs when the rotate happen. So just a way to keep the journal quite fresh. -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch openSUSE Member & Board GPG KEY : D5C9B751C4653227 irc: tigerfoot ~~~Don't take Life too serious. Nobody gets out alive anyway!~~~ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Tuesday 08 April 2014 22.07:21 Stefan Seyfried wrote:

Hi Bruno,

Am 07.04.2014 09:26, schrieb Bruno Friedmann:

...

Upstream have always promise that the journal has not problem, just the parser. So I keep them ;-)

I keep the messages-$DATE.xz, I know that I can read them without relying on someone else :-) Journal is just for "fun" (well, strange values of "fun") for now and the foreseeable future.

...

I'm using a poor dirty man hack rotate way. I've no formal code but my hack follow the idea of

Find journal storage D=$(journalctl --header --system | grep "Machine ID" | awk '{print $3}') R=${D}_$(date --utc "+%F_%H%M%S") $B="/var/log/journal"

systemctl stop systemd-journald.service mv ${B}/${D} ${B}/${R} systemctl start systemd-journald.service

That's cool. I'll keep that for reference :-)

...

Journal is recreated and old one are still accessible with journalctl --directory=PATH could be exported to another host etc...

Not that ideal as you can see, but I know that I don't have 100 lines of incoming logs when the rotate happen.

IMO you should be able to just move the directory and then restart journald, since the FDs will stay open during move, so you'll only have a very short outage? Hey that need tests too. :-)

I'm also investigating in the different export format journalctl could do For example using it to feed sshguard daemon LANG=C /usr/bin/journalctl $jnlunits -alfb --no-pager -p info -n1 -o cat | /usr/sbin/sshguard $cmdline -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch openSUSE Member & Board GPG KEY : D5C9B751C4653227 irc: tigerfoot ~~~Don't take Life too serious. Nobody gets out alive anyway!~~~ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Fri, Apr 4, 2014 at 6:16 PM, Stefan Seyfried wrote:

...

That's completely unworkable for rotating media (they're *orders of magnitude* slower for random access).

Never ever put the persistant journal onto rotating rust. Just use syslog. Due to the genius design, the files fragment heavily and thrash the disk. Even plain "systemctl status foo.service" which looks into journal for logs will take 30+ seconds with journal on non-SSD.

...

Can you test that on rotating media please? A great deal of us are still using spinning rust for our storage needs, testing on SSD and calling it representative is probably as wrong as you can get.

Putting the journal on anything but SSD or RAM is as wrong as you can get :-)

Why not preallocate the space? That would get rid of fragmentation, and it should be a mostly-oneliner patch (call to posix_fallocate). For reading, posix_fadvise (or madvise) could read it whole (to avoid reading it in random fashion). Has this been tried? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Bruno Friedmann writes:

...

So these files are 25MByte on average? Nope from what I'm seeing actually […]

Your active journal seems to be around 100MiB then.

It's already a old computer 3 years old i7-2nd generation with 16GB Ram and a 480GB Corsair SSD GT-Force3 (6gbs sata)

SSD, I thought so. You'd certainly not like the performance if this was on spinning disk regardless of the CPU performance. Even on my old system and only a single thread, I/O wait time already is noticeable. It doesn't help that journald hopelessly fragments its journal files.

11 years old ? I would not like to pay the electricity bill :-)

For starters, it's not 24/7 and it actually isn't all that bad except when idle (and then only if you compare it to Haswell).

active journal take the same time almost cause everything is active.

The "closed" journals actually don't seem to be delaying the shutdown of the system as far as I am able to determine. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org