Logrotate, /var/log but logging disabled by default
Hi, we have quite some packages, which: 1. require logrotate and ship a logrotate configuration file 2. have a special directory in /var/log 3. never create a log file as they are controlled by systemd The reason the packages never write to the log file by default is, that they log to stdout/stderr and systemd/journald captures them. What should we do with such packages? Requiring logrotate, even if never used by default, is already a bad idea. It runs regular (so uses resources) without doing anything. Clobbering /var/log with special directories and files owned by special users will also only trigger actions, even if never used. What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways. Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it - let the admin create the configuration file and directory Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg)
On Wed, 2020-12-02 at 10:44 +0100, Thorsten Kukuk wrote:
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it - let the admin create the configuration file and directory
Thorsten
Maybe like this: Package the logrotate stuff separately (e.g. as $X-logrotate) and use Requires: ($X and logrotate) Supplements: ($X and logrotate) and Recommends: ($X-logrotate unless systemd) http://rpm.org/user_doc/boolean_dependencies.html Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Software Solutions Germany GmbH HRB 36809, AG Nürnberg GF: Felix Imendörffer
Le mercredi 02 décembre 2020 à 12:11 +0000, Martin Wilck a écrit :
On Wed, 2020-12-02 at 10:44 +0100, Thorsten Kukuk wrote:
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it - let the admin create the configuration file and directory
Thorsten
Maybe like this:
Package the logrotate stuff separately (e.g. as $X-logrotate) and use
Requires: ($X and logrotate) Supplements: ($X and logrotate)
and
Recommends: ($X-logrotate unless systemd)
which means it will never be recommended. I'd go with Recommends: ($X-logrotate unless systemd-logger) instead. -- Frederic Crozat Release Manager SUSE Linux Enterprise SUSE
On Wed, 2 Dec 2020, Frederic Crozat wrote:
Le mercredi 02 décembre 2020 à 12:11 +0000, Martin Wilck a écrit :
On Wed, 2020-12-02 at 10:44 +0100, Thorsten Kukuk wrote:
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it - let the admin create the configuration file and directory
Thorsten
Maybe like this:
Package the logrotate stuff separately (e.g. as $X-logrotate) and use
Requires: ($X and logrotate) Supplements: ($X and logrotate)
and
Recommends: ($X-logrotate unless systemd)
which means it will never be recommended. I'd go with
Recommends: ($X-logrotate unless systemd-logger)
instead.
Plus have $X-systemd and require either $X-logrotate or $X-systemd and ship the actual configuration in them so when $X-logrotate is installed we actually log to /var/log? Richard. -- Richard Biener <rguenther@suse.de> SUSE Software Solutions Germany GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany; GF: Felix Imendörffer; HRB 36809 (AG Nuernberg)
On Wed, Dec 2, 2020 at 7:23 AM Frederic Crozat <FCrozat@suse.com> wrote:
Le mercredi 02 décembre 2020 à 12:11 +0000, Martin Wilck a écrit :
On Wed, 2020-12-02 at 10:44 +0100, Thorsten Kukuk wrote:
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it - let the admin create the configuration file and directory
Thorsten
Maybe like this:
Package the logrotate stuff separately (e.g. as $X-logrotate) and use
Requires: ($X and logrotate) Supplements: ($X and logrotate)
and
Recommends: ($X-logrotate unless systemd)
which means it will never be recommended. I'd go with
Recommends: ($X-logrotate unless systemd-logger)
instead.
Why isn't the journal the default log *anyway*? Especially with Cockpit pretty much relying on a persistent journal, there's no reason not to just do that. -- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, Dec 02, Neal Gompa wrote:
Why isn't the journal the default log *anyway*? Especially with Cockpit pretty much relying on a persistent journal, there's no reason not to just do that.
The default log is the journal? The question is, what to do with all the in the default confiuration useless logfiles, logdirectories, log-configurations, dependencies ... Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg)
On 12/2/20 1:23 PM, Frederic Crozat wrote:
Le mercredi 02 décembre 2020 à 12:11 +0000, Martin Wilck a écrit :
On Wed, 2020-12-02 at 10:44 +0100, Thorsten Kukuk wrote:
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it - let the admin create the configuration file and directory
Thorsten
Maybe like this:
Package the logrotate stuff separately (e.g. as $X-logrotate) and use
Requires: ($X and logrotate) Supplements: ($X and logrotate)
and
Recommends: ($X-logrotate unless systemd)
which means it will never be recommended. I'd go with
Recommends: ($X-logrotate unless systemd-logger)
AFAICS systemd-logger is related to syslog socket. But Thorsten was talking about logrotate for custom log *files*. Ciao, Michael.
Le mercredi 02 décembre 2020 à 15:07 +0100, Michael Ströder a écrit :
On 12/2/20 1:23 PM, Frederic Crozat wrote:
Le mercredi 02 décembre 2020 à 12:11 +0000, Martin Wilck a écrit :
On Wed, 2020-12-02 at 10:44 +0100, Thorsten Kukuk wrote:
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it - let the admin create the configuration file and directory
Thorsten
Maybe like this:
Package the logrotate stuff separately (e.g. as $X-logrotate) and use
Requires: ($X and logrotate) Supplements: ($X and logrotate)
and
Recommends: ($X-logrotate unless systemd)
which means it will never be recommended. I'd go with
Recommends: ($X-logrotate unless systemd-logger)
AFAICS systemd-logger is related to syslog socket. But Thorsten was talking about logrotate for custom log *files*.
No, systemd-logger is just a package which contains /var/log/journal (to cause systemd journal to be persistent) and has the needed provides to allow uninstallation of other syslogs. -- Frederic Crozat Release Manager SUSE Linux Enterprise SUSE
On 12/2/20 1:11 PM, Martin Wilck wrote:
On Wed, 2020-12-02 at 10:44 +0100, Thorsten Kukuk wrote:
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it - let the admin create the configuration file and directory
Maybe like this: Package the logrotate stuff separately (e.g. as $X-logrotate) and use
Extra packages will be an extra maintenance burden. I like Thorsten's first proposal. It's simple. Ciao, Michael.
Am Mittwoch, 2. Dezember 2020, 10:44:50 CET schrieb Thorsten Kukuk:
Hi,
we have quite some packages, which:
1. require logrotate and ship a logrotate configuration file 2. have a special directory in /var/log 3. never create a log file as they are controlled by systemd
The reason the packages never write to the log file by default is, that they log to stdout/stderr and systemd/journald captures them.
What should we do with such packages? Requiring logrotate, even if never used by default, is already a bad idea. It runs regular (so uses resources) without doing anything.
Clobbering /var/log with special directories and files owned by special users will also only trigger actions, even if never used.
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it - let the admin create the configuration file and directory
Hello Thorsten, the first thing i do on a server is to disable systemd-journal and use logrotate. How are you going to look for a bug in the log if you don't know exactly what to look for? So you're just looking for a pattern in the log. That doesn't work with systemd. This only works with text files. And so systemd-journal is completely pointless for servers. Please let it enable. Regards Eric
Hi, On Wed, Dec 02, Eric Schirra wrote:
Hello Thorsten,
the first thing i do on a server is to disable systemd-journal and use logrotate.
logrotate does not write any logfiles, it moves them away if they get to big. So I assume, you disable systemd-journal and use a syslog implementation instead? Doesn't change anything on the picture, applications run via systemd service files still log to stdout/stderr and not in their log file.
How are you going to look for a bug in the log if you don't know exactly what to look for?
Sorry, I don't understand. I don't see a difference in filtering a file or the journalctl output.
So you're just looking for a pattern in the log.
That doesn't work with systemd.
Of course it works? I do that all the day.
This only works with text files.
journalctl output is pure ascii text?
And so systemd-journal is completely pointless for servers.
Please let it enable.
I don't want to disable anything, it is already disabled since a very long time. The output to log files is disabled in 99% of the packages who creates files or directories in /var/log I looked at. Seems like you mix up journald, syslogd, logrotate and applications creating their own log files. Thorsten
Regards
Eric
_______________________________________________ openSUSE Packaging mailing list -- packaging@lists.opensuse.org To unsubscribe, email packaging-leave@lists.opensuse.org List Netiquette: https://en.opensuse.org/openSUSE:Mailing_list_netiquette List Archives: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org
-- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg)
Am Mittwoch, 2. Dezember 2020, 17:10:20 CET schrieb Thorsten Kukuk:
the first thing i do on a server is to disable systemd-journal and use logrotate.
logrotate does not write any logfiles, it moves them away if they get to big. So I assume, you disable systemd-journal and use a syslog implementation instead? Doesn't change anything on the picture, applications run via systemd service files still log to stdout/stderr and not in their log file.
Of course syslog. I know that. And you don't have to make all people stupid here.
How are you going to look for a bug in the log if you don't know exactly what to look for?
optical pattern
Sorry, I don't understand. I don't see a difference in filtering a file or the journalctl output.
So you're just looking for a pattern in the log.
That doesn't work with systemd.
Of course it works? I do that all the day.
This only works with text files.
journalctl output is pure ascii text?
And so systemd-journal is completely pointless for servers.
Please let it enable.
I don't want to disable anything, it is already disabled since a very long time. The output to log files is disabled in 99% of the packages who creates files or directories in /var/log I looked at.
Seems like you mix up journald, syslogd, logrotate and applications creating their own log files.
I don't use journal. It's to slow to search. Not searchable for optical pattern. And i install always on server syslog and logrotate. REgards
On 02.12.20 17:10, Thorsten Kukuk wrote:
Sorry, I don't understand. I don't see a difference in filtering a file or the journalctl output.
Speed. Taking 8 hours to just provide the logs of this boot from the non-persistent ram disk journal(!!) on a pretty big server is a no-no. Just look at the customers support request that caused supportconfig to grow a flag to disable or trim journal collection :-)
Of course it works? I do that all the day.
You seem to only have tiny systems or not really much to do ;-)
journalctl output is pure ascii text?
but it is slllooooooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwww. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
To no one in particular here -- Journald has, for years now, been a standard with overwhelming adoption and precedent, and logging patterns that rely on other mechanisms like logrotate should be viewed as legacy in 2020 -- and generally are by most users. The SystemV fight was lost. It's done. it's over. It's no longer time to adopt systemd. That time has passed. Pursuing legacy logging patterns over personal preference is generally bad for package continuity. Maybe consider moving the logrotate options as annexes in a secondary repository or something. This would be a great place to also move packages maintained by folks who don't think systemd is a base integration requirement until they either drop package or modernize. If you go this no-systemd route discussed here you'll eventually be doing it alone. -C On Sun, Dec 6, 2020 at 5:16 AM Stefan Seyfried < stefan.seyfried@googlemail.com> wrote:
On 02.12.20 17:10, Thorsten Kukuk wrote:
Sorry, I don't understand. I don't see a difference in filtering a file or the journalctl output.
Speed.
Taking 8 hours to just provide the logs of this boot from the non-persistent ram disk journal(!!) on a pretty big server is a no-no.
Just look at the customers support request that caused supportconfig to grow a flag to disable or trim journal collection :-)
Of course it works? I do that all the day.
You seem to only have tiny systems or not really much to do ;-)
journalctl output is pure ascii text?
but it is slllooooooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwww. -- Stefan Seyfried
"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman _______________________________________________ openSUSE Packaging mailing list -- packaging@lists.opensuse.org To unsubscribe, email packaging-leave@lists.opensuse.org List Netiquette: https://en.opensuse.org/openSUSE:Mailing_list_netiquette List Archives: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org
I will also add that if it's taking 8 hours to pull logs from a server, there is a problem elsewhere or perhaps the log aggregation model in that environment should be revisited. That is not typical. On Sun, Dec 6, 2020 at 6:18 AM Chris Punches <punches.chris@gmail.com> wrote:
To no one in particular here --
Journald has, for years now, been a standard with overwhelming adoption and precedent, and logging patterns that rely on other mechanisms like logrotate should be viewed as legacy in 2020 -- and generally are by most users.
The SystemV fight was lost. It's done. it's over. It's no longer time to adopt systemd. That time has passed. Pursuing legacy logging patterns over personal preference is generally bad for package continuity.
Maybe consider moving the logrotate options as annexes in a secondary repository or something. This would be a great place to also move packages maintained by folks who don't think systemd is a base integration requirement until they either drop package or modernize.
If you go this no-systemd route discussed here you'll eventually be doing it alone.
-C
On Sun, Dec 6, 2020 at 5:16 AM Stefan Seyfried < stefan.seyfried@googlemail.com> wrote:
On 02.12.20 17:10, Thorsten Kukuk wrote:
Sorry, I don't understand. I don't see a difference in filtering a file or the journalctl output.
Speed.
Taking 8 hours to just provide the logs of this boot from the non-persistent ram disk journal(!!) on a pretty big server is a no-no.
Just look at the customers support request that caused supportconfig to grow a flag to disable or trim journal collection :-)
Of course it works? I do that all the day.
You seem to only have tiny systems or not really much to do ;-)
journalctl output is pure ascii text?
but it is slllooooooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwww. -- Stefan Seyfried
"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman _______________________________________________ openSUSE Packaging mailing list -- packaging@lists.opensuse.org To unsubscribe, email packaging-leave@lists.opensuse.org List Netiquette: https://en.opensuse.org/openSUSE:Mailing_list_netiquette List Archives: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org
On 06.12.20 12:23, Chris Punches wrote:
I will also add that if it's taking 8 hours to pull logs from a server, there is a problem elsewhere or perhaps the log aggregation model in that environment should be revisited. That is not typical.
It's not "8 hours to pull the logs". "pulling the logs" would have been a matter of seconds. This were machines in the faster end of the spectrum that you could get at that time. They were not particularly big, but had fast CPUs. Something in the 256-512GB RAM, 244 cores class. They had no persistent journal configured, so journal was running from tmpfs. "journalctl -b > file", file also on tmpfs or very fast storage took 8 hours and more, producing some GB (much less than 10GB IIRC) of output. We noticed because SUSE's supportconfig tool took days to finish. It does a "journalctl -b" to collect the journal... ;-) So we created a service request and were told by SUSE professional support that it is unfixable, because the database design of the journal database just does not allow it to be read efficiently. And this mirrors my previous experiences with journald, that it works well on its developers notebook, but not very well on setups that differ from that. Some years ago, persistent journal on rotating rust (aka HDD) was unbearably slow due to fragmentation (somewhen around openSUSE 12.x timeline). -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
In a situation like that if I understand it correctly you've got a tradeoff between default storage and plaintext storage -- size vs speed. You also have aggregation options that, without knowing more details, I would imagine would resolve this. This sounds like an infrastructure architecture problem and not necessarily a journald problem from what I've read here. Interesting stuff. I wouldn't recommend in any distribution to use the default logging configuration for scale. Configuration management and some planning are due there. Sure, it's slow. I can't blame journald for that, I would expect it to be slow. Re: breaking everybody's setup by moving legacy logging tools to a secondary repo-- I would imagine a transition window would be needed. Suse dropped lilo eventually. I'm sure out there, somewhere under various rocks and lurching in untold caves are people still mad about it because their setup worked. If it gets too much pushback we could always just call it DevLogOps or something and let the problem solve itself. -C On Sun, Dec 6, 2020 at 6:48 AM Stefan Seyfried < stefan.seyfried@googlemail.com> wrote:
On 06.12.20 12:23, Chris Punches wrote:
I will also add that if it's taking 8 hours to pull logs from a server, there is a problem elsewhere or perhaps the log aggregation model in that environment should be revisited. That is not typical.
It's not "8 hours to pull the logs". "pulling the logs" would have been a matter of seconds.
This were machines in the faster end of the spectrum that you could get at that time. They were not particularly big, but had fast CPUs. Something in the 256-512GB RAM, 244 cores class. They had no persistent journal configured, so journal was running from tmpfs. "journalctl -b > file", file also on tmpfs or very fast storage took 8 hours and more, producing some GB (much less than 10GB IIRC) of output.
We noticed because SUSE's supportconfig tool took days to finish. It does a "journalctl -b" to collect the journal... ;-)
So we created a service request and were told by SUSE professional support that it is unfixable, because the database design of the journal database just does not allow it to be read efficiently.
And this mirrors my previous experiences with journald, that it works well on its developers notebook, but not very well on setups that differ from that. Some years ago, persistent journal on rotating rust (aka HDD) was unbearably slow due to fragmentation (somewhen around openSUSE 12.x timeline). -- Stefan Seyfried
"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
On Sunday 2020-12-06 12:48, Stefan Seyfried wrote:
On 06.12.20 12:23, Chris Punches wrote:
I will also add that if it's taking 8 hours to pull logs from a server, there is a problem elsewhere or perhaps the log aggregation model in that environment should be revisited. That is not typical.
We noticed because SUSE's supportconfig tool took days to finish. It does a "journalctl -b" to collect the journal... ;-)
So we created a service request and were told by SUSE professional support that it is unfixable, because the database design of the journal database just does not allow it to be read efficiently.
It speaks volumes that the professional support providers are not tempted to fix it. "Got a problem? Come back when you have the logs. That is, if you ever do - *evil laughter*" Somebody should really cancel their support subscriptions now.
On 06.12.20 14:10, Jan Engelhardt wrote:
On Sunday 2020-12-06 12:48, Stefan Seyfried wrote: >> We noticed because SUSE's supportconfig tool took days to finish. It does a
"journalctl -b" to collect the journal... ;-)
So we created a service request and were told by SUSE professional support that it is unfixable, because the database design of the journal database just does not allow it to be read efficiently.
It speaks volumes that the professional support providers are not tempted to fix it.
Oh, they were trying and willing to fix it. IIRC actually the first response was "well, let's just fix journal". Then they looked closer at the problem and saw that this was not really possible.
"Got a problem? Come back when you have the logs. That is, if you ever do - *evil laughter*"
they did fix our problem -- by just adding the option to supportconfig to disable fetching of journal logs (important parts of /var/log/messages were collected anyway) I would not really want or expect SUSE to rewrite journald for a released product in a backwards incompatible way (since it is the "database format design" of the journald, it's basically impossible to fix in a backwards-compatible way). -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
On Sunday 2020-12-06 14:34, Stefan Seyfried wrote:
So we created a service request and were told by SUSE professional support that it is unfixable, because the database design of the journal database just does not allow it to be read efficiently.
It speaks volumes that the professional support providers are not tempted to fix it.
Oh, they were trying and willing to fix it. IIRC actually the first response was "well, let's just fix journal". Then they looked closer at the problem and saw that this was not really possible.
"Everyone said that's not possible. Then someone who didn't know that came along, and did it." It's software. No situation is truly unfixable. In other places, they are replacing pulseaudio by the next "implementation du jour" pipewire, and somehow the same can't be done with sd-journal?
On 06.12.20 12:18, Chris Punches wrote:
To no one in particular here --
Journald has, for years now, been a standard with overwhelming adoption and precedent, and logging patterns that rely on other mechanisms like logrotate should be viewed as legacy in 2020 -- and generally are by most users.
This still does not make ist suitable for every use case. I do use the journal, usually have even persistent journal enabled, but am also additionally using a syslog daemon for the case that I actually need to urgently find something in the logs. Which is easy and fast with grep and friends, and takes days with journalctl. The journal database has been described by a developer that was tasked to fix the issue for us as "generally designed as a write-only system", optimizes for fast writes. But apparently this makes it almost impossible to fix the "it is slow as molasses and cannot even be enhanced with more CPU power" problems. So the only solution to this is "add another level of indirection" (see RFC1925) and use a syslog solution that additionally logs to plain files. Which can be read quickly.
The SystemV fight was lost. It's done. it's over. It's no longer time to adopt systemd. That time has passed. Pursuing legacy logging patterns over personal preference is generally bad for package continuity.
This has nothing to do with the sysvinit fight. It is just that journald is unsuitable beyond the scope of its developer's notebook. It has proven so multiple times in the past. So we need a workaround since we no longer realistically can get rid of it as you correctly state.
If you go this no-systemd route discussed here you'll eventually be doing it alone.
Nobody in this thread talked about going a "no-systemd route2. It's just about not breaking everyone's setup. While Thorsten is right that it should be possible to install a system without syslog and logrotate, and that packages that do not even log anyhting in /var/log in their shipped default configuration have no business to recommend or even require syslog and logrotate at all, your idea of "just rip that all out and drop logrotate and syslog" will lead to useless, unusable systems. Being a useful solution for lots of use cases (and many beyound "journal developer's notebook") is one of openSUSE's goals, at least that's how I understand it. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
On 12/6/20 12:18 PM, Chris Punches wrote:
Journald has, for years now, been a standard with overwhelming adoption and precedent, and logging patterns that rely on other mechanisms like logrotate should be viewed as legacy in 2020 -- and generally are by most users.
But using systemd-journald for everything has serious real-world performance issues on non-desktop systems. It's astonishing that they were not adressed during these years of broader adoption. Ciao, Michael. P.S.: To make it very clear: I'm not a systemd opponent, e.g. I highly appreciate the security features in systemd units and would like to see broader adoption of these in packaged unit files. But some systemd components are still not up to be fully useable with systemd-journald being one of them.
No it doesn't. On Sun, Dec 6, 2020 at 10:37 AM Michael Ströder <michael@stroeder.com> wrote:
On 12/6/20 12:18 PM, Chris Punches wrote:
Journald has, for years now, been a standard with overwhelming adoption and precedent, and logging patterns that rely on other mechanisms like logrotate should be viewed as legacy in 2020 -- and generally are by most users.
But using systemd-journald for everything has serious real-world performance issues on non-desktop systems. It's astonishing that they were not adressed during these years of broader adoption.
Ciao, Michael.
P.S.: To make it very clear: I'm not a systemd opponent, e.g. I highly appreciate the security features in systemd units and would like to see broader adoption of these in packaged unit files. But some systemd components are still not up to be fully useable with systemd-journald being one of them. _______________________________________________ openSUSE Packaging mailing list -- packaging@lists.opensuse.org To unsubscribe, email packaging-leave@lists.opensuse.org List Netiquette: https://en.opensuse.org/openSUSE:Mailing_list_netiquette List Archives: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org
On Sun, 2020-12-06 at 15:49 -0500, Chris Punches wrote:
No it doesn't.
What do you mean? What doesn't what? Martin PS: You might avoid questions like this by refraining from top-posting.
On Sun, Dec 6, 2020 at 10:37 AM Michael Ströder <michael@stroeder.com
Journald has, for years now, been a standard with overwhelming adoption and precedent, and logging patterns that rely on other mechanisms
wrote: On 12/6/20 12:18 PM, Chris Punches wrote: like
logrotate should be viewed as legacy in 2020 -- and generally are by most users.
But using systemd-journald for everything has serious real-world performance issues on non-desktop systems. It's astonishing that they were not adressed during these years of broader adoption.
Ciao, Michael.
P.S.: To make it very clear: I'm not a systemd opponent, e.g. I highly appreciate the security features in systemd units and would like to see broader adoption of these in packaged unit files. But some systemd components are still not up to be fully useable with systemd- journald being one of them.
-- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Software Solutions Germany GmbH HRB 36809, AG Nürnberg GF: Felix Imendörffer
Am Sonntag, 6. Dezember 2020, 11:16:11 CET schrieb Stefan Seyfried:
On 02.12.20 17:10, Thorsten Kukuk wrote:
Sorry, I don't understand. I don't see a difference in filtering a file or the journalctl output. Speed.
+1
Of course it works? I do that all the day. You seem to only have tiny systems or not really much to do ;-)
+1
journalctl output is pure ascii text? but it is slllooooooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwww.
+1 Regards Eric
On Mittwoch, 2. Dezember 2020 17:00:47 CET Eric Schirra wrote:
How are you going to look for a bug in the log if you don't know exactly what to look for? So you're just looking for a pattern in the log. That doesn't work with systemd. This only works with text files.
Would be nice if people stopped spreading nonsense ... You can either pipe the journalctl output wherever you want, or even better, use journalctl's "--grep" option.
And so systemd-journal is completely pointless for servers.
It is only pointless for the few who reject to learn something new every few years. Regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen phone: +49 241 53809034 mobile: +49 151 50412019
On Wed, 2020-12-02 at 20:11 +0100, Stefan Brüns wrote:
On Mittwoch, 2. Dezember 2020 17:00:47 CET Eric Schirra wrote:
How are you going to look for a bug in the log if you don't know exactly what to look for? So you're just looking for a pattern in the log. That doesn't work with systemd. This only works with text files.
Would be nice if people stopped spreading nonsense ...
You can either pipe the journalctl output wherever you want, or even better, use journalctl's "--grep" option.
And so systemd-journal is completely pointless for servers.
It is only pointless for the few who reject to learn something new every few years.
Perhaps also for those who are dissatisfied by the journal's performance. https://lists.opensuse.org/opensuse-factory/2019-11/msg00287.html Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Software Solutions Germany GmbH HRB 36809, AG Nürnberg GF: Felix Imendörffer
Am Mittwoch, 2. Dezember 2020, 20:11:40 CET schrieb Stefan Brüns:
On Mittwoch, 2. Dezember 2020 17:00:47 CET Eric Schirra wrote:
How are you going to look for a bug in the log if you don't know exactly what to look for? So you're just looking for a pattern in the log. That doesn't work with systemd. This only works with text files.
Would be nice if people stopped spreading nonsense ...
This is not nonsense. It seems you've never looked for a real mistake. Not an error that is marked with an error, but an error that makes the log file look "different". Systemd-journal cannot do that.
You can either pipe the journalctl output wherever you want, or even better, use journalctl's "--grep" option.
I know that. Although totally awkward.
And so systemd-journal is completely pointless for servers.
It is only pointless for the few who reject to learn something new every few years.
It has absolutely nothing to do with the fact that I don't want to learn anything. But not everything new is good. And you don't get as far with everything new as with the tried and tested. And just as you assume that I don't want to learn anything new, I'll assume that you are a systemd fanboy who ignores the practice. REgards
Hello, Am Mittwoch, 2. Dezember 2020, 10:44:50 CET schrieb Thorsten Kukuk:
we have quite some packages, which:
1. require logrotate and ship a logrotate configuration file 2. have a special directory in /var/log 3. never create a log file as they are controlled by systemd
The reason the packages never write to the log file by default is, that they log to stdout/stderr and systemd/journald captures them.
Correct me if I'm wrong, but when using syslog-ng, rsyslog etc., these messages will probably end up in /var/log/messages. (At least I have quite some kmail and akonadiserver messages there, and I'm sure they don't write to syslog directly.) And even if your mail didn't target syslog daemons, they are not too different when it comes to requiring logrotate.
What should we do with such packages? Requiring logrotate, even if never used by default, is already a bad idea. It runs regular (so uses resources) without doing anything.
Agreed in theory, even if the resource usage is minor. However, a superfluous logrotate run is still much better than not having logrotate installed, and to get multi-GB logfiles because nothing rotates and compresses them. (I've seen more than one full disk because of such a monster logfile.)
Clobbering /var/log with special directories and files owned by special users will also only trigger actions, even if never used.
Which actions? As long as those files stay empty, I'm sure that logrotate won't rotate them ;-)
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it
I'd guesstimate ;-) that you'll still find lots of syslog usage, especially on servers. Also many people use --no-recommends on servers which in the end would mean no logrotate and multi-GB logfiles. Let me extend your proposal a bit: - if a program always writes its own logfiles (for example apache, salt, also syslog-ng, rsyslog etc.), it should _require_ logrotate - if a program writes to stdout/stderr (which ends up in the journal or gets forwarded to syslog* that, see above, already requires logrotate) and only writes its own logfile after a config change, recommending logrotate is ok.
- let the admin create the configuration file and directory
I guess you meant "_log_ file and directory"? An unused logfile or directory might be an annoyance, but hurts much less than a problem caused by a missing log directory which stops a program from writing its log. Well, you can always look for the startup error of $program in its logfile - oh wait... ;-)) Therefore I'd apply a similar policy as for requiring logrotate - programs that write a logfile in the default config (apache, salt, ...) should also ship their log directories. As a sort-of-disclaimer, even if it should be obvious: I prefer syslog over journal, and have syslog* running everywhere. Regards, Christian Boltz -- Was spricht gegen einen Punkt im Expertenmodus: [ ] Ich weiß nicht, was eine Partition ist. Wenn einer das anklickt, ist klar, daß er Anfänger ist. [Bernd Brodesser in suse-linux]
On Wed, Dec 02, Christian Boltz wrote:
Hello,
Am Mittwoch, 2. Dezember 2020, 10:44:50 CET schrieb Thorsten Kukuk:
we have quite some packages, which:
1. require logrotate and ship a logrotate configuration file 2. have a special directory in /var/log 3. never create a log file as they are controlled by systemd
The reason the packages never write to the log file by default is, that they log to stdout/stderr and systemd/journald captures them.
Correct me if I'm wrong, but when using syslog-ng, rsyslog etc., these messages will probably end up in /var/log/messages. (At least I have quite some kmail and akonadiserver messages there, and I'm sure they don't write to syslog directly.)
And even if your mail didn't target syslog daemons, they are not too different when it comes to requiring logrotate.
I'm not speaking about syslog daemons. It's absolut necessary that syslog daemons require logrotate to make sure /var/log/messages isn't growing to much. I'm speaking about tools like chrony, minidlna and similar, which don't log via syslog(), but either write to stderr/stdout and if not started by systemd, write into their own log files somewhere below /var/log.
What should we do with such packages? Requiring logrotate, even if never used by default, is already a bad idea. It runs regular (so uses resources) without doing anything.
Agreed in theory, even if the resource usage is minor.
However, a superfluous logrotate run is still much better than not having logrotate installed, and to get multi-GB logfiles because nothing rotates and compresses them. (I've seen more than one full disk because of such a monster logfile.)
If package still creates log files, it has to require logrotate, correct. But I'm speaking about all the cases, where the packages could theoretically create log files, but disabled it on purpose.
Clobbering /var/log with special directories and files owned by special users will also only trigger actions, even if never used.
Which actions? As long as those files stay empty, I'm sure that logrotate won't rotate them ;-)
Of course not, but logrotate is run regulary by a sysemd timer or cron job, calls a stat on the files, etc. That's, if you log everything to systemd-journal as we do by default, is pretty useless and e.g. only activates disks from power save modes for nothing.
What would be a good solution? I understand that some people may want to run this tools without control by systemd. But on the other side, in this cases the people have to adjust the configuration anyways.
Maybe a good solution could be: - continue to ship the logrotate configuration file - only Recommend logrotate, not require it
I'd guesstimate ;-) that you'll still find lots of syslog usage, especially on servers. Also many people use --no-recommends on servers which in the end would mean no logrotate and multi-GB logfiles.
Again, we are not speaking about syslog, we are speaking about applications which can optional write directly their own log files. To make this happen, you have to: - adjust the configuration - don't start the service by systemd (starting by systemd is the default) Not sure if this is really a valid use case today anymore.
Let me extend your proposal a bit: - if a program always writes its own logfiles (for example apache, salt, also syslog-ng, rsyslog etc.), it should _require_ logrotate - if a program writes to stdout/stderr (which ends up in the journal or gets forwarded to syslog* that, see above, already requires logrotate) and only writes its own logfile after a config change, recommending logrotate is ok.
- let the admin create the configuration file and directory
I guess you meant "_log_ file and directory"?
An unused logfile or directory might be an annoyance, but hurts much less than a problem caused by a missing log directory which stops a program from writing its log. Well, you can always look for the startup error of $program in its logfile - oh wait... ;-))
Therefore I'd apply a similar policy as for requiring logrotate - programs that write a logfile in the default config (apache, salt, ...) should also ship their log directories.
That's not the case about which we are speaking.
As a sort-of-disclaimer, even if it should be obvious: I prefer syslog over journal, and have syslog* running everywhere.
And it is completly irrelevant if you use syslog or systemd-journal, the applications will still not write in their optional log files, but continue to write to stderr/stdout and systemd is writing this to either systemd-journal or syslogd. In all this cases it still doesn't make sense to me that we create all the directories and log files, which will never be used and only confuse people if they look e.g. for error messages. Thorsten
Regards,
Christian Boltz -- Was spricht gegen einen Punkt im Expertenmodus: [ ] Ich weiß nicht, was eine Partition ist. Wenn einer das anklickt, ist klar, daß er Anfänger ist. [Bernd Brodesser in suse-linux] _______________________________________________ openSUSE Packaging mailing list -- packaging@lists.opensuse.org To unsubscribe, email packaging-leave@lists.opensuse.org List Netiquette: https://en.opensuse.org/openSUSE:Mailing_list_netiquette List Archives: https://lists.opensuse.org/archives/list/packaging@lists.opensuse.org
-- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg)
Hello, Am Donnerstag, 3. Dezember 2020, 09:19:43 CET schrieb Thorsten Kukuk:
On Wed, Dec 02, Christian Boltz wrote:
Am Mittwoch, 2. Dezember 2020, 10:44:50 CET schrieb Thorsten Kukuk:
[...]
I'm not speaking about syslog daemons. It's absolut necessary that syslog daemons require logrotate to make sure /var/log/messages isn't growing to much.
I'm speaking about tools like chrony, minidlna and similar, which don't log via syslog(), but either write to stderr/stdout and if not started by systemd, write into their own log files somewhere below /var/log.
This either wasn't clear in your initial mail, or I overlooked it. Thanks for the clarification, it adresses my concerns. I agree that in these cases requiring logrotate indeed doesn't make sense. [...]
If package still creates log files, it has to require logrotate, correct. But I'm speaking about all the cases, where the packages could theoretically create log files, but disabled it on purpose.
Agreed once more ;-)
Clobbering /var/log with special directories and files owned by special users will also only trigger actions, even if never used.
Which actions? As long as those files stay empty, I'm sure that logrotate won't rotate them ;-)
Of course not, but logrotate is run regulary by a sysemd timer or cron job, calls a stat on the files, etc. That's, if you log everything to systemd-journal as we do by default, is pretty useless and e.g. only activates disks from power save modes for nothing.
Agreed in general, but /etc/logrotate.d/zypper.lr (rotating /var/log/ zypper.log) and /etc/logrotate.d/firewalld (rotating /var/log/firewalld) basically means we won't get rid of logrotate anyway ;-) (Well, we probably don't have zypper and firewalld inside containers, but that's a different story.) [...]
Again, we are not speaking about syslog, we are speaking about applications which can optional write directly their own log files. To make this happen, you have to: - adjust the configuration - don't start the service by systemd (starting by systemd is the default)
Not sure if this is really a valid use case today anymore.
If both conditions need to be fulfilled to make a program write its own logfile, that's indeed a bit exotic. I would still call it a valid (and rare) usecase ;-) but I also understand that in such cases requiring logrotate would be too much. However, we also have programs where adjusting the configuration is enough to make them write a logfile, even if they are still started by systemd. One example I know is vsftpd - you can switch between using syslog and logging to its own logfile with a config option. (I'm saying this based on the config file, never tested it myself.) Actually chrony (one of the examples you mentioned) also belongs to this category - I just enabled log measurements statistics tracking in chrony.conf, and after restarting chronyd, I now have # ls -l /var/log/chrony/ -rw-r--r-- 1 chrony chrony 2603 4. Dez 00:26 measurements.log -rw-r--r-- 1 chrony chrony 1287 4. Dez 00:26 statistics.log -rw-r--r-- 1 chrony chrony 660 4. Dez 00:26 tracking.log This also means chrony should continue to ship the empty log directory - otherwise it will probably run into problems when trying to create these logfiles. We could discuss if in such cases logrotate should be required or only recommended - but given that it's already required for zypper.log, it won't change much ;-) Regards, Christian Boltz -- The only reason coders' computers work better than non-coders' computers is coders know computers are schizophrenic little children with auto-immune diseases and we don't beat them when they're bad. [http://www.stilldrinking.org/programming-sucks]
participants (12)
-
Chris Punches -
Christian Boltz -
Eric Schirra -
Frederic Crozat -
Jan Engelhardt -
Martin Wilck -
Michael Ströder -
Neal Gompa -
Richard Biener -
Stefan Brüns -
Stefan Seyfried -
Thorsten Kukuk