[opensuse] Why are systemd's logs stored as binaries?
I'd like to know more about this. In particular from a authenticity and integrity perspective. Comments about those, or pointers to discussion about same would be appreciated. ==== Here is Lennart Poettering's list of problems with what journald replaces: 1) The message data is generally not authenticated, every local process can claim to be Apache under PID 4711, and syslog will believe that and store it on disk. 2) The data logged is very free-form. Automated log-analyzers need to parse human language strings to a) identify message types, and b) parse parameters from them. This results in regex horrors, and a steady need to play catch-up with upstream developers who might tweak the human language log strings in new versions of their software. Effectively, in a away, in order not to break user-applied regular expressions all log messages become ABI of the software generating them, which is usually not intended by the developer. 3) The timestamps generally do not carry timezone information, even though some newer specifications define support for it. 4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems. 5) Reading log files is simple but very inefficient. Many key log operations have a complexity of O(n). Indexing is generally not available. 6) The syslog network protocol is very simple, but also very limited. Since it generally supports only a push transfer model, and does not employ store-and-forward, problems such as Thundering Herd or packet loss severely hamper its use. 7) Log files are easily manipulable by attackers, providing easy ways to hide attack information from the administrator 8) Access control is non-existent. Unless manually scripted by the administrator a user either gets full access to the log files, or no access at all. 9) The meta data stored for log entries is limited, and lacking key bits of information, such as service name, audit session or monotonic timestamps. 10) Automatic rotation of log files is available, but less than ideal in most implementations: instead of watching disk usage continuously to enforce disk usage limits rotation is only attempted in fixed time intervals, thus leaving the door open to many DoS attacks. 11) Rate limiting is available in some implementations, however, generally does not take the disk usage or service assignment into account, which is highly advisable. 12) Compression in the log structure on disk is generally available but usually only as effect of rotation and has a negative effect on the already bad complexity behaviour of many key log operations. 13) Classic Syslog traditionally is not useful to handle early boot or late shutdown logging, even though recent improvements (for example in systemd) made this work. 14) Binary data cannot be logged, which in some cases is essential (Examples: ATA SMART blobs or SCSI sense data, firmware dumps) Thanks Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sunday, 18 December 2016 14:21:50 GMT Greg Freemyer wrote:
I'd like to know more about this. In particular from a authenticity and integrity perspective.
Comments about those, or pointers to discussion about same would be appreciated.
You are asking for trouble, are you wanting a thread to keep you busy over christmas?... :o)
==== Here is Lennart Poettering's list of problems with what journald replaces:
1) The message data is generally not authenticated, every local process can claim to be Apache under PID 4711, and syslog will believe that and store it on disk.
2) The data logged is very free-form. Automated log-analyzers need to parse human language strings to a) identify message types, and b) parse parameters from them. This results in regex horrors, and a steady need to play catch-up with upstream developers who might tweak the human language log strings in new versions of their software. Effectively, in a away, in order not to break user-applied regular expressions all log messages become ABI of the software generating them, which is usually not intended by the developer.
3) The timestamps generally do not carry timezone information, even though some newer specifications define support for it.
4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.
5) Reading log files is simple but very inefficient. Many key log operations have a complexity of O(n). Indexing is generally not available.
6) The syslog network protocol is very simple, but also very limited. Since it generally supports only a push transfer model, and does not employ store-and-forward, problems such as Thundering Herd or packet loss severely hamper its use.
7) Log files are easily manipulable by attackers, providing easy ways to hide attack information from the administrator
8) Access control is non-existent. Unless manually scripted by the administrator a user either gets full access to the log files, or no access at all.
9) The meta data stored for log entries is limited, and lacking key bits of information, such as service name, audit session or monotonic timestamps.
10) Automatic rotation of log files is available, but less than ideal in most implementations: instead of watching disk usage continuously to enforce disk usage limits rotation is only attempted in fixed time intervals, thus leaving the door open to many DoS attacks.
11) Rate limiting is available in some implementations, however, generally does not take the disk usage or service assignment into account, which is highly advisable.
12) Compression in the log structure on disk is generally available but usually only as effect of rotation and has a negative effect on the already bad complexity behaviour of many key log operations.
13) Classic Syslog traditionally is not useful to handle early boot or late shutdown logging, even though recent improvements (for example in systemd) made this work.
14) Binary data cannot be logged, which in some cases is essential (Examples: ATA SMART blobs or SCSI sense data, firmware dumps)
Thanks Greg -- Greg Freemyer
-- opensuse:tumbleweed:20161216 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 kmail5-16.08.3-1.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Dec 19, 2016 at 5:22 AM, ianseeks
On Sunday, 18 December 2016 14:21:50 GMT Greg Freemyer wrote:
I'd like to know more about this. In particular from a authenticity and integrity perspective.
Comments about those, or pointers to discussion about same would be appreciated.
You are asking for trouble, are you wanting a thread to keep you busy over christmas?... :o)
I'm hoping the discussion can stick to help understand what benefits journald and its binary logs brings to the table. I don't care if the discussion hits pros and cons, but hopefully it can stick to logging specific issues and not get dragged into an overall systemd discussion. Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 11:40 AM, Greg Freemyer wrote:
I'm hoping the discussion can stick to help understand what benefits journald and its binary logs brings to the table.
Well, here are some illustrations of things that are remarkable easy with journalctl but would be difficult with syslog. https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-vi... https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-vi... Under https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-vi... there is the example of output in JSON format. This gets back to my point about processing by such tools as PERL. Read in the JSON format as hashes and slice and dice and correlate to your heart's content :-) -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Monday, 19 December 2016 11:40:05 GMT Greg Freemyer wrote:
On Mon, Dec 19, 2016 at 5:22 AM, ianseeks
wrote: On Sunday, 18 December 2016 14:21:50 GMT Greg Freemyer wrote:
I'd like to know more about this. In particular from a authenticity and integrity perspective.
Comments about those, or pointers to discussion about same would be appreciated.
You are asking for trouble, are you wanting a thread to keep you busy over christmas?... :o)
I'm hoping the discussion can stick to help understand what benefits journald and its binary logs brings to the table.
Yep, we all hope for that.
I don't care if the discussion hits pros and cons, but hopefully it can stick to logging specific issues and not get dragged into an overall systemd discussion.
Greg
-- Greg Freemyer
-- opensuse:tumbleweed:20161217 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
For me more important is the question
"How can I access the binary systemd log files if the system itself
doesn't run anymore?"
Do I think to complicated?
Hints on recommended reading how to handle this situation are very
appreciated.
Am Sun, 18 Dec 2016 14:21:50 -0500
schrieb Greg Freemyer
I'd like to know more about this. In particular from a authenticity and integrity perspective.
Comments about those, or pointers to discussion about same would be appreciated.
[snip of Lennart Poettering's long list ]
Thanks Greg -- Greg Freemyer
-- Mit freundlichen Grüßen Best Regards Peter Ragosch -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
from a usb boot and pass the journal path to journalctl or am i
missing something?
On 19 December 2016 at 12:24, Peter Ragosch
For me more important is the question
"How can I access the binary systemd log files if the system itself doesn't run anymore?"
Do I think to complicated?
Hints on recommended reading how to handle this situation are very appreciated.
Am Sun, 18 Dec 2016 14:21:50 -0500 schrieb Greg Freemyer
: I'd like to know more about this. In particular from a authenticity and integrity perspective.
Comments about those, or pointers to discussion about same would be appreciated.
[snip of Lennart Poettering's long list ]
Thanks Greg -- Greg Freemyer
-- Mit freundlichen Grüßen Best Regards
Peter Ragosch
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-19 12:24, Peter Ragosch wrote:
For me more important is the question
"How can I access the binary systemd log files if the system itself doesn't run anymore?"
Do I think to complicated?
Hints on recommended reading how to handle this situation are very appreciated.
Maybe this (man journalctl): --file=GLOB Takes a file glob as an argument. If specified, journalctl will operate on the specified journal files matching GLOB instead of the default runtime and system journal paths. May be specified multiple times, in which case files will be suitably interleaved. --root=ROOT Takes a directory path as an argument. If specified, journalctl will operate on catalog file hierarchy underneath the specified directory instead of the root directory (e.g. --update-catalog will create ROOT/var/lib/systemd/catalog/database). But probably has to be of the same version. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYA+cACgkQja8UbcUWM1wXhgD9Fsn1ABUbSBLTJQ+2juoaDl+S PGhUHTVyv8pNuVgkp+gA/1g33wH3jTLbWLuNnRzNOs5iiPZ14O13uEajvJHwj0pL =sNYY -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/18/2016 02:21 PM, Greg Freemyer wrote:
4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.
Hmm. So if I run journald can I get rid of all those other log files? either delete them and hope the subsystem won't recreate them or link them to /dev/null ? More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier. we just need tools! -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-19 16:29, Anton Aylward wrote:
On 12/18/2016 02:21 PM, Greg Freemyer wrote:
4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.
Hmm. So if I run journald can I get rid of all those other log files?
What other log files? You mean non-syslog log files? Those other log files are not handled by journald, either. They are handled by the specific applications that create them.
either delete them and hope the subsystem won't recreate them or link them to /dev/null ?
More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier.
No correlation unless you have a database viewer and analysis tool. But then, logs may be as difficult to analyze as they currently are in Windows. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYAxQACgkQja8UbcUWM1xzMQD/bYOxs07UsLzibXgc/8GshY1w Ht80nDT8rYrvDmydfcwA/3VfOLD8rmhJcCoC5Wx//4mG9+HivIuI4+nVB2HNOWKA =4zVW -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 10:56 AM, Carlos E. R. wrote:
On 2016-12-19 16:29, Anton Aylward wrote:
On 12/18/2016 02:21 PM, Greg Freemyer wrote:
4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.
Hmm. So if I run journald can I get rid of all those other log files?
What other log files? You mean non-syslog log files? Those other log files are not handled by journald, either. They are handled by the specific applications that create them.
In that any program can, yes, but that's not my point. My point is that that journald logs so many of these things, or can. journalctl _COMM=sshd journalctl _COMM=login journalctl _COMM=su journalctl _COMM=systemd-logind Of course most access involves PAM one way or another so just make sure you make use of pam_logind !!! if you really want to be paranoid, you can forget about the indexoing that journalctl can do and simple grep journalctl |grep login | more Oh My! Well that soaks up quite a few of those log files: utmp/wtmp, lastlog
More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier.
No correlation unless you have a database viewer and analysis tool.
Ah, tools! Well, for a start, journalctl does index. Then we have a;; the regular tools, from 'cut'/'join' of V7 UNIX vintage, on though 'awk' (see the appropriate White Book for how to convert text to database and do correlations and produce nicely formatted output) on though Perl, which was designed for doing that kind of thing. You've been around long enough, Carlos, to know and have done this sort of thing. What's this, some anti-systemd feeling?
But then, logs may be as difficult to analyze as they currently are in Windows.
This isn't Windows. <obscentity> Windows. part of the nature of Windows is to make you buy some third party tool to do things that with UNIX/Linux can be done with a few lies of scripting. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-19 18:24, Anton Aylward wrote:
On 12/19/2016 10:56 AM, Carlos E. R. wrote:
On 2016-12-19 16:29, Anton Aylward wrote:
On 12/18/2016 02:21 PM, Greg Freemyer wrote:
4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.
Hmm. So if I run journald can I get rid of all those other log files?
What other log files? You mean non-syslog log files? Those other log files are not handled by journald, either. They are handled by the specific applications that create them.
In that any program can, yes, but that's not my point.
My point is that that journald logs so many of these things, or can.
journalctl _COMM=sshd journalctl _COMM=login journalctl _COMM=su journalctl _COMM=systemd-logind
Of course most access involves PAM one way or another so just make sure you make use of pam_logind !!!
That's just login sessions of different types. I don't remember where these where kept before.
if you really want to be paranoid, you can forget about the indexoing that journalctl can do and simple grep
journalctl |grep login | more
Takes minutes with rotating rust.
Oh My!
Well that soaks up quite a few of those log files: utmp/wtmp, lastlog
More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier.
No correlation unless you have a database viewer and analysis tool.
Ah, tools!
Well, for a start, journalctl does index.
What does that mean? Examples, please.
Then we have a;; the regular tools, from 'cut'/'join' of V7 UNIX vintage, on though 'awk' (see the appropriate White Book for how to convert text to database and do correlations and produce nicely formatted output) on though Perl, which was designed for doing that kind of thing.
Nonono. I want ready made user end tools. I do not want to learn programming awk now or have to write my own tools.
You've been around long enough, Carlos, to know and have done this sort of thing. What's this, some anti-systemd feeling?
No, not at all.
But then, logs may be as difficult to analyze as they currently are in Windows.
This isn't Windows. <obscentity> Windows. part of the nature of Windows is to make you buy some third party tool to do things that with UNIX/Linux can be done with a few lies of scripting.
The journal thing in Linux is getting closer to what Windows does with logs. That's my fear. I'm unable to find events in Windows system log, that I know they are there. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYGZsACgkQja8UbcUWM1y0RAD8CLPB3Ruw2jSkTVi5fsdrMf6y qUR0g754lfaI6GNgIXcBAJ9uFDDxqZDmrmR6d8TcGZqNV/AbkWLCiBCLTVnbmezj =alyw -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 12:32 PM, Carlos E. R. wrote:
On 2016-12-19 18:24, Anton Aylward wrote:
That's just login sessions of different types. I don't remember where these where kept before.
In at least three different files, perhaps five or more depending on the mode of access.
if you really want to be paranoid, you can forget about the indexoing that journalctl can do and simple grep
journalctl |grep login | more
Takes minutes with rotating rust.
Minutes to do what? On my rotating rust I get output immediately. Yes, if you want to wait for the very end and you are one of the people who have a gazillion of gigabytes of log files, but that reasoning applies too if you want to get every login from every individual source (since this covers in my case dovecot access, smptd/postfix access, Xlog and more, it ends up being about 10 different files). Rotating rust of syslog files isn't going to be any faster. You're clutching at straws here, Carlos.
No correlation unless you have a database viewer and analysis tool.
Ah, tools!
Well, for a start, journalctl does index.
What does that mean? Examples, please.
I gave some examples, perhaps you didn't notice. Something like journalctl _COMM=sshd vs journalctl | grep ssh
Nonono. I want ready made user end tools. I do not want to learn programming awk now or have to write my own tools.
That's fine. Go off and use Windows then, accept someone elses' idea of how the UI should look, what data you're allowed to display and how to see it and what you can do with it and more to the point what it won't permit you to do with it. You really shouldn't be using Linux if that's your attitude.
You've been around long enough, Carlos, to know and have done this sort of thing. What's this, some anti-systemd feeling?
No, not at all.
Well, that's how its coming across.
But then, logs may be as difficult to analyze as they currently are in Windows.
This isn't Windows. <obscentity> Windows. part of the nature of Windows is to make you buy some third party tool to do things that with UNIX/Linux can be done with a few lies of scripting.
The journal thing in Linux is getting closer to what Windows does with logs. That's my fear. I'm unable to find events in Windows system log, that I know they are there.
There are forms of anti-systemd manifestation. A reluctance to learn how to manage the new facilities is one of them. This is all new to e as well, but I'm willing to accept it, not fight it, to experiment and learn. I figure that Lennart Poettering is smarter than me. What I understand makes sense and what I understand allows me to, slowly, learn more. UNIX, Linux has always had for me a joy in learning and a WOW! factor and a 'hey, that's neat!" aspect. I've never had that with Windows (which was always A source of frustration) or even VMS (and yes even when I used the RATFOR Software Tools). -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-19 19:09, Anton Aylward wrote:
On 12/19/2016 12:32 PM, Carlos E. R. wrote:
On 2016-12-19 18:24, Anton Aylward wrote:
journalctl |grep login | more
Takes minutes with rotating rust.
Minutes to do what? On my rotating rust I get output immediately. Yes, if you want to wait for the very end and you are one of the people who have a gazillion of gigabytes of log files, but that reasoning applies too if you want to get every login from every individual source (since this covers in my case dovecot access, smptd/postfix access, Xlog and more, it ends up being about 10 different files).
Rotating rust of syslog files isn't going to be any faster.
No, it is not. I measured the speed of the same test against the traditional log and journal logs, and finding entries with grep was an order of magnitude faster. I demonstrated that months ago on this list. I can not prove it today again because journalctl output is empty in this machine. I disabled it.
No correlation unless you have a database viewer and analysis tool.
Ah, tools!
Well, for a start, journalctl does index.
What does that mean? Examples, please.
I gave some examples, perhaps you didn't notice.
Something like journalctl _COMM=sshd vs journalctl | grep ssh
Ah, that's an index. New to me. Thanks. What are the possible indexes, how does it work?
Nonono. I want ready made user end tools. I do not want to learn programming awk now or have to write my own tools.
That's fine. Go off and use Windows then, accept someone elses' idea of how the UI should look, what data you're allowed to display and how to see it and what you can do with it and more to the point what it won't permit you to do with it.
You really shouldn't be using Linux if that's your attitude.
Well, if all people help that way, we are going very far... In Windows I accesses such things using Microsft Access, for instance, where I could apply any filter I could desire. now, if you tell me how I can import the journal into Libre Office...
You've been around long enough, Carlos, to know and have done this sort of thing. What's this, some anti-systemd feeling?
No, not at all.
Well, that's how its coming across.
It is all in the eyes of the beholder. You are waiting to see something, so you see it even if it is not there.
But then, logs may be as difficult to analyze as they currently are in Windows.
This isn't Windows. <obscentity> Windows. part of the nature of Windows is to make you buy some third party tool to do things that with UNIX/Linux can be done with a few lies of scripting.
The journal thing in Linux is getting closer to what Windows does with logs. That's my fear. I'm unable to find events in Windows system log, that I know they are there.
There are forms of anti-systemd manifestation. A reluctance to learn how to manage the new facilities is one of them.
Then teach!
This is all new to e as well, but I'm willing to accept it, not fight it, to experiment and learn. I figure that Lennart Poettering is smarter than me. What I understand makes sense and what I understand allows me to, slowly, learn more.
UNIX, Linux has always had for me a joy in learning and a WOW! factor and a 'hey, that's neat!" aspect. I've never had that with Windows (which was always A source of frustration) or even VMS (and yes even when I used the RATFOR Software Tools).
I'm not fighting it. I simply do not embrace new things blindly. Just exposing the problems is not being against it. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYWBkACgkQja8UbcUWM1zgzgD/VFak5qG1uLHT5vK2SsswSxAs Gc2B3JJFyMRk30vEjukA/jJKRB/IfqlRRaK6Jg8el+ZSr+usWtoP40v5t5hj8jCl =z+AS -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 04:58 PM, Carlos E. R. wrote:
Ah, that's an index. New to me. Thanks. What are the possible indexes, how does it work?
Well, when I RTFM and experimented, I learnt about tab expansion journalctl _COMM=<tab><tab> In fact I got the find out about "_COMM" by using journalctl <tab><tab> There's a pattern here, one that seem familiar from BASH. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 00:03, Anton Aylward wrote:
On 12/19/2016 04:58 PM, Carlos E. R. wrote:
Ah, that's an index. New to me. Thanks. What are the possible indexes, how does it work?
Well, when I RTFM and experimented, I learnt about tab expansion
journalctl _COMM=<tab><tab>
Does not work here. I get: minas-tirith:~ # journalctl _COMM= Display all 186 possibilities? (y or n) .ICEauthority .xauthTWEnQl .Xauthority .xauthTfEeG2 .Xresources~ .xauthVDMJu3 .bash_history .xauthWJGMVo and many more. It is the current directory listing. I understand it is a new feature that my 13.1 does not have. Ok, I have a 42.2 machine available. Let's try: cer@Isengard:~> journalctl _COMM= accounts-daemon cleanup gnomesu-pam-bac logger pickup sddm start-ntpd systemd-journal umount alsactl config_postfix groupadd login pkexec sddm-greeter su systemd-localed unix2_chkpwd audispd console-kit-dae haveged lxdm polkitd sddm-helper swapoff systemd-logind useradd auditctl cron helloworld master postlog (sd-pam) systemctl systemd-shutdow usermod auditd dbus-daemon hp-upgrade mcelog pulseaudio smartd systemd systemd-sleep wicked avahi-daemon display-manager httpd-prefork ModemManager purge-kernels sm-notify (systemd) systemd-sysctl wickedd bluetoothd dnsmasq iscsiadm mount python smtp systemd-coredum systemd-sysuser wickedd-dhcp4 boot.apparmor dracut-cmdline killproc mtp-probe qmgr smtpd systemd-cryptse systemd-udevd wickedd-dhcp6 bounce echo lightdm nscd rpc.statd sshd systemd-fsck trivial-rewrite xfpm-power-back checkproc gnome-keyring-d local ntpd rtkit-daemon sshd-gen-keys-s systemd-hiberna udisksd cer@Isengard:~> journalctl _COMM= That's not correct. I see there "helloworld" which I recognize as a command I wrote, a script: /usr/local/bin/helloworld - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYbP0ACgkQja8UbcUWM1wRRgD/VFF/36wQEMj/S1a+MIS7WQWy +c4Cn1eaML1Biob1BlsA+wWMHbBB0c1Q3ddLMkD+FvUG7nYHFi8iyIdEigQeXRdD =esw7 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Dec 19, 2016 at 6:27 PM, Carlos E. R.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2016-12-20 00:03, Anton Aylward wrote:
On 12/19/2016 04:58 PM, Carlos E. R. wrote:
Ah, that's an index. New to me. Thanks. What are the possible indexes, how does it work?
Well, when I RTFM and experimented, I learnt about tab expansion
journalctl _COMM=<tab><tab>
Does not work here. I get:
minas-tirith:~ # journalctl _COMM= Display all 186 possibilities? (y or n) .ICEauthority .xauthTWEnQl .Xauthority .xauthTfEeG2 .Xresources~ .xauthVDMJu3 .bash_history .xauthWJGMVo
and many more. It is the current directory listing.
I understand it is a new feature that my 13.1 does not have.
Ok, I have a 42.2 machine available. Let's try:
cer@Isengard:~> journalctl _COMM= <snip> That's not correct. I see there "helloworld" which I recognize as a command I wrote, a script:
Same here. Is there something we need to install first? (Leap 42.2) Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 00:33, Greg Freemyer wrote:
On Mon, Dec 19, 2016 at 6:27 PM, Carlos E. R. <> wrote:
That's not correct. I see there "helloworld" which I recognize as a command I wrote, a script:
Same here. Is there something we need to install first? (Leap 42.2)
We must be missing some tab completion package. cer@Isengard:~> ls /etc/bash_completion.d/ aria2c dbus-bash-completion.sh dracut grub lsinitrd scout.sh subversion yast2-completion.sh zypper.sh cer@Isengard:~> - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYcdEACgkQja8UbcUWM1z55QD/Q4AvvNStxykVWPtHVEPhytVK xw4JcynuJWs1zJ3PiVwA/jHViC+jY2RaMH3njifN18WBnkdH1jo1gRjssQRbMo0U =/qvj -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 06:48 PM, Carlos E. R. wrote:
On 2016-12-20 00:33, Greg Freemyer wrote:
On Mon, Dec 19, 2016 at 6:27 PM, Carlos E. R. <> wrote:
That's not correct. I see there "helloworld" which I recognize as a command I wrote, a script:
Same here. Is there something we need to install first? (Leap 42.2)
We must be missing some tab completion package.
cer@Isengard:~> ls /etc/bash_completion.d/ aria2c dbus-bash-completion.sh dracut grub lsinitrd scout.sh subversion yast2-completion.sh zypper.sh
Hmm, I don't know about that. As I understood it, the BASH completion is to list the files. I was under the impression systemctl had its own completion mechanism. I'm running systemd-210.1477297097.83231f8-25.51.1.x86_64 What do you have in -- Reboot -- Dec 19 09:48:05 Mainbox echo[1753]: Starting mail service (Postfix) Is there a file 'systemctl' there? I do and its from systemd-bash-completion-210.1477297097.83231f8-25.51.1.noarch -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 01:15, Anton Aylward wrote:
On 12/19/2016 06:48 PM, Carlos E. R. wrote:
On 2016-12-20 00:33, Greg Freemyer wrote:
On Mon, Dec 19, 2016 at 6:27 PM, Carlos E. R. <> wrote:
That's not correct. I see there "helloworld" which I recognize as a command I wrote, a script:
Same here. Is there something we need to install first? (Leap 42.2)
We must be missing some tab completion package.
cer@Isengard:~> ls /etc/bash_completion.d/ aria2c dbus-bash-completion.sh dracut grub lsinitrd scout.sh subversion yast2-completion.sh zypper.sh
Hmm, I don't know about that. As I understood it, the BASH completion is to list the files.
No, it is more complex than that.
I was under the impression systemctl had its own completion mechanism.
systemctl has to tell bash what bash completion to use, by installing a file into that directory. It is bash which is reading the keyboard.
I'm running systemd-210.1477297097.83231f8-25.51.1.x86_64
What do you have in -- Reboot -- Dec 19 09:48:05 Mainbox echo[1753]: Starting mail service (Postfix)
Is there a file 'systemctl' there?
In where? :-? Where comes that "reboot" into?
I do and its from systemd-bash-completion-210.1477297097.83231f8-25.51.1.noarch
I have: Isengard:~ # rpm -qa | grep systemd-bash-completion systemd-bash-completion-228-13.1.noarch Isengard:~ # Isengard:~ # rpm -ql systemd-bash-completion /usr/share/bash-completion /usr/share/bash-completion/completions /usr/share/bash-completion/completions/bootctl /usr/share/bash-completion/completions/busctl /usr/share/bash-completion/completions/coredumpctl /usr/share/bash-completion/completions/hostnamectl /usr/share/bash-completion/completions/journalctl <====== /usr/share/bash-completion/completions/kernel-install /usr/share/bash-completion/completions/localectl /usr/share/bash-completion/completions/loginctl /usr/share/bash-completion/completions/machinectl /usr/share/bash-completion/completions/systemctl /usr/share/bash-completion/completions/systemd-analyze /usr/share/bash-completion/completions/systemd-cat /usr/share/bash-completion/completions/systemd-cgls /usr/share/bash-completion/completions/systemd-cgtop /usr/share/bash-completion/completions/systemd-delta /usr/share/bash-completion/completions/systemd-detect-virt /usr/share/bash-completion/completions/systemd-nspawn /usr/share/bash-completion/completions/systemd-run /usr/share/bash-completion/completions/timedatectl /usr/share/bash-completion/completions/udevadm Isengard:~ # Still, something is wrong, it does not work in our 42.2 machines. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYe+UACgkQja8UbcUWM1wg1wD+K2bKUi17uXsLkmcmG4rA98/b zZveHI+ws/zpGih0MVEA/0RmPZ7dkIQRIYp0AwU3oKVD4aWnMPwqfpg6/qGTg8/m =1cqa -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
Hmm, I don't know about that. As I understood it, the BASH completion is to list the files. I was under the impression systemctl had its own completion mechanism.
--- Guess you need to work a bit more for your "oh wow" moment. All the autocompletion from the command line is via bash. How it is programmed differentiates its behavior (i.e. file completion, or other). For example, I wrote a modprobe completion that shows all modules that have yet to be loaded. If you load a module, then a new completion won't show that module. It's all in the bash manpage.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 06:27 PM, Carlos E. R. wrote:
That's not correct. I see there "helloworld" which I recognize as a command I wrote, a script:
/usr/local/bin/helloworld
And so? What's the output if you do a journalctl _COMM=helloworld Just to make the point, when I <tab><tab> I get all the things I've set up and run like docecot, postfix, and even "echo". YES! even "echo". -- Reboot -- Dec 19 09:48:05 Mainbox echo[1753]: Starting mail service (Postfix) -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 00:57, Anton Aylward wrote:
On 12/19/2016 06:27 PM, Carlos E. R. wrote:
That's not correct. I see there "helloworld" which I recognize as a command I wrote, a script:
/usr/local/bin/helloworld
And so? What's the output if you do a
journalctl _COMM=helloworld
Ah, I see. I'm starting to understand. - -- Reboot -- Dec 05 13:05:47 Isengard helloworld[2031]: ----- Dec 05 13:05:47 Isengard helloworld[2031]: UNIT LOAD ACTIVE SUB Dec 05 13:05:47 Isengard helloworld[2031]: ● apache2.service loaded failed failed The Apache Webserver Dec 05 13:05:47 Isengard helloworld[2031]: LOAD = Reflects whether the unit definition was properly loaded. Dec 05 13:05:47 Isengard helloworld[2031]: ACTIVE = The high-level unit activation state, i.e. generalization of SUB. Dec 05 13:05:47 Isengard helloworld[2031]: SUB = The low-level unit activation state, values depend on unit type. Dec 05 13:05:47 Isengard helloworld[2031]: 1 loaded units listed. Pass --all to see loaded but inactive units, too. Dec 05 13:05:47 Isengard helloworld[2031]: To show all installed unit files use 'systemctl list-unit-files'. - -- Reboot -- Dec 05 21:04:22 Isengard helloworld[2145]: ----- Dec 05 21:04:22 Isengard helloworld[2145]: 0 loaded units listed. Pass --all to see loaded but inactive units, too. Dec 05 21:04:22 Isengard helloworld[2145]: To show all installed unit files use 'systemctl list-unit-files'. - -- Reboot --
Just to make the point, when I <tab><tab> I get all the things I've set up and run like docecot, postfix, and even "echo". YES! even "echo".
Mmm... - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYfUgACgkQja8UbcUWM1zDNAD/ZYQd8s4GkvVEkGxj4J0yiUSZ l8jcS7O5ZNFUNIrZCUkA/18jZ7eD3ZID8uV4O0zBBxiiG1psycSHolW3f9FTEtkj =goTc -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 04:58 PM, Carlos E. R. wrote:
... now, if you tell me how I can import the journal into Libre Office...
OK, assuming you are applying the various filters to extract the items you want... I'm assuming you don't mean input as text cut and past into a "Word" document much as you might cut and paste text into a mail message, as we do quite regularly, that you mean paste it into the spreadhseet too. For that I suggest you use the JSON export option. Again, RTFM or google, it very well documented and described. https://www.freedesktop.org/wiki/Software/systemd/json/ Of course if you see no other way, you can convert JSON to CSV and import the CSV to OO http://www.convertcsv.com/json-to-csv.htm Or, if you prefer, there's a macro to convert JSON that's been pasted into one cell. https://extensions.libreoffice.org/extensions/libreoffice-getrest-plugin-1 It depends, entirely, on your specific needs. Some of us are familiar enough with things like Perl, for which JSON is a native format, that we'd have no problem manipulating it. IIR there's an option in OO/LO to use python as the scripting language, and a JSON importer in python is a well documented tool. Most certainly, JSON is a more robust, more portable, more application-friendly (particularly for transmission over the 'Net) format than CSV. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 00:51, Anton Aylward wrote:
On 12/19/2016 04:58 PM, Carlos E. R. wrote:
... now, if you tell me how I can import the journal into Libre Office...
OK, assuming you are applying the various filters to extract the items you want...
I'm assuming you don't mean input as text cut and past into a "Word" document much as you might cut and paste text into a mail message, as we do quite regularly, that you mean paste it into the spreadhseet too.
No. I mean import a database file directly into a calc sheet with all the fields properly done. I assume and understand that the journal is a database, so I want to read it directly into a calc sheet, without me doing an iota of work. Click, point, and open. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYfgMACgkQja8UbcUWM1wtKgEAhNe7js6PkNQkIGs287/JtSIU fFiT1Ss8Prvly3koM90A+wWc2k82H5jgn3FDql57G503Imfvb/BHidQN8xUmv0i+ =6RlZ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
Rotating rust of syslog files isn't going to be any faster.
It was already pre-sorted w/syslog. If you used ng-syslog, you had all the filtering of grep available on each log message, so zero waiting for filtered syslog output.
You're clutching at straws here, Carlos.
Hardly.
That's fine. Go off and use Windows then, accept someone elses' idea of how the UI should look, what data you're allowed to display and how to see it and what you can do with it and more to the point what it won't permit you to do with it.
You really shouldn't be using Linux if that's your attitude.
---- If you want text logs by default, you shouldn't be using linux? What should you be using?
There are forms of anti-systemd manifestation. A reluctance to learn how to manage the new facilities is one of them.
--- When you only see in black and white, everything and everyone that is not with us is against us, right? Shows how religious/biblical this topic is to some.
This is all new to e as well, but I'm willing to accept it, not fight it, to experiment and learn. I figure that Lennart Poettering is smarter than me. What I understand makes sense and what I understand allows me to, slowly, learn more.
==== You want something new? How about logs in rot13? That's the next step to prevent casual browsing of system data entries (as MS uses in the registry).
UNIX, Linux has always had for me a joy in learning and a WOW! factor and a 'hey, that's neat!" aspect. I've never had that with Windows (which was always A source of frustration)
Did you ever read "Windows Internals" (by Russinovich)? Eventually divided into 2 volumes with other contributors... The earlier versions were better. Shows you how to dig in as deep as you want. You just had to work harder for your "oh wow".
or even VMS (and yes even when I used the RATFOR Software Tools).
---- Tools? Was only used to the compiler to generate a C like language (Rational Fortran)... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 03:54, L A Walsh wrote:
Anton Aylward wrote:
Rotating rust of syslog files isn't going to be any faster.
It was already pre-sorted w/syslog. If you used ng-syslog, you had all the filtering of grep available on each log message, so zero waiting for filtered syslog output.
Apparently converting to text the whole journal needs seeking in the database, and if it is large (I have had a journal fo a gigabyte or two) the disk head movements are considerable. People using an SSD did the job in seconds, whereas people with rotating disk measured it in minutes. I'm not joking, I measured this and posted results in the mail list months ago. I don't know how to exactly seek for them; perhaps grep for 'journalctl' and 'time'. It is possible that producing output on "_COMM" instead, would be faster. I did not know about that command. Then we could repeat the tests I did months ago - but in this machine, I no longer have a journal. I would have to enable it again, and run it for some days to produce content. Why did I disable the journal on disk (temporary and long term)? Because it takes gigabytes... I have an nntp local server (proxy), and it produces tons of log messages per day. With syslog I can dump them all after, say, a week. With journal, I can't. If I want to keep a reasonable long term journal of all important events in my machine, it also keeps the useless debug logs of everything else... I can not purge the database of selected content. With syslog, I can keep logs dating back for a year or two and it is no issue. I simply dump the verbose debug logs of some facilities like nntp. And yes, I'm aware of the advantages of a log database. I worked with them, professionally. We had a system that converted to a binary database the text logs of several machines, so that we could run analysis (real time or later time) of the data. A hugely expensive system.
This is all new to e as well, but I'm willing to accept it, not fight it, to experiment and learn. I figure that Lennart Poettering is smarter than me. What I understand makes sense and what I understand allows me to, slowly, learn more.
==== You want something new? How about logs in rot13? That's the next step to prevent casual browsing of system data entries (as MS uses in the registry).
You got to be kidding. Really? MS uses rot13 in the registry? Wow.
UNIX, Linux has always had for me a joy in learning and a WOW! factor and a 'hey, that's neat!" aspect. I've never had that with Windows (which was always A source of frustration)
Did you ever read "Windows Internals" (by Russinovich)? Eventually divided into 2 volumes with other contributors... The earlier versions were better.
Shows you how to dig in as deep as you want. You just had to work harder for your "oh wow".
or even VMS (and yes even when I used the RATFOR Software Tools).
I read at the time the official MsDOS programmer book (expensive), and it was fascinating. I also read a non official guide to Windows 3 internals, and it also was fascinating. Finally, I read all the Borland C and Turbo Pascal for Windows manuals, and I found them very entertaining books. There was documentation, good reading material, written to be read by people, and it was good and very interesting. And expensive. It was possible to understand how the system worked and why it did things as they did. Of course, there were hidden, non disclosed functionalities... which some people found out and published. But that was another time and age. I no longer do programming for Windows. Nor for Linux... just a little bit. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
20.12.2016 06:43, Carlos E. R. пишет: ...
You want something new? How about logs in rot13? That's the next step to prevent casual browsing of system data entries (as MS uses in the registry).
You got to be kidding. Really? MS uses rot13 in the registry? Wow.
Apparently one single application storing data in registry does use it. I have not seen any evidence there is general rule or format requirement. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tuesday, 20 December 2016 04:43:03 GMT Carlos E. R. wrote:
On 2016-12-20 03:54, L A Walsh wrote:
Anton Aylward wrote:
Rotating rust of syslog files isn't going to be any faster.
----
It was already pre-sorted w/syslog. If you used ng-syslog, you had
all the filtering of grep available on each log message, so zero waiting for filtered syslog output.
Apparently converting to text the whole journal needs seeking in the database, and if it is large (I have had a journal fo a gigabyte or two) the disk head movements are considerable. People using an SSD did the job in seconds, whereas people with rotating disk measured it in minutes. I'm not joking, I measured this and posted results in the mail list months ago. I don't know how to exactly seek for them; perhaps grep for 'journalctl' and 'time'.
It is possible that producing output on "_COMM" instead, would be faster. I did not know about that command. Then we could repeat the tests I did months ago - but in this machine, I no longer have a journal. I would have to enable it again, and run it for some days to produce content.
Why did I disable the journal on disk (temporary and long term)? Because it takes gigabytes...
I have an nntp local server (proxy), and it produces tons of log messages per day. With syslog I can dump them all after, say, a week. With journal, I can't. If I want to keep a reasonable long term journal of all important events in my machine, it also keeps the useless debug logs of everything else... I can not purge the database of selected content.
Are you sure about that? There are loads of configuration options in journald.conf regarding storage and storage sizes and retention times
With syslog, I can keep logs dating back for a year or two and it is no issue. I simply dump the verbose debug logs of some facilities like nntp. Syslog/rsyslog can now read all log records directly from the journal (might depend on the version you use) and then you could continue as you normally do, just set storage to "volatile" or "none" and forward records to syslog socket (according to journald.conf) I've not tried any of these as opensuse defaults suit me fine but there are loads of config options.
And yes, I'm aware of the advantages of a log database. I worked with them, professionally. We had a system that converted to a binary database the text logs of several machines, so that we could run analysis (real time or later time) of the data. A hugely expensive system.
This is all new to e as well, but I'm willing to accept it, not fight it, to experiment and learn. I figure that Lennart Poettering is smarter than me. What I understand makes sense and what I understand allows me to, slowly, learn more.
====
You want something new? How about logs in rot13? That's the next step
to prevent casual browsing of system data entries (as MS uses in the registry).
You got to be kidding. Really? MS uses rot13 in the registry? Wow.
UNIX, Linux has always had for me a joy in learning and a WOW! factor and a 'hey, that's neat!" aspect. I've never had that with Windows (which was always A source of frustration)
----
Did you ever read "Windows Internals" (by Russinovich)? Eventually divided
into 2 volumes with other contributors... The earlier versions were better.
Shows you how to dig in as deep as you want. You just had to work harder
for your "oh wow".
or even VMS (and yes even when I used the RATFOR Software Tools).
I read at the time the official MsDOS programmer book (expensive), and it was fascinating. I also read a non official guide to Windows 3 internals, and it also was fascinating. Finally, I read all the Borland C and Turbo Pascal for Windows manuals, and I found them very entertaining books.
There was documentation, good reading material, written to be read by people, and it was good and very interesting. And expensive. It was possible to understand how the system worked and why it did things as they did.
Of course, there were hidden, non disclosed functionalities... which some people found out and published.
But that was another time and age. I no longer do programming for Windows. Nor for Linux... just a little bit.
-- opensuse:tumbleweed:20161217 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Dec 20, 2016 at 12:29 PM, ianseeks
Why did I disable the journal on disk (temporary and long term)? Because it takes gigabytes...
I have an nntp local server (proxy), and it produces tons of log messages per day. With syslog I can dump them all after, say, a week. With journal, I can't. If I want to keep a reasonable long term journal of all important events in my machine, it also keeps the useless debug logs of everything else... I can not purge the database of selected content.
Are you sure about that?
Yes.
There are loads of configuration options in journald.conf regarding storage and storage sizes and retention times
If you take time to actually read about these configuration options you will see that they refer to total size or retention time of *all* journal entries. So you can trivially cause mild DoS by simply filling logs with garbage, thus causing important entries to be displaced out of journal. Syslog allows filtering of logs into different places for long term storage and you can configure individual retention rules for each (although that is strictly speaking unrelated to syslog). Systemd author is actively opposed to any idea to filter log entries in journald. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tuesday, 20 December 2016 14:24:08 GMT Andrei Borzenkov wrote:
On Tue, Dec 20, 2016 at 12:29 PM, ianseeks
wrote: ... Why did I disable the journal on disk (temporary and long term)? Because it takes gigabytes...
I have an nntp local server (proxy), and it produces tons of log messages per day. With syslog I can dump them all after, say, a week. With journal, I can't. If I want to keep a reasonable long term journal of all important events in my machine, it also keeps the useless debug logs of everything else... I can not purge the database of selected content.> Are you sure about that?
Yes.
I placed my comment in the wrong place, i wasn't referring to filtering content, wouldn't want to do that, it was more a reference to retention and storage. If i wanted to filter content for a specific purpose, i'd journalctl it via grep it into a place of my choosing (or configure it all to go to syslog if that was my preferred logger and have no persistent storage).
There are loads of configuration options in journald.conf regarding storage and storage sizes and retention times
If you take time to actually read about these configuration options you will see that they refer to total size or retention time of *all* journal entries. So you can trivially cause mild DoS by simply filling logs with garbage, thus causing important entries to be displaced out of journal. I don't understand how it can displace important entries from the journal. As far as i know, it continues into a new file once it reaches its limit.
You can journalctl --vacuum-time=xmonths/days any redundant journal files away once you have dealt with them.
Syslog allows filtering of logs into different places for long term storage and you can configure individual retention rules for each (although that is strictly speaking unrelated to syslog).
Systemd author is actively opposed to any idea to filter log entries in journald.
I admit my knowledge on the journal is not great and this is why i ask questions to hopefully get the answers -- opensuse:tumbleweed:20161217 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 06:58 AM, ianseeks wrote:
I admit my knowledge on the journal is not great and this is why i ask questions to hopefully get the answers
Despite everything, neither is mine. Its a continual learning process for me and threads like this encourage me to probe and learn more. I know more, much more that when this thread started, about journalctl, and realise some of what I said earlier in the thread ... well if not actually wrong is certainly in need of revision! An important part of learning the new is being able to let go of the old. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tuesday, 20 December 2016 07:43:02 GMT Anton Aylward wrote:
On 12/20/2016 06:58 AM, ianseeks wrote:
I admit my knowledge on the journal is not great and this is why i ask questions to hopefully get the answers
Despite everything, neither is mine. Its a continual learning process for me and threads like this encourage me to probe and learn more. I know more, much more that when this thread started, about journalctl, and realise some of what I said earlier in the thread ... well if not actually wrong is certainly in need of revision!
An important part of learning the new is being able to let go of the old.
Indeed, i quite agree
> Q: Are you sure? > >> A: Because it reverses the logical flow of conversation. >> >>> Q: Why is top posting frowned upon?
-- opensuse:tumbleweed:20161217 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 04:43 AM, Anton Aylward wrote:
An important part of learning the new is being able to let go of the old.
Exactly. Being as systemd and journald is several years old now, and the systemd wars are all fought and decided, I have to wonder why this thread is raging on at at such length at this time. There are settings to pass through all the logging to old school logs. Use those or don't, its up to you. Either use it or configure your system so you don't need to. Lets move on, for pete sake people let it go. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 22:56, John Andersen wrote:
On 12/20/2016 04:43 AM, Anton Aylward wrote:
An important part of learning the new is being able to let go of the old.
Exactly.
Being as systemd and journald is several years old now, and the systemd wars are all fought and decided, I have to wonder why this thread is raging on at at such length at this time.
There are settings to pass through all the logging to old school logs. Use those or don't, its up to you. Either use it or configure your system so you don't need to.
we want settings that do not exist. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhZqYMACgkQja8UbcUWM1zvjQEAk9AtNcfKL2t2ipd4WeYqLIu0 /Tn983gIF3XSjvvAtFQBAJwBuvq0iSxi49+3lOIoP4mHkpKmUdIYsBdq4HabZ5UO =aNbq -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 04:58 PM, Carlos E. R. wrote:
we want settings that do not exist.
I wonder some times, googling around and seeing what's being done on Redhat and Arch for example, if the Suse policy of not adopting the latest upstream revisions or altering them in various way is to our benefit. We seem well behind the curve as far as systemd goes. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 12:58, ianseeks wrote:
On Tuesday, 20 December 2016 14:24:08 GMT Andrei Borzenkov wrote:
I don't understand how it can displace important entries from the journal. As far as i know, it continues into a new file once it reaches its limit.
You can journalctl --vacuum-time=xmonths/days any redundant journal files away once you have dealt with them.
Syslog allows filtering of logs into different places for long term storage and you can configure individual retention rules for each (although that is strictly speaking unrelated to syslog).
Systemd author is actively opposed to any idea to filter log entries in journald.
I admit my knowledge on the journal is not great and this is why i ask questions to hopefully get the answers
Yes, you don't understand. Say that I have to keep logs of events for a year or two. This is not rare (you can be mandated for legal reasons). Yes, I can configure journal to do this, no problem so far. Now, journal keeps *all* entries of *everything*. That is the problem. Suppose I have a service that talks a lot, debug entries. Say nntp, for instance. I'm only interested in a week of those, for instance. But journal does not allow me to keep a week and delete the rest, while keeping mail log for two years. This nntp service may write to the log, for instance, ten megabytes of entries per day. On a year, that is 3650 megabytes (I have seen my journal have two gigabytes after an uptime of a month). If I limit the size of the log to a reasonable size, say a hundred megabytes, the log will be full of nntp entries and little else. It will be difficult to investigate a crash or a reboot that happened two weeks ago because due to the amount of entries from nntp, it keeps less than ten days. The solution is to *erase* all debug entries from nntp older than a week, say. But journal does not allow this, and the developer refuses to do it. So no, journal is not useful to me, I have to keep syslog. Yes, I know how to do it and I do. On some machines I keep both, on some only syslog. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tuesday, 20 December 2016 15:17:43 GMT Carlos E. R. wrote:
On 2016-12-20 12:58, ianseeks wrote:
On Tuesday, 20 December 2016 14:24:08 GMT Andrei Borzenkov wrote:
I don't understand how it can displace important entries from the journal. As far as i know, it continues into a new file once it reaches its limit.
You can journalctl --vacuum-time=xmonths/days any redundant journal files away once you have dealt with them.
Syslog allows filtering of logs into different places for long term storage and you can configure individual retention rules for each (although that is strictly speaking unrelated to syslog).
Systemd author is actively opposed to any idea to filter log entries in journald.
I admit my knowledge on the journal is not great and this is why i ask questions to hopefully get the answers
Yes, you don't understand.
Say that I have to keep logs of events for a year or two. This is not rare (you can be mandated for legal reasons). Yes, I can configure journal to do this, no problem so far. Now, journal keeps *all* entries of *everything*. That is the problem.
Suppose I have a service that talks a lot, debug entries. Say nntp, for instance. I'm only interested in a week of those, for instance. But journal does not allow me to keep a week and delete the rest, while keeping mail log for two years.
This nntp service may write to the log, for instance, ten megabytes of entries per day. On a year, that is 3650 megabytes (I have seen my journal have two gigabytes after an uptime of a month).
If I limit the size of the log to a reasonable size, say a hundred megabytes, the log will be full of nntp entries and little else. It will be difficult to investigate a crash or a reboot that happened two weeks ago because due to the amount of entries from nntp, it keeps less than ten days.
The solution is to *erase* all debug entries from nntp older than a week, say. But journal does not allow this, and the developer refuses to do it.
So no, journal is not useful to me, I have to keep syslog. Yes, I know how to do it and I do. On some machines I keep both, on some only syslog.
Ok, thanks for the explanation -- opensuse:tumbleweed:20161217 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 09:17 AM, Carlos E. R. wrote:
Say that I have to keep logs of events for a year or two. This is not rare (you can be mandated for legal reasons). Yes, I can configure journal to do this, no problem so far. Now, journal keeps *all* entries of *everything*. That is the problem.
Um, NOT. Those mandated to keep logs of events, banks and other regulated industries, and with the government attitudes that's going to include internet service providers too pretty soon, need to keep all "relevant" events. That "relevant" means the ability to cross reference and attribute the whys and wherefores, the who and the when. That amounts to *everything* in so far as every access, every reboot, every reconfiguration. Some industries, I've met this at banks, brokerages and with a regulated utility, are under more than one set of demands and regulations, so you can't be too comprehensive. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 12/20/2016 09:17 AM, Carlos E. R. wrote:
Say that I have to keep logs of events for a year or two. This is not rare (you can be mandated for legal reasons). Yes, I can configure journal to do this, no problem so far. Now, journal keeps *all* entries of *everything*. That is the problem.
Um, NOT.
Agrees, it's not up to the journal to filter stuff. It's easily filtered out afterwards. -- Per Jessen, Zürich (2.0°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 20/12/2016 à 15:53, Anton Aylward a écrit :
the when. That amounts to *everything* in so far as every access, every reboot, every reconfiguration.
if a log is too intensive, may be blame the original application, not the logger, may be ask the application to log elsewhere? ùmay be this is obsolete, but still a good stating point? https://www.eyrie.org/~eagle/software/inn/docs-2.4/install.html#S10 jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
jdd wrote:
Le 20/12/2016 à 15:53, Anton Aylward a écrit :
the when. That amounts to *everything* in so far as every access, every reboot, every reconfiguration.
if a log is too intensive, may be blame the original application, not the logger, may be ask the application to log elsewhere?
Yep.
ùmay be this is obsolete, but still a good stating point? https://www.eyrie.org/~eagle/software/inn/docs-2.4/install.html#S10
The openSUSE default for innd is to log to separate files under /var/log/news/ - Carlos is probably running leafnode or something like that. -- Per Jessen, Zürich (1.9°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 16:25, Per Jessen wrote:
The openSUSE default for innd is to log to separate files under /var/log/news/ - Carlos is probably running leafnode or something like that.
Correct, I use leafnode. It is a proxy server. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 20/12/2016 à 17:37, Carlos E. R. a écrit :
On 2016-12-20 16:25, Per Jessen wrote:
The openSUSE default for innd is to log to separate files under /var/log/news/ - Carlos is probably running leafnode or something like that.
Correct, I use leafnode. It is a proxy server.
then http://leafnode.sourceforge.net/wuerzburg/faq.html this thread: https://forums.opensuse.org/showthread.php/506886-systemd-journald-Settings-... discuss your problem. what I see id that syslog uses a special file for leafnode. You still can use that and don't use the standard log system for debugging purpose jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
20.12.2016 19:55, jdd пишет:
what I see id that syslog uses a special file for leafnode.
It CAN BE CONFIGURED BY ADMINISTRATOR to use special file for leafnode. That is exactly what is not possible with journal.
You still can use that and don't use the standard log system for debugging purpose
Which rather defeats the very idea of having standard log system in the first place. Not to mention that this presumes "debugging" is something performed by developers only. In reality I have to deal with those logs every time I investigate problem on customer systems. They are extremely helpful in finished programs, the more logs - the better. But you also have no reasons to keep them for years. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 20/12/2016 à 18:24, Andrei Borzenkov a écrit :
20.12.2016 19:55, jdd пишет:
what I see id that syslog uses a special file for leafnode.
It CAN BE CONFIGURED BY ADMINISTRATOR to use special file for leafnode. That is exactly what is not possible with journal.
really? I don't use leafnode, but don't see why the official instructions couldn't work, even is that mean use an other system than journalctl logs
You still can use that and don't use the standard log system for debugging purpose
Which rather defeats the very idea of having standard log system in the first place.
why? the old way still ask to use a special file Not to mention that this presumes "debugging" is something
performed by developers only. In reality I have to deal with those logs every time I investigate problem on customer systems. They are extremely helpful in finished programs, the more logs - the better. But you also have no reasons to keep them for years.
and? they have to be used by admins. Anyway, It's unuseful to discuss this very problem only. debug mode is pretty common and everybody know it can generate a lot of logs. So the question on subject should be really: "how to manage debug mode" and not challenge what is one of the main journalctl feature (binary). I don't see neither why a debug mode should be kept for month if the logs are not analyzed in details. We should really focus on trying to solve real problems, not imaginary ones nor try to promote not viable solutions thanks jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 18:42, jdd wrote:
Le 20/12/2016 à 18:24, Andrei Borzenkov a écrit :
20.12.2016 19:55, jdd пишет:
what I see id that syslog uses a special file for leafnode.
It CAN BE CONFIGURED BY ADMINISTRATOR to use special file for leafnode. That is exactly what is not possible with journal.
really? I don't use leafnode, but don't see why the official instructions couldn't work, even is that mean use an other system than journalctl logs
Of course I can configure leafnode to produce less logs, that's not the problem! I want to have the debug logs for a few days in order to investigate a problem I may have, But I'm not interested in keeping gigabytes of nntp debug messages for years! There is no problem at all with syslog. The problem is with systemd journal. It doesn't cope with this situation.
You still can use that and don't use the standard log system for debugging purpose
Which rather defeats the very idea of having standard log system in the first place.
why? the old way still ask to use a special file
No, it doesn't. You can choose a special file, or not. you have the choice.
debug mode is pretty common and everybody know it can generate a lot of logs. So the question on subject should be really: "how to manage debug mode" and not challenge what is one of the main journalctl feature (binary).
I don't see neither why a debug mode should be kept for month if the logs are not analyzed in details.
We should really focus on trying to solve real problems, not imaginary ones nor try to promote not viable solutions
No, it is a very real problem. I can not use journal logs as long term logs precisely for this issue, which the developer is aware of and refuses to handle. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 16:20, jdd wrote:
Le 20/12/2016 à 15:53, Anton Aylward a écrit :
the when. That amounts to *everything* in so far as every access, every reboot, every reconfiguration.
if a log is too intensive, may be blame the original application, not the logger, may be ask the application to log elsewhere?
No, because with syslog and variants you can keep debug logs for a different period of time than main logs. The applications were designed with that in view, spewing log entries at different "priorities". Thus, for example, you can keep news.info and news.debug files, and rotate them differently.
ùmay be this is obsolete, but still a good stating point?
https://www.eyrie.org/~eagle/software/inn/docs-2.4/install.html#S10
I don't use inn :-) -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 17:59, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
Traditional syslog has a method to cope with that flood of messages, journal doesn't. Which is why I do not use journal. It does not work. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 01:14 PM, Carlos E. R. wrote:
On 2016-12-20 17:59, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
Traditional syslog has a method to cope with that flood of messages, journal doesn't. Which is why I do not use journal. It does not work.
You have that exactly backwards. Syslog needs those rate-limiting tools to deal with the flood because it can't keep up. This has been a long standing traditional problems with syslog and one reason why its UDP based, so that packets from remote hosts that flood can be dropped with impunity. We've had to get faster and dedicated syslog servers to be able to keep up in large commercial settings. Journald was, however, designed to be able to cope with the floods. That's why it has not such mechanism. Go read the design documents. Its there. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 22:38, Anton Aylward wrote:
On 12/20/2016 01:14 PM, Carlos E. R. wrote:
On 2016-12-20 17:59, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
Traditional syslog has a method to cope with that flood of messages, journal doesn't. Which is why I do not use journal. It does not work.
You have that exactly backwards.
Syslog needs those rate-limiting tools to deal with the flood because it can't keep up. This has been a long standing traditional problems with syslog and one reason why its UDP based, so that packets from remote hosts that flood can be dropped with impunity. We've had to get faster and dedicated syslog servers to be able to keep up in large commercial settings.
Journald was, however, designed to be able to cope with the floods. That's why it has not such mechanism. Go read the design documents. Its there.
Go read my posts, it is not there. You have it wrong. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhZqbwACgkQja8UbcUWM1xAFAD/ZJswus6MzZVQGlD1fn/PQBqQ Yg5IPLaDyDnWpowMyg0A/RIaSA/YilziXYhirV7vajQKXVi+P/94XyEP95iwmy5B =krI+ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 12/20/2016 01:14 PM, Carlos E. R. wrote:
On 2016-12-20 17:59, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
Traditional syslog has a method to cope with that flood of messages, journal doesn't. Which is why I do not use journal. It does not work.
You have that exactly backwards.
Syslog needs those rate-limiting tools to deal with the flood because it can't keep up.
You misunderstand, it's not about rate-limiting. With syslog, you can direct different messages (all kinds of filters) to different destinations, and then use e.g. logrotate to administer different retention times. With the systemd journal, there is only one storage location and only one retention time.
Journald was, however, designed to be able to cope with the floods. That's why it has not such mechanism. Go read the design documents. Its there.
The problem Carlos describes is real, and just one of many reasons for continuing to use a syslog daemon. -- Per Jessen, Zürich (0.9°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/21/2016 03:13 AM, Per Jessen wrote:
The problem Carlos describes is real, and just one of many reasons for continuing to use a syslog daemon.
we have, as I've implied before, a bifurcation here. The problem Carlos has is a real problem for Carlos. As I've said, the comprehensive, no filter etc no limiting approach of jounald is what Big Business is requiring this, along with Docker rather than VM, Cgroups and more, are things that are of value to Big business, banks, Brokerages, Telcos, "Wall Street". They are necessary if Linux is to be part of the IT DNA there. Traditionally UNIX and Linux were counter-culture. FOSS was reactionary. It started as a hobby project and has been a source of joy and sideline activity for programmers and and people who want to drill down and experiment for some decades now. There's a bifurcation. What Carlos want journald to be isn't going to happen. EVER. No Way, simply because that would be contrary to the needs of Big Business in the ways I've outlined in past posts. Its a "that was then, this is now" situation, just like "I lived in a small town when I was a kid and no-one locked their doors, now I live in a big city and we do ..." sort of 'that was then, this is now'. Yes, there are people who attend RenFairs and dress up in medieval costume, but they omit that health care was lacking back then. There are people who maintain steam engines and biplanes. And of course KDE3. But the mainstream moves on. So we have a bifurcation: Linux for the Big Business and Linux for the small guys. It seems that Suse has chose the "Big Business" path. That there has been the integration we see in Leap should be taken as an indicator of that. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 12/21/2016 03:13 AM, Per Jessen wrote:
The problem Carlos describes is real, and just one of many reasons for continuing to use a syslog daemon.
we have, as I've implied before, a bifurcation here. The problem Carlos has is a real problem for Carlos.
As I've said, the comprehensive, no filter etc no limiting approach of jounald is what Big Business is requiring this, along with Docker rather than VM, Cgroups and more, are things that are of value to Big business, banks, Brokerages, Telcos, "Wall Street". They are necessary if Linux is to be part of the IT DNA there.
You're preaching to the choir. -- Per Jessen, Zürich (1.7°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 21/12/2016 à 16:58, Anton Aylward a écrit :
What Carlos want journald to be isn't going to happen. EVER. No Way, simply
who knows :-), after all we are on the opensource side :-) jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-21 17:06, jdd wrote:
Le 21/12/2016 à 16:58, Anton Aylward a écrit :
What Carlos want journald to be isn't going to happen. EVER. No Way, simply
Because the main systemd developer is very thick headed and he does not listen.
who knows :-), after all we are on the opensource side :-)
Yes, now someone will tell me to branch systemd. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhbKiYACgkQja8UbcUWM1zXhwD/Sl2QxsX5O1KptAb2ViaxJSzK D6lSasqVqowwRzg1z/IA/1IpspBCxU2MZjy2rn4LkIKW8Cy00BqMXv/7dBoJGDUj =gmNY -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/21/2016 08:19 PM, Carlos E. R. wrote:
On 2016-12-21 17:06, jdd wrote:
Le 21/12/2016 à 16:58, Anton Aylward a écrit :
What Carlos want journald to be isn't going to happen. EVER. No Way, simply Because the main systemd developer is very thick headed and he does not listen.
You are assigning to him an emotional childish churlishness that is not apparent in his reasoning for the design decisions he has made. I would say that he simply has a different agenda. A reasonable and reasoned one. Just different from yours. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 02:34, Anton Aylward wrote:
On 12/21/2016 08:19 PM, Carlos E. R. wrote:
On 2016-12-21 17:06, jdd wrote:
Le 21/12/2016 à 16:58, Anton Aylward a écrit :
What Carlos want journald to be isn't going to happen. EVER. No Way, simply Because the main systemd developer is very thick headed and he does not listen.
You are assigning to him an emotional childish churlishness that is not apparent in his reasoning for the design decisions he has made.
I would say that he simply has a different agenda. A reasonable and reasoned one. Just different from yours.
Not only mine. Obviously other people asked him to add the features and he refused. Something that many people have said about him. I do not have an agenda, I simply want to have a complete feature set, so that new programs have the same features as the old, and more, new, features. But not less. Not missing features. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhbLmkACgkQja8UbcUWM1xa6AEAhgbJFRcAHnzBVsJmdbh4qT/i roz9Kx8hINg+/qV2fckBAJ2YFQ5+Teq646NECJVC5XBfYtJvzZQI97Fjry3E4zo+ =wP65 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/21/2016 08:37 PM, Carlos E. R. wrote:
On 2016-12-22 02:34, Anton Aylward wrote:
On 12/21/2016 08:19 PM, Carlos E. R. wrote:
On 2016-12-21 17:06, jdd wrote:
Le 21/12/2016 à 16:58, Anton Aylward a écrit :
What Carlos want journald to be isn't going to happen. EVER. No Way, simply Because the main systemd developer is very thick headed and he does not listen.
You are assigning to him an emotional childish churlishness that is not apparent in his reasoning for the design decisions he has made.
I would say that he simply has a different agenda. A reasonable and reasoned one. Just different from yours.
Not only mine. Obviously other people asked him to add the features and he refused. Something that many people have said about him.
So? Without seeing that those features are and wht his reasons for refusing the request were that's just hearsay. Phrasing it that way is argumentative and prejudicial.
I do not have an agenda,
Oh yes you do, as you go on to make quite clear!
I simply want to have a complete feature set, so that new programs have the same features as the old, and more, new, features. But not less. Not missing features.
That isn't what progress is about. With that attitude the car is would have ropes and purs rather than a steering wheel and that it has a steering wheel and an accelerator pedal. so as to compatible with a horse. progress is about the new, a different way of doing things. It requires that you let go of the old way of doing things, that the means of evaluating worthiness changes. That the tools and the methods of support and deployment change. That has always been the way of progress. https://www.youtube.com/watch?v=yUQRbqc2qtY We've had this debate before. Rob Pike said that 'each thing should do one thing and only one thing' and that lead to be debate about line numbering in 'cat' http://harmful.cat-v.org/cat-v/ Here, we have a debate about the purity of log collection vs the use of external tools put together in a more 'traditional' UNIX manner as scripts using pipes and filters in scripting languages. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 03:16, Anton Aylward wrote:
On 12/21/2016 08:37 PM, Carlos E. R. wrote:
I simply want to have a complete feature set, so that new programs have the same features as the old, and more, new, features. But not less. Not missing features.
That isn't what progress is about.
This is not progress, it is retrocesion. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhbxk4ACgkQja8UbcUWM1xF2gD/c4aakfbj1RVau4h4zdcfrYIS FH/cPZy+BZUnPMpTa4UA+wYPHoqf5+tqqgcC+KoKvwrD9qHfNVfpYptajHcFxLFb =Q7jt -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/22/2016 07:25 AM, Carlos E. R. wrote:
This is not progress, it is retrocesion.
https://www.youtube.com/watch?v=xzm9urjQbWU -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 14:38, Anton Aylward wrote:
On 12/22/2016 07:25 AM, Carlos E. R. wrote:
This is not progress, it is retrocesion.
Whatever the point may be is totally lost on me. English nursery rymms means nothing to me. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdzkoACgkQja8UbcUWM1wuDwD8CQ5xecZbnaxJIaWXTfin8RGU aOZBRBSePhpRaMvveYAA/1OVDyyQauvmwg6N36b0WDv9QIc+VRtikR0Q7+EUoq7A =+6ov -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/23/2016 08:24 PM, Carlos E. R. wrote:
On 2016-12-22 14:38, Anton Aylward wrote:
On 12/22/2016 07:25 AM, Carlos E. R. wrote:
This is not progress, it is retrocesion.
Whatever the point may be is totally lost on me. English nursery rymms means nothing to me.
The point is that just like the guy in the nursery rhyme you are making a (multi stage) circular argument. its only justification is its own premise. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 22/12/2016 à 02:19, Carlos E. R. a écrit :
Yes, now someone will tell me to branch systemd.
- --
if you are the only one complaining, yes. If there are many people, there should be enough man power to do the job. open source is like this: if you want, you can. If you can't, you have to use what exists or pay... and as long as syslog fit your needs, you only have to pray to have it continue... For my use, I simply use what I have, having no other way jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 08:43, jdd wrote:
Le 22/12/2016 à 02:19, Carlos E. R. a écrit :
Yes, now someone will tell me to branch systemd.
- --
if you are the only one complaining, yes. If there are many people, there should be enough man power to do the job.
I know I'm not the only one, as the request for those changes has been made and rejected. Here, others do not talk because they do not want to be accused of things by the systemd zeleots. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhbxr4ACgkQja8UbcUWM1wVNgD/Yw05TaM/vqFdUnhB9rvhUwHF HYBYBRRyiBZjIybfOnsA/3tg7sKm++M0WiPCfB2tQg6FbWaHZULDs19ftVWWq1X8 =Ni17 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 11:59 AM, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
I think what JDD means, and I agree with him here, is that using syslog for debug is a bit of a kludge. Some programs need an out of band way of recording their debug messages and syslog is it :-( It became a convention even if that was not the real intent. The old STDIO had STDIN, STDOUT, STDERR as the three channels. Perhaps there needed to be a fourth, STDDEBUG :-) -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 12/20/2016 11:59 AM, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
I think what JDD means, and I agree with him here, is that using syslog for debug is a bit of a kludge.
Well, syslog() has had the LOG_DEBUG level for decades. It may not be exactly "debug" messages, but simply an extensive audit trail. -- Per Jessen, Zürich (1.0°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-21 09:15, Per Jessen wrote:
Anton Aylward wrote:
On 12/20/2016 11:59 AM, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
I think what JDD means, and I agree with him here, is that using syslog for debug is a bit of a kludge.
Well, syslog() has had the LOG_DEBUG level for decades. It may not be exactly "debug" messages, but simply an extensive audit trail.
Yes. It allows, for instance, tracking where an email went wrong, tracing how it went through the different daemons. The information is different from the audit the authorities want for email: from, to, subject, date. At least, maybe all the headers. Dunno. And debug messages do not flood /var/log/messages if you want. They would go to a different file, with earlier rotation. In case of email, it can be mail.debug, but the convention on openSUSE was /var/log/mail (there is mail.err and mail.info, maybe mail.warn) - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhaVfsACgkQja8UbcUWM1x8fwEAl5swJq8LQc1yiZ/A8KcYCAan 9eY5tE1NetAREwSxpMkA/2g2dZ5rt4pWkH1FTqbRmWLylxmwEnMZKyBevxXn0psa =rHRF -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/21/2016 05:14 AM, Carlos E. R. wrote:
On 2016-12-21 09:15, Per Jessen wrote:
Anton Aylward wrote:
On 12/20/2016 11:59 AM, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I think what JDD means, and I agree with him here, is that using syslog for debug is a bit of a kludge.
Well, syslog() has had the LOG_DEBUG level for decades. It may not be exactly "debug" messages, but simply an extensive audit trail.
Yes. It allows, for instance, tracking where an email went wrong, tracing how it went through the different daemons.
The information is different from the audit the authorities want for email: from, to, subject, date. At least, maybe all the headers. Dunno.
And debug messages do not flood /var/log/messages if you want. They would go to a different file, with earlier rotation. In case of email, it can be mail.debug, but the convention on openSUSE was /var/log/mail (there is mail.err and mail.info, maybe mail.warn)
You are making a very good case for there being a completely separate mechanism of DEBUG messaging from management messaging and audit trail messaging. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/21/2016 03:15 AM, Per Jessen wrote:
Anton Aylward wrote:
On 12/20/2016 11:59 AM, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
I think what JDD means, and I agree with him here, is that using syslog for debug is a bit of a kludge.
Well, syslog() has had the LOG_DEBUG level for decades. It may not be exactly "debug" messages, but simply an extensive audit trail.
Maybe I'm a dinosaur but I recall in UNIX 5 and UNIX 6 and UNIX 7 of the early and mid 1970s when there was no syslog. It didn't happen until the 1980s. Think about it: it uses UDP. That needs networking. Adopting syslog for an out of band debug mechanism was and is a kludge. As I said, a proper redirect-able STDDEBUG as part of STDIO would have been the right thing to do. So why didn't it happen? There's a simple reason for this. Back when, there was no non-blocking IO. Eventually, the only way to do non-blocking IO was to make use of the 'select()' call because there was no kernel support for non-blocking IO on a normal STDIO 'read()' as there is now. Today we could implement a non-blocking STDDEBUG channel along with STDIO's STDIN, STDSOUT and STDERR. It might make more sense and in the way that we can use a simple shell redirection of any channel (q.v. man page) it actually offers more flexibility for what Carlos is doing than using syslog as an debug channel. Being able to redirect, filter etc on the command like, tee off STDERR, run it separately through grep ... Oh wait, have you never split an IO channel into a fan-out tree with multiple tee commands? By comparison to what you can do on the command line dynamically compared to having to edit a syslog config file is ...well, I'm a CLI/shell maven of old. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Tue, 20 Dec 2016, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
Bog standard openSUSE syslog-ng default config (/usr/share/doc/packages/syslog-ng/syslog-ng.conf.default) has this: ==== [..] filter f_console { level(warn) and facility(kern) and not filter(f_iptables) or level(err) and not facility(authpriv); }; filter f_newsnotice { level(notice) and facility(news); }; filter f_newscrit { level(crit) and facility(news); }; filter f_newserr { level(err) and facility(news); }; filter f_news { facility(news); }; [..] filter f_messages { not facility(news, mail) and not filter(f_iptables); }; filter f_warn { level(warn, err, crit) and not filter(f_iptables); }; filter f_alert { level(alert); }; [..] # # Enable this, if you want to keep all messages in one file: # (don't forget to provide logrotation config) # #destination allmessages { file("/var/log/allmessages"); }; #log { source(src); destination(allmessages); }; [..] # # News-messages in separate files: # destination newscrit { file("/var/log/news/news.crit" suppress(30) owner(news) group(news)); }; log { source(src); source(chroots); filter(f_newscrit); destination(newscrit); }; destination newserr { file("/var/log/news/news.err" suppress(30) owner(news) group(news)); }; log { source(src); source(chroots); filter(f_newserr); destination(newserr); }; destination newsnotice { file("/var/log/news/news.notice" suppress(30) owner(news) group(news)); }; log { source(src); source(chroots); filter(f_newsnotice); destination(newsnotice); }; # # and optionally also all in one file: # (don't forget to provide logrotation config) # #destination news { file("/var/log/news.all"); }; #log { source(src); source(chroots); filter(f_news); destination(news); }; [..] # # All messages except iptables and the facilities news and mail: # destination messages { file("/var/log/messages" suppress(30) owner(-1) group(-1) perm(-1)); }; log { source(src); source(chroots); filter(f_messages); destination(messages); }; [..] # # Warnings (except iptables) in one file: # destination warn { file("/var/log/warn" suppress(30) fsync(yes)); }; log { source(src); source(chroots); filter(f_warn); destination(warn); }; ==== That means you can pick and choose which files stuff goes where and choose to keep each file as long as you want, e.g. deleting newsnotice after a week, but keep mail logs for years. That's what Carlos was talking about. And BTW: syslog-ng has a sql() destination option, so you could log (some messages) directly into a database[1], e.g. just the news notice stuff: ==== destination d_newsnotice_sql { sql(type(pgsql) host("localhost") username("syslog-ng") password("password") database("logs_newsnotice") table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}") columns("datetime", "host", "program", "pid", "message") values("{$R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}") indexes("datetime", "host", "program", "pid", "message")); }; log { source(src); source(chroots); filter(f_newsnotice); destination(d_newsnotice_sql); }; ==== Or whatever. Just grabbed the first example from [1]. And BTW: I have logs since Jan. 2011 in my /var/log (and there's quite a bit of debugging noise in them). # du -hs /var/log 249M /var/log And logging coredumps??? -dnh [1] Currently the Microsoft SQL (MSSQL), MySQL, Oracle, PostgreSQL, and SQLite databases are supported. https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose-... -- Get back there in front of the computer NOW. Christmas can wait. -- Linus "the Grinch" Torvalds, 24 Dec 2000 on linux-kernel -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-21 04:02, David Haller wrote:
Hello,
On Tue, 20 Dec 2016, jdd wrote:
Le 20/12/2016 à 17:36, Carlos E. R. a écrit :
I don't use inn :-)
well, it's the same problem. Don't use standard system log for debug an application. you could as well flood /var/log/messages with this
Bog standard openSUSE syslog-ng default config (/usr/share/doc/packages/syslog-ng/syslog-ng.conf.default) has this:
... ...
That means you can pick and choose which files stuff goes where and choose to keep each file as long as you want, e.g. deleting newsnotice after a week, but keep mail logs for years.
That's what Carlos was talking about.
Exactly.
And BTW: syslog-ng has a sql() destination option, so you could log (some messages) directly into a database[1], e.g. just the news notice stuff:
Yes, exactly. A binary database in a standard format, which LibreOffice (for example) can read directly and dynamically, no translation.
Or whatever. Just grabbed the first example from [1].
And BTW: I have logs since Jan. 2011 in my /var/log (and there's quite a bit of debugging noise in them).
Me, 2012 at least.
# du -hs /var/log 249M /var/log
And logging coredumps???
I think currently they go to a different directory, not inside the journal. Fortunately. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhaVw8ACgkQja8UbcUWM1zXIwD+Mg9MlvMeGligiUate6n+s/h4 XFVvgO30eWLbc6efAFgA/ja2AjA15QumbAMlMca0PJ/0h3kpz+2c6S4lGstbQJ8U =dk1c -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 06:24 AM, Andrei Borzenkov wrote:
Systemd author is actively opposed to any idea to filter log entries in journald.
So here I am at a bank or a brokerage firm or some other regulated commodity (and yes, I've been though similar at a electricity provider and natural gas distributor) and its important that many things, event logs, business transaction logs, various types of communication, NOT be filtered and in some cases not deleted, perhaps never, perhaps only for seven years. The ones I've worked -- oh and many major NOCs I've visited and interviewed at as well - at have a dedicated team with dedicated high powered machines and huge amounts of disk devoted to logging and log analysis. Some are dealing with attacks and intrusions, insider violations, and of course maintenance events. As I've posted before, one such was simply a ITIL event management interface: if anything happened at one of nearly 50,000 branches of the bank across Canada and the Caribbean they knew about it, if it was more than just an out-of-paper or paper jam they'd notify the office or the support contractor for that area. This is how BIG FIRMS operate. (You could say that this is how IBM taught them to operate by supplying this elves of service for them back in the days ...) The reality is that more and more Linux, as well as Windows, is addressing the needs of Business, and Big Business/Wall Street at that, rather than the singleton home user, the hobbyist. This has become true of the desktop and the laptop as well, with RAID on the laptop, BtrFS for snapshotting, lots more. To be more comprehensive: the market has polarized. The casual user of old has not moved to the phone or tablet for browsing, email/gmail, news pages, Twitter, Facebook. The high demand singleton users are gamers who probably spend more on a graphics board than on the rest of the machine. Other such 'verticals'. That makes many of us 'hobbyists' on PC a bit of an anachronism. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 13:38, Anton Aylward wrote:
On 12/20/2016 06:24 AM, Andrei Borzenkov wrote:
Systemd author is actively opposed to any idea to filter log entries in journald.
So here I am at a bank or a brokerage firm or some other regulated commodity (and yes, I've been though similar at a electricity provider and natural gas distributor) and its important that many things, event logs, business transaction logs, various types of communication, NOT be filtered and in some cases not deleted, perhaps never, perhaps only for seven years.
I a professional setup I would keep syslog, never rely on journal alone. At worst, I would have to keep both, because journal is not up to the task. And yes, I have worked in that area. And we kept both, plain compressed text logs, and binary database logs. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 09:21 AM, Carlos E. R. wrote:
I a professional setup I would keep syslog, never rely on journal alone. At worst, I would have to keep both, because journal is not up to the task.
In what way is it not up to the task? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 15:55, Anton Aylward wrote:
On 12/20/2016 09:21 AM, Carlos E. R. wrote:
I a professional setup I would keep syslog, never rely on journal alone. At worst, I would have to keep both, because journal is not up to the task.
In what way is it not up to the task?
Not customizable, and no tools ready made. Also no guarantee that I will be able to read the logs after an update or on a different machine. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 10:29, ianseeks wrote:
On Tuesday, 20 December 2016 04:43:03 GMT Carlos E. R. wrote:
I have an nntp local server (proxy), and it produces tons of log messages per day. With syslog I can dump them all after, say, a week. With journal, I can't. If I want to keep a reasonable long term journal of all important events in my machine, it also keeps the useless debug logs of everything else... I can not purge the database of selected content.
Are you sure about that? There are loads of configuration options in journald.conf regarding storage and storage sizes and retention times
Yes, I can control size. My current solution is zero size. It is not that. What I need is a method to delete from the journal thousands of messages automatically that match a certain criteria. Say, debug level messages older than a week for facility nntp. Ie, purging.
With syslog, I can keep logs dating back for a year or two and it is no issue. I simply dump the verbose debug logs of some facilities like nntp. Syslog/rsyslog can now read all log records directly from the journal (might depend on the version you use) and then you could continue as you normally do, just set storage to "volatile" or "none" and forward records to syslog socket (according to journald.conf)
Yes, that's is what I do, because I can't do it with journal, because it grows to an insane size. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Monday, 19 December 2016 18:32:11 GMT Carlos E. R. wrote:
On 2016-12-19 18:24, Anton Aylward wrote:
On 12/19/2016 10:56 AM, Carlos E. R. wrote:
On 2016-12-19 16:29, Anton Aylward wrote:
On 12/18/2016 02:21 PM, Greg Freemyer wrote:
4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.
Hmm. So if I run journald can I get rid of all those other log files?
What other log files? You mean non-syslog log files? Those other log files are not handled by journald, either. They are handled by the specific applications that create them.
In that any program can, yes, but that's not my point.
My point is that that journald logs so many of these things, or can.
journalctl _COMM=sshd journalctl _COMM=login journalctl _COMM=su journalctl _COMM=systemd-logind
Of course most access involves PAM one way or another so just make sure you make use of pam_logind !!!
That's just login sessions of different types. I don't remember where these where kept before.
if you really want to be paranoid, you can forget about the indexoing that journalctl can do and simple grep
journalctl |grep login | more
Takes minutes with rotating rust.
Oh My!
Well that soaks up quite a few of those log files: utmp/wtmp, lastlog
More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier.
No correlation unless you have a database viewer and analysis tool.
Ah, tools!
Well, for a start, journalctl does index.
What does that mean? Examples, please.
Then we have a;; the regular tools, from 'cut'/'join' of V7 UNIX vintage, on though 'awk' (see the appropriate White Book for how to convert text to database and do correlations and produce nicely formatted output) on though Perl, which was designed for doing that kind of thing.
Nonono. I want ready made user end tools. I do not want to learn programming awk now or have to write my own tools.
You've been around long enough, Carlos, to know and have done this sort of thing. What's this, some anti-systemd feeling?
No, not at all.
But then, logs may be as difficult to analyze as they currently are in Windows.
This isn't Windows. <obscentity> Windows. part of the nature of Windows is to make you buy some third party tool to do things that with UNIX/Linux can be done with a few lies of scripting.
The journal thing in Linux is getting closer to what Windows does with logs. That's my fear. I'm unable to find events in Windows system log, that I know they are there.
Windows is a special case, they do make some things as difficult as possible to hide things they don't want you to see.
-- Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 "Bottle" (Minas Tirith))
-- opensuse:tumbleweed:20161217 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* ianseeks
On Monday, 19 December 2016 18:32:11 GMT Carlos E. R. wrote:
On 2016-12-19 18:24, Anton Aylward wrote:
On 12/19/2016 10:56 AM, Carlos E. R. wrote:
On 2016-12-19 16:29, Anton Aylward wrote:
On 12/18/2016 02:21 PM, Greg Freemyer wrote:
4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.
Hmm. So if I run journald can I get rid of all those other log files?
What other log files? You mean non-syslog log files? Those other log files are not handled by journald, either. They are handled by the specific applications that create them.
In that any program can, yes, but that's not my point.
My point is that that journald logs so many of these things, or can.
journalctl _COMM=sshd journalctl _COMM=login journalctl _COMM=su journalctl _COMM=systemd-logind
Of course most access involves PAM one way or another so just make sure you make use of pam_logind !!!
That's just login sessions of different types. I don't remember where these where kept before.
if you really want to be paranoid, you can forget about the indexoing that journalctl can do and simple grep
journalctl |grep login | more
Takes minutes with rotating rust.
Oh My!
Well that soaks up quite a few of those log files: utmp/wtmp, lastlog
More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier.
No correlation unless you have a database viewer and analysis tool.
Ah, tools!
Well, for a start, journalctl does index.
What does that mean? Examples, please.
Then we have a;; the regular tools, from 'cut'/'join' of V7 UNIX vintage, on though 'awk' (see the appropriate White Book for how to convert text to database and do correlations and produce nicely formatted output) on though Perl, which was designed for doing that kind of thing.
Nonono. I want ready made user end tools. I do not want to learn programming awk now or have to write my own tools.
You've been around long enough, Carlos, to know and have done this sort of thing. What's this, some anti-systemd feeling?
No, not at all.
But then, logs may be as difficult to analyze as they currently are in Windows.
This isn't Windows. <obscentity> Windows. part of the nature of Windows is to make you buy some third party tool to do things that with UNIX/Linux can be done with a few lies of scripting.
The journal thing in Linux is getting closer to what Windows does with logs. That's my fear. I'm unable to find events in Windows system log, that I know they are there.
Windows is a special case, they do make some things as difficult as possible to hide things they don't want you to see.
-- Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 "Bottle" (Minas Tirith))
-- opensuse:tumbleweed:20161217 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
while you are unable to trim *anything* :( and add nothing of any value to the conversation :0: * ^From:.*\bingmybong@btinternet.com /dev/null I will no longer be bothered! -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 Photos: http://wahoo.no-ip.org/piwigo @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Patrick Shanahan composed on 2016-12-19 15:33 (UTC-0500):
while you are unable to trim *anything* :( and add nothing of any value to the conversation
Annoyed and complaining about absence of trimming I get. It bothers me too.
:0: * ^From:.*\bingmybong@btinternet.com /dev/null
I will no longer be bothered!
Announcing who and how you killfile I do not get. You think people who don't trim even understand what you've done? What's the point? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-19 21:52, Felix Miata wrote:
Patrick Shanahan composed on 2016-12-19 15:33 (UTC-0500):
I will no longer be bothered!
Announcing who and how you killfile I do not get. You think people who don't trim even understand what you've done? What's the point?
What's happened today? I see people very grumpy. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYWKgACgkQja8UbcUWM1wJIQD/VVrDSEMTzjMSscvD0/KxhvtH /tUMkCK+xPvvy63SQjQA/0AfPAHuml82m/z5PBv4tGUKSe4X+fPonGb/eE5AMqxs =RTua -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 5:01 PM, Carlos E. R. wrote:
What's happened today? I see people very grumpy.
- -- Cheers / Saludos, give me coffee and no one gets hurt? :-) here it is either that or insufficient donuts.
Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Problems such as not trimming, seem to be more prevalent on lists where users demand bottom posting on the basis of it providing a top-down readable history of the conversation. In order for bottom posters to be able to read previous conversation, they often include the full text of such conversations. On lists that don't force bottom posting, the users quickly learn to just include the relevant parts they are responding to. (just my experience and certainly not a 100% rule)... -l Patrick Shanahan wrote:
while you are unable to trim *anything* :( and add nothing of any value to the conversation
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* L A Walsh
Problems such as not trimming, seem to be more prevalent on lists where users demand bottom posting on the basis of it providing a top-down readable history of the conversation.
In order for bottom posters to be able to read previous conversation, they often include the full text of such conversations.
On lists that don't force bottom posting, the users quickly learn to just include the relevant parts they are responding to.
(just my experience and certainly not a 100% rule)...
-l
Patrick Shanahan wrote:
while you are unable to trim *anything* :( and add nothing of any value to the conversation
I would hazzard that you are mis-speaking. *Most* top poster cannot be bothered with *anything* below their comment. But you have continually exhibited actions contrary to the opensuse list netiquette statements including but not limited to posting to the list and duplicating to the poster(s). i'm done! -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 Photos: http://wahoo.no-ip.org/piwigo @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Patrick Shanahan wrote:
But you have continually exhibited actions contrary to the opensuse list netiquette statements including but not limited to posting to the list and duplicating to the poster(s).
---- Oh? Continually? I certainly didn't do that with you nor do I do it continually -- only those who invite me to by having their emailer asking me to -- which is NOT most people on this list. So what are you complaining about? That time of the year?
i'm done!
Over an observation? My aren't we touchy. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Patrick Shanahan wrote:
* L A Walsh
[12-19-16 22:07]: On lists that don't force bottom posting, the users quickly learn to just include the relevant parts they are responding to.
I would hazzard that you are mis-speaking. *Most* top poster cannot be bothered with *anything* below their comment.
That is my impression too. When you're bottom posting, you're forced to page through whatever it is you're responding to. Kind of inspires one to chop away at it. -- Per Jessen, Zürich (0.7°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 20/12/2016 à 08:58, Per Jessen a écrit :
That is my impression too. When you're bottom posting, you're forced to page through whatever it is you're responding to. Kind of inspires one to chop away at it.
some users answer through smartphones trimming is very difficult with the ones I know, so I usually wait I have a real keyboard to write and only use smartphone to read/delete jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 03:29 AM, jdd wrote:
some users answer through smartphones
trimming is very difficult with the ones I know, so I usually wait I have a real keyboard to write and only use smartphone to read/delete
+1 to all that. We really need to complain to the developers of the UIs for phones about this. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tuesday, 20 December 2016 08:58:38 GMT Per Jessen wrote:
Patrick Shanahan wrote:
* L A Walsh
[12-19-16 22:07]: On lists that don't force bottom posting, the users quickly learn to just include the relevant parts they are responding to.
I would hazzard that you are mis-speaking. *Most* top poster cannot be bothered with *anything* below their comment.
That is my impression too. When you're bottom posting, you're forced to page through whatever it is you're responding to. Kind of inspires one to chop away at it.
Seems if you can be deemed a **** if you top post, bottom post, trim and no trim. I've seen moaning about all of those actions. -- opensuse:tumbleweed:20161217 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 20/12/2016 à 10:33, ianseeks a écrit :
Seems if you can be deemed a **** if you top post, bottom post, trim and no trim. I've seen moaning about all of those actions.
It's always better to trim. I mostly don't read posts written below my screen write bottom if you answer to a sentence of the post (like I do here). write top if your answer is more general, not related to the post content (for example to answer the subject if the thread derived to other content, like it could be here). keep the previous content is the reader may not have read it, for example restarting an old thread or answering to a commercial letter the sender may not have archived my two cents :-) jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 05:06 AM, jdd wrote:
Le 20/12/2016 à 10:33, ianseeks a écrit :
Seems if you can be deemed a **** if you top post, bottom post, trim and no trim. I've seen moaning about all of those actions.
It's always better to trim. I mostly don't read posts written below my screen
There's another reason to trim: You might call it FOCUS, which is an important aspect of communication. If you don't, it accumulates and becomes TL;DR. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 11:58 PM, Per Jessen wrote: <- possibly important context snipped ->
That is my impression too. When you're bottom posting, you're forced to page through whatever it is you're responding to. Kind of inspires one to chop away at it.
Indeed. This is why there's justification for both. There are certain environments where you want to carry thread history for future reference. These would tend to be professional team incident response and operational threads. I personally prefer top-posting for everything, but I'm not religiously fanatic about it. In the immortal words of Rodney King: "...can we all get along?" Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Monday, 19 December 2016 22:15:09 GMT Patrick Shanahan wrote:
* L A Walsh
[12-19-16 22:07]: Problems such as not trimming, seem to be more prevalent on lists
where users demand bottom posting on the basis of it providing a top-down readable history of the conversation.
In order for bottom posters to be able to read previous conversation, they
often include the full text of such conversations.
On lists that don't force bottom posting, the users quickly learn to just
include the relevant parts they are responding to.
(just my experience and certainly not a 100% rule)...
-l
Patrick Shanahan wrote:
while you are unable to trim *anything* :( and add nothing of any value to the conversation
I would hazzard that you are mis-speaking. *Most* top poster cannot be bothered with *anything* below their comment.
But you have continually exhibited actions contrary to the opensuse list netiquette statements including but not limited to posting to the list and duplicating to the poster(s).
i'm done!
Is being rude, rather than giving a polite reminder, to people when pointing out such things also part of the "netiquette"? -- opensuse:tumbleweed:20161217 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.13-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-19 19:44, ianseeks wrote:
The journal thing in Linux is getting closer to what Windows does with logs. That's my fear. I'm unable to find events in Windows system log, that I know they are there.
Windows is a special case, they do make some things as difficult as possible to hide things they don't want you to see.
The thing in Windows is that you have to know the correct "token" to do the filter, or you find nothing. Like getting the output on a particular index in journalctl. Either you know how the particular item you seek for is indexed, or you don't get anything. There is no text see it all output. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYWTQACgkQja8UbcUWM1ze5gD+LP7ohKB6WB+qtKvRWzuJXJ+D 9R+dYI8i+JoaRMBbeqAA/1DXvWP43ZhxxzGqy06P7m+WE0166bsMyQg/nZbamgF7 =RD7u -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 05:03 PM, Carlos E. R. wrote:
On 2016-12-19 19:44, ianseeks wrote:
The journal thing in Linux is getting closer to what Windows does with logs. That's my fear. I'm unable to find events in Windows system log, that I know they are there.
Windows is a special case, they do make some things as difficult as possible to hide things they don't want you to see.
The thing in Windows is that you have to know the correct "token" to do the filter, or you find nothing. Like getting the output on a particular index in journalctl. Either you know how the particular item you seek for is indexed, or you don't get anything. There is no text see it all output.
Yes, but journalctl will tell you - see my post about <tab><tab> -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 00:12, Anton Aylward wrote:
On 12/19/2016 05:03 PM, Carlos E. R. wrote:
On 2016-12-19 19:44, ianseeks wrote:
The journal thing in Linux is getting closer to what Windows does with logs. That's my fear. I'm unable to find events in Windows system log, that I know they are there.
Windows is a special case, they do make some things as difficult as possible to hide things they don't want you to see.
The thing in Windows is that you have to know the correct "token" to do the filter, or you find nothing. Like getting the output on a particular index in journalctl. Either you know how the particular item you seek for is indexed, or you don't get anything. There is no text see it all output.
Yes, but journalctl will tell you - see my post about <tab><tab>
Well, it doesn't work on the two machines I have available... :-? - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYbzMACgkQja8UbcUWM1y1nAD/UNF+AN994vYOHqDB6ReImBtm T9M4kf1Mn3vL9zzzMCoA/2cvQZ5GZvcaa284BzKoabcKhOgRZ7dPaJh5386vbmwg =aeiX -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 06:37 PM, Carlos E. R. wrote:
On 2016-12-20 00:12, Anton Aylward wrote:
Yes, but journalctl will tell you - see my post about <tab><tab>
Well, it doesn't work on the two machines I have available... :-?
*sigh* It seems you don't have the completion package installed/active -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Mon, Dec 19, 2016 at 10:29 AM, Anton Aylward
On 12/18/2016 02:21 PM, Greg Freemyer wrote:
4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.
Hmm. So if I run journald can I get rid of all those other log files? either delete them and hope the subsystem won't recreate them or link them to /dev/null ?
More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier.
we just need tools!
RE: tools I don't know all the details, but: libprelude is an opensource log centralization toolkit. It allows a central logserver to collect logs from various other servers/devices. I believe there is a lot of IT security community support behind libprelude. libprelude went into factory 6 months ago and is in Leap 42.2 The Prelude SIEM builds on libprelude and does event correlation and notification. It too is now in factory and Leap 42.2. Suricata is also an opensource SIEM. Better known than the Prelude SIEM I think. It layers on top of libprelude as well. It is in OBS, but not factory or Leap. I don't know if there is any interrelationship between journald and libprelude? So there is lots of activity in the system logging space, but I've fallen behind. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier.
--- That was always your choice. If you found multiple log files unsuitable for your purposes there was no one forcing you to keep them that way. They way they were configure was _ONE_ way out of hundreds or thousands of ways you could choose to have your log pre-parsed and pre-stored. Other utilities could then operate on a specific log or on 1 "all-in-one" log. Examples in the syslog utils showed how to configure an "all-in-one" log AND subsystem or program specific logs, at the same time.
we just need tools!
There were 100's or 1000's, any thing that processed text files could process log files. Now? I guess it will be a while before the tool count gets back to parity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 03:43, L A Walsh wrote:
Anton Aylward wrote:
More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier.
--- That was always your choice. If you found multiple log files unsuitable for your purposes there was no one forcing you to keep them that way. They way they were configure was _ONE_ way out of hundreds or thousands of ways you could choose to have your log pre-parsed and pre-stored.
Yes, you can have all syslog entries stored in a single file. The syslog configuration usually contains a commented out entry for /var/log/allmessages. But he doesn't refer to that one. He refers to other services that use their independent log system, like the login services. Have all services log to the same log service, the journal. Apache, ssh, etc. I don't know. Without a database viewer, like an automatic import to LibreOffice where we can choose filters to see what we want or not, I see no advantage in a database. Somewhere where we can click on an event, follow it, discard garbage, etc. No ready made tools that I know of. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/19/2016 10:52 PM, Carlos E. R. wrote:
I don't know. Without a database viewer, like an automatic import to LibreOffice where we can choose filters to see what we want or not, I see no advantage in a database. Somewhere where we can click on an event, follow it, discard garbage, etc.
I think perhaps that's the wrong approach. Yes, as I've mentioned, journalctl -o json | json-to-csv or similar lets you import a snapshot into ooclac. So what? You can do the same with an Oracle database,a MySyql/MariaDB database. But in reality what I do when I want to manipulate a MySql database with a GUI is to use a tool someone wrote in Python, 'phpMyAdmin'. There's one for sqlight3 as well. It works on the database itself, not on an export. Of course 'phpMyAdmin' hasn't always existed. MySQL was around for a long time before that tool was written for it. MySQL was initially released in 1995. Tobias Ratschiller started working on what became phpMyAdmin in 1998. Perhaps someone is working on a GUI that like phpMyAdmin access the systemctl database directly. The documentation is out there .. In the mean time, the real issue isn't that you can't import a snapshot of the journalctl into oocalc, its that ooclc ignores the JSON standard unless you have the add-in for it. Seems dumb to me that it should handle CSV and not JSON in this day and age. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 12:35 AM, Anton Aylward wrote: phpMyAdmin is network bnased. Journal can be accessed over the net as well See https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.se... Maybe your hypothetical GUI would like a real-time feed? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
In the mean time, the real issue isn't that you can't import a snapshot of the journalctl into oocalc, its that ooclc ignores the JSON standard unless you have the add-in for it. Seems dumb to me that it should handle CSV and not JSON in this day and age.
For largish amounts of data (megabytes), JSON or XML is overkill, but CSV is very efficient. -- Per Jessen, Zürich (0.6°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 02:55 AM, Per Jessen wrote:
Anton Aylward wrote:
In the mean time, the real issue isn't that you can't import a snapshot of the journalctl into oocalc, its that ooclc ignores the JSON standard unless you have the add-in for it. Seems dumb to me that it should handle CSV and not JSON in this day and age.
For largish amounts of data (megabytes), JSON or XML is overkill, but CSV is very efficient.
if you're talking "byte efficient" as in low overhead, then yes. But the whole point of something like XML is not about byte efficiency, it is about semantic clarity and error handling. A few garbled bytes of a CSV writes off everything that follows. A few garbled bytes of a XML transmission resyncs on the following stanza because of the semantic structure. JSON is similar though the contrast between XML and JSON is rather like between ALGOL and C with respect to keywords vs curly brackets. Error resilience, semantic coherency, correct structuring, are going to be more important when dealing with large amounts of data in a stream. The present costs of storage and bandwidth mean the ideas of 'efficiency' we old fashioned engineers were brought up to now take second place. There is in my database of DotSigQuotes; ------------------ [Alice] has courage which can only be described as awesome. Against all odds, over a noisy telephone line, tapped by the tax authorities and the secret police, Alice will happily attempt, with someone [Bob] she doesn't trust, whom she cannot hear clearly, and who is probably someone else, to fiddle her tax returns and to organize a coup d'etat, while at the same time minimizing the cost of the phone call. A coding theorist is someone who doesn't think Alice is crazy. -------------------- The moral here is that most people are not coding theorists. Or engineers striving for 'byte efficiency'. The efficiencies of XML and JSON have more to do with Alice's noisy telephone line in that they are streaming protocols, the data being supplied in large amounts from a remote source over a high speed network. The cost, considering the large mount of data, of going back to to the start if a burst error occurs, is enormous. Not just in terms of "the phone call" but in terms of time, since this is real-time streaming data. Yes, I'm perfectly aware that you cold code up a packetized protocol for CSV with checksum for each packet and acknowledgement of good receipt vs request of retransmission, sort of like TCP writ large. But pretty soon you get into the overhead and you get into all the issues with queueing and flow control that has produced hundreds of studies and papers about TCP packet management over the decades. You really don't want to go there. Use XML or JSON instead. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 12/20/2016 02:55 AM, Per Jessen wrote:
Anton Aylward wrote:
In the mean time, the real issue isn't that you can't import a snapshot of the journalctl into oocalc, its that ooclc ignores the JSON standard unless you have the add-in for it. Seems dumb to me that it should handle CSV and not JSON in this day and age.
For largish amounts of data (megabytes), JSON or XML is overkill, but CSV is very efficient.
if you're talking "byte efficient" as in low overhead, then yes.
Not only that, but compression works very well, so transmission time and storage footprint are less important. I was more thinking of subsequent processing, loading into databases and such.
But the whole point of something like XML is not about byte efficiency, it is about semantic clarity and error handling. A few garbled bytes of a CSV writes off everything that follows. A few garbled bytes of a XML transmission resyncs on the following stanza because of the semantic structure.
Anton, you're not actually trying to argue that you would use XML to ensure proper data transmission? :-) XML and JSON both have their places, but when I'm downloading 1million lines of log entries and "cooking" them down to 125.000 new database entries, they're both out place, IMHO. (and that's the logdata from just one mirror site). Massaging that data in and out of XML or JSON format would be 100% overhead.
JSON is similar though the contrast between XML and JSON is rather like between ALGOL and C with respect to keywords vs curly brackets.
They're both just structured formats for data interchange - you could also mention EDI or the PHP serialize format. When sender and recipient cannot or will not agree on a common format, using XML or JSON can solve the problem.
Yes, I'm perfectly aware that you cold code up a packetized protocol for CSV with checksum for each packet and acknowledgement of good receipt vs request of retransmission, sort of like TCP writ large.
Right, that's called TCP. Or maybe rsync over TCP.
But pretty soon you get into the overhead and you get into all the issues with queueing and flow control that has produced hundreds of studies and papers about TCP packet management over the decades. You really don't want to go there.
Use XML or JSON instead.
Not for volume data transmission between two cooperating parties. It's only 100% overhead. -- Per Jessen, Zürich (1.7°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 08:27 AM, Per Jessen wrote:
Anton, you're not actually trying to argue that you would use XML to ensure proper data transmission? :-)
LOL! No, I use streaming XHTML, or for some data feeds stream JSON. We all do, in case you hadn't noticed. For example, I use Firefox and store my session snapshots as JSON. I store all my mindmaps as XML - the semantic content is important. I get my weather reports and other streaming data from services that supply it as streaming JSON. If you use any web application that does behind the scenes updating of data it probably uses the AJAX/JavaScript mechanisms to get asynchronous data from the server. you see that with google when you type a string and it presents suggestions as as pull-down options. What's going on behind the scenes uses data transmission in XML or more likely JSON. The whole point of JSON, which, lets face it, is not that much of an overhead wrt CSV, is the structuring of the data. It makes streaming assimilation much easier and as I say, had some error handling capabilities that do not exist with CSV. Why do I think streaming is so important? Well, try this. Instead of using a web browser, access all your pages by downloading them using wget or curl and then displaying them. you have to wait for the whole file to be dowloaded, all the CSS to be downloaded, all the images to be downloaded all before you see ANYTHING AT ALL. Most sites now make use of streaming mechanism. They may not display the whole page, but you don't get too frustrated as they display the structure, title, the take-away synopses and keep up with you as you read down the page and scroll. To do this there must be structural and possibly semantic data n the stream so that the parser & display know where to put things. Yes, sometimes they get it wrong and piss poor design or piss poor implementation means you page jitters around :-( There are so many situations where real time data flow from remote sources is important. Databases? Well I've seen brokerage terminals where the database of stock trades, performance is both historic and up to the minute because its drawing on both the archived data and the streaming current data. The volumes involved are very large and they are continuous. But their structure and semantic content is very important as the INTEGRITY must be guaranteed. You can't do that with CSV. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 12/20/2016 08:27 AM, Per Jessen wrote:
Anton, you're not actually trying to argue that you would use XML to ensure proper data transmission? :-)
LOL! No, I use streaming XHTML, or for some data feeds stream JSON.
We all do, in case you hadn't noticed.
For example, I use Firefox and store my session snapshots as JSON.
I store all my mindmaps as XML - the semantic content is important.
I get my weather reports and other streaming data from services that supply it as streaming JSON.
If you use any web application that does behind the scenes updating of data it probably uses the AJAX/JavaScript mechanisms to get asynchronous data from the server. you see that with google when you type a string and it presents suggestions as as pull-down options. What's going on behind the scenes uses data transmission in XML or more likely JSON.
The whole point of JSON, which, lets face it, is not that much of an overhead wrt CSV, is the structuring of the data. It makes streaming assimilation much easier and as I say, had some error handling capabilities that do not exist with CSV.
Sure, but none of this applies to what I started out saying - large data volumes.
Why do I think streaming is so important?
Well, try this.
Instead of using a web browser, access all your pages by downloading them using wget or curl and then displaying them. you have to wait for the whole file to be dowloaded, all the CSS to be downloaded, all the images to be downloaded all before you see ANYTHING AT ALL.
Same in the browser, Anton. It just downloads a number of items concurrently, probably prioritising those that are needed for the initial page build-up, leaving e.g. large graphics for later. I would not talk about streaming, it's just parallel downloads. Streaming is what happens when I listen to the radio or watch youtube or TV.
Most sites now make use of streaming mechanism. They may not display the whole page, but you don't get too frustrated as they display the structure, title, the take-away synopses and keep up with you as you read down the page and scroll. To do this there must be structural and possibly semantic data n the stream so that the parser & display know where to put things. Yes, sometimes they get it wrong and piss poor design or piss poor implementation means you page jitters around :-(
There are so many situations where real time data flow from remote sources is important. Databases? Well I've seen brokerage terminals where the database of stock trades, performance is both historic and up to the minute because its drawing on both the archived data and the streaming current data. The volumes involved are very large and they are continuous. But their structure and semantic content is very important as the INTEGRITY must be guaranteed. You can't do that with CSV.
I never claimed I could. I only said there is a place and time for both and that formatting structured data can be 100% overhead, depending on the situation. -- Per Jessen, Zürich (1.8°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
Why do I think streaming is so important?
Well, try this.
Instead of using a web browser, access all your pages by downloading them using wget or curl and then displaying them. you have to wait for the whole file to be dowloaded, all the CSS to be downloaded, all the images to be downloaded all before you see ANYTHING AT ALL.
Most sites now make use of streaming mechanism.
--- You are confusing incremental display & page update with streaming. They aren't the same. Example. If you store a jpg, you have the option to optimize for the web. Part of that is displaying, say, every 4th row from line 0 of the image every forth row on your screen (0 4 8...). In 1/4th the time of displaying the whole image, you have a low-ish resolution picture. Then it does the same offset by 2 (2 6 10...). Now your pic has 2x the resolution and looks clearer. Finally rows 1 and 3 are filled in in each group of 4. A way that looks slower and is more frustrating is strictly going from top to bottom (0,1,2,3,4...). However, in both cases, you are getting a stream of binary data from the server. Similarly, when a webpage displays, you first see text content, then it jumps into position as the css stylesheet is next downloaded and applied, then pictures will start filling in. Those are the result of how the browser asks for information. URL 1st, sees 'CSS' stylesheet ref at top and asks for it while displaying text from the URL. As the text is parsed, images and other embed's are requested by the client. Multiple requests are usually being downloaded at the same time and filled in **incrementally** (!=streaming). No web brower waits until everything is downloaded to display anything -- its *ALWAYS* been that way and has nothing to do with streaming. -l -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 06:35, Anton Aylward wrote:
On 12/19/2016 10:52 PM, Carlos E. R. wrote:
Yes, as I've mentioned,
journalctl -o json | json-to-csv
or similar lets you import a snapshot into ooclac.
Minas-Anor:~ # journalctl -o json | json-to-csv If 'json-to-csv' is not a typo you can use command-not-found to lookup the package that contains it, like this: cnf json-to-csv Minas-Anor:~ # cnf json-to-csv json-to-csv: command not found Minas-Anor:~ # -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2016-12-20 06:35, Anton Aylward wrote:
On 12/19/2016 10:52 PM, Carlos E. R. wrote:
Yes, as I've mentioned,
journalctl -o json | json-to-csv
or similar lets you import a snapshot into ooclac.
Minas-Anor:~ # journalctl -o json | json-to-csv If 'json-to-csv' is not a typo you can use command-not-found to lookup the package that contains it, like this: cnf json-to-csv Minas-Anor:~ # cnf json-to-csv json-to-csv: command not found
Take it as an example, it would be straight forward to write such a converter. You can on doubt find one out there already. -- Per Jessen, Zürich (2.0°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 10:11 AM, Per Jessen wrote:
Carlos E. R. wrote:
On 2016-12-20 06:35, Anton Aylward wrote:
On 12/19/2016 10:52 PM, Carlos E. R. wrote:
Yes, as I've mentioned,
journalctl -o json | json-to-csv
or similar lets you import a snapshot into ooclac.
Minas-Anor:~ # journalctl -o json | json-to-csv If 'json-to-csv' is not a typo you can use command-not-found to lookup the package that contains it, like this: cnf json-to-csv Minas-Anor:~ # cnf json-to-csv json-to-csv: command not found
Take it as an example, it would be straight forward to write such a converter. You can on doubt find one out there already.
Yes, IIR giving an example. There are libraries for this in PHP, Python, Perl and Ruby, as well as on-line conversion services. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 16:29, Anton Aylward wrote:
On 12/20/2016 10:11 AM, Per Jessen wrote:
Carlos E. R. wrote:
On 2016-12-20 06:35, Anton Aylward wrote:
On 12/19/2016 10:52 PM, Carlos E. R. wrote:
Yes, as I've mentioned,
journalctl -o json | json-to-csv
or similar lets you import a snapshot into ooclac.
Minas-Anor:~ # journalctl -o json | json-to-csv If 'json-to-csv' is not a typo you can use command-not-found to lookup the package that contains it, like this: cnf json-to-csv Minas-Anor:~ # cnf json-to-csv json-to-csv: command not found
Take it as an example, it would be straight forward to write such a converter. You can on doubt find one out there already.
Yes, IIR giving an example. There are libraries for this in PHP, Python, Perl and Ruby, as well as on-line conversion services.
Useless to me. I can not write that tool. Thus, "no tools" -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
20.12.2016 19:41, Carlos E. R. пишет:
On 2016-12-20 16:29, Anton Aylward wrote:
On 12/20/2016 10:11 AM, Per Jessen wrote:
Carlos E. R. wrote:
On 2016-12-20 06:35, Anton Aylward wrote:
On 12/19/2016 10:52 PM, Carlos E. R. wrote:
Yes, as I've mentioned,
journalctl -o json | json-to-csv
or similar lets you import a snapshot into ooclac.
Minas-Anor:~ # journalctl -o json | json-to-csv If 'json-to-csv' is not a typo you can use command-not-found to lookup the package that contains it, like this: cnf json-to-csv Minas-Anor:~ # cnf json-to-csv json-to-csv: command not found
Take it as an example, it would be straight forward to write such a converter. You can on doubt find one out there already.
Yes, IIR giving an example. There are libraries for this in PHP, Python, Perl and Ruby, as well as on-line conversion services.
Useless to me. I can not write that tool. Thus, "no tools"
https://github.com/jehiah/json2csv -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 18:19, Andrei Borzenkov wrote:
20.12.2016 19:41, Carlos E. R. пишет:
Useless to me. I can not write that tool. Thus, "no tools"
Apparently, I need to do: go get github.com/jehiah/json2csv First I have to install "go" (zypper install go). Then I try: cer@Isengard:~> md src cer@Isengard:~> cd src cer@Isengard:~/src> md json2csv cer@Isengard:~/src> cd json2csv cer@Isengard:~/src/json2csv> go get github.com/jehiah/json2csv package github.com/jehiah/json2csv: cannot download, $GOPATH not set. For more details see: go help gopath cer@Isengard:~/src/json2csv> Too complicated, I will not go that route. Removing go. If those fantastic tools are not in oss repo, they do not exist as far as I'm concerned. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhZrM8ACgkQja8UbcUWM1x4ewD9FQUw/HTi8/YXgsT4Se5LE0RP vzq6wRqTXByi2HtPjHwBAJ3jl6UUUH72zSWiMnkIjwkluNVD10NXXf4I1FzZMoE8 =v7Zm -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 11:41 AM, Carlos E. R. wrote:
On 2016-12-20 16:29, Anton Aylward wrote:
Yes, IIR giving an example. There are libraries for this in PHP, Python, Perl and Ruby, as well as on-line conversion services.
Useless to me. I can not write that tool. Thus, "no tools"
so you don't use shell either? you can't download something already written using zypper? Well, how about running a web browser? http://www.convertcsv.com/json-to-csv.htm -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 22:04, Anton Aylward wrote:
On 12/20/2016 11:41 AM, Carlos E. R. wrote:
On 2016-12-20 16:29, Anton Aylward wrote:
Yes, IIR giving an example. There are libraries for this in PHP, Python, Perl and Ruby, as well as on-line conversion services.
Useless to me. I can not write that tool. Thus, "no tools"
so you don't use shell either? you can't download something already written using zypper?
Well, how about running a web browser? http://www.convertcsv.com/json-to-csv.htm
Online tool? No way, no privacy. You are grasping at straws. There are no tools to view directly journal logs in all the complexity. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhZrToACgkQja8UbcUWM1zSagD/aD91M2rE3HKlA9LHf9nBAOp1 K9d4cT9cp4Hvw++SbWMBAIJvtow+YaQwQ5UUxc4Y/dKd7QSI+D6HXsOyxbQNiGEh =Y+SU -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2016-12-20 03:43, L A Walsh wrote:
Anton Aylward wrote:
More and more, logs are are getting to be a means of detection intrusions, hacks. But that means correlating logs, which used to be a tedious and error-prone process. We have in journald the opportunity to see all the activity in one log file, making correlation of events much easier.
--- That was always your choice. If you found multiple log files unsuitable for your purposes there was no one forcing you to keep them that way. They way they were configure was _ONE_ way out of hundreds or thousands of ways you could choose to have your log pre-parsed and pre-stored.
Yes, you can have all syslog entries stored in a single file. The syslog configuration usually contains a commented out entry for /var/log/allmessages.
But he doesn't refer to that one. He refers to other services that use their independent log system, like the login services. Have all services log to the same log service, the journal. Apache, ssh, etc.
I don't know. Without a database viewer, like an automatic import to LibreOffice
Way too inefficient, *Office is not suited for processing large amounts(*) of data. We rotate all logs at midnight, then rsync them to a central archive and run daily analysis during the night. All command line tools. (*) I am assuming large amounts because small amounts don't require any tools, really. -- Per Jessen, Zürich (0.6°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 08:53, Per Jessen wrote:
Carlos E. R. wrote:
I don't know. Without a database viewer, like an automatic import to LibreOffice
Way too inefficient, *Office is not suited for processing large amounts(*) of data. We rotate all logs at midnight, then rsync them to a central archive and run daily analysis during the night. All command line tools.
(*) I am assuming large amounts because small amounts don't require any tools, really.
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages? Without coding. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 09:30 AM, Carlos E. R. wrote:
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
Without coding.
That's being ridiculous. But if you want you could take the JSON output, read it into a MySQL database and use the very power phpMyAdmin took, which has lots of filter and slicing and dicing and querying capabilities. Reality is that OOCalc is just a spreadsheet. Even if you're using it to view syslog generated logs then its not going to as good as a dedicated (aka 'coded' for the job) logfile parser such as can be purchased. Those do import into a MySQL or Oracle database and let you manipulate and query it properly. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 16:44, Anton Aylward wrote:
On 12/20/2016 09:30 AM, Carlos E. R. wrote:
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
Without coding.
That's being ridiculous.
But if you want you could take the JSON output, read it into a MySQL database and use the very power phpMyAdmin took, which has lots of filter and slicing and dicing and querying capabilities.
That is absurd. You are telling me to make my own tools, and I say I want open source tools that directly read the journal and dynamically apply to it point and click functions like filter or follow event. Unless the journal developers create those tools, the journal is irrelevant to me. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 11:49 AM, Carlos E. R. wrote:
That is absurd. You are telling me to make my own tools, and I say I want open source tools that directly read the journal and dynamically apply to it point and click functions like filter or follow event.
Unless the journal developers create those tools, the journal is irrelevant to me.
UNIX, and by implication Linux, has always been about making your own tools and assembling them using pipes and filters and regular expressions and the like. Languages like Perl and Python are really supersets of the shell which directly incorporate some things that were external programs[1], such as using regular expressions in a more general way than just GREP or SED. One paper from the 1980s shows how you can build a relational database using the file system, the shell, grep, sed, cut, paste. UNIX/Linux is pretty much predicated on the idea that you build tools out of these primitives. We take this for granted, but back when it was something revolutionary since the "mainframe" world was all about submitting requests to the IT department to have a program written .... In this sense, MS-DOS and Windows were great steps backwards. All these GUI tools we take for granted are actually scripted. Some, like YAST, are a lot of script, but then there is a lot to YAST. The upside is that it is very modular :-) I've mentioned the GUI interface to MySQL databases that presents a table view but also lets you make SQL queries - phpMyAdmin. That too was put together using a scripting language, PHP. Some scripting languages, Perl, Python, PHP come with compliers. Others, shell, Ruby, have compilers for a limited subset. The moment we use a pipeline, taking a log file, greping it, then using awk or cut to sort out the fields we want to see and pretty-print as a table, or print as a CSV to import into OOCalc, by definition we are programming. In the halcyon days of IBM Mainframes this is the kind thing, generating reports that were filtered various ways, that took a special request of a program to the IT department and MAYBE they would deal with it within the next three months with a FORTRAN or COBOL programme. We do it with a one-liner without thinking much about it, its just built around the grep and a regular expression ... so what? We choose not to call it 'programming' any more that people who do their accounts and budgets and projections and keep track of their investments using a spreadsheet think that entering formula is programming. Some of us are more ambitious than others. Some of us see the boundary between a CLI expression that has a nested loop and goes on for 10 lines as just another thing we type in directly, not even put in a file and mark executable. There are some amazing Perl "one-liners" like that which arose from a competition for the same :-) (Go Google) I'll have to pour through my web history, but I recall seeing a page which had a plug-in for manipulating the journal database directly, I think it was in PHP. I hope not; I'm not familiar with PHP. Hmm, something like this, but for a scripting language like Perl or Python http://www.openlmi.org/provider_journald Perhaps .. https://www.freedesktop.org/software/systemd/python-systemd/journal.html# [1] This happened with the shell as well, it incorporated the external programs 'echo', 'test' -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 22:00, Anton Aylward wrote:
On 12/20/2016 11:49 AM, Carlos E. R. wrote:
That is absurd. You are telling me to make my own tools, and I say I want open source tools that directly read the journal and dynamically apply to it point and click functions like filter or follow event.
Unless the journal developers create those tools, the journal is irrelevant to me.
UNIX, and by implication Linux, has always been about making your own tools and assembling them using pipes and filters and regular expressions and the like.
No, I will not accept that argument. The truth is, those fantastic tools to analyze the journal do not exist, you have to create them yourself. Well, no, I prefer to keep using syslog with the current tools. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhZrcQACgkQja8UbcUWM1wUlQD/VhpOsWWlcyJHhN6r7LUXnnx/ 4vZJHfEHS68BrANrcnYA/ir2qZgnh2M9T4QDZtkeXmkHCX1W966hpa3UYD1W9liO =/xjO -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 20/12/2016 à 23:16, Carlos E. R. a écrit :
yourself. Well, no, I prefer to keep using syslog with the current tools.
ypu said the journald dev didn't azccept, do you have a pointer to the discussion? did you ask leafnode dev? thanks jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-21 08:04, jdd wrote:
Le 20/12/2016 à 23:16, Carlos E. R. a écrit :
yourself. Well, no, I prefer to keep using syslog with the current tools.
ypu said the journald dev didn't azccept, do you have a pointer to the discussion?
No. Andrei mentioned this fact on another post, so perhaps he has it.
did you ask leafnode dev?
Why? Leafnode is doing the correct thing. For decades. It is the newcomer which doesn't, IMO. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhaV5IACgkQja8UbcUWM1zjFwD/W7l+d7b04k9waSVkOjEfnEgg YX2MlkyjZzqrWbPl7E8A/1EGcXKtD8I+RZRXU3+9sFZwf4DPpiOGPLYU7kySqbfJ =bKUs -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
UNIX, and by implication Linux, has always been about making your own tools and assembling them using pipes and filters and regular expressions and the like.
---- Except that the new log format isn't something easily parsed by all the text tools on Unix/Linux. It went binary. Text tools don't do binary very well if at all. A log format that turns its back on the *nix toolset by going binary needs some tool set other than the *nix one. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 02:08, L A Walsh wrote:
Anton Aylward wrote:
UNIX, and by implication Linux, has always been about making your own tools and assembling them using pipes and filters and regular expressions and the like.
---- Except that the new log format isn't something easily parsed by all the text tools on Unix/Linux. It went binary. Text tools don't do binary very well if at all.
A log format that turns its back on the *nix toolset by going binary needs some tool set other than the *nix one.
That's right. It needs new tools that have not being designed. Yes, you can build your own, apparently. Not good. Or you can convert to text and use traditional tools on that text. Then there is no advantage... - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhbKwoACgkQja8UbcUWM1y9uwEAj3DEg2Xqf3snOTaWFjKYU2/D d5z6f1FSMxsiXglA03YBAIY59gLxCkdedQQdR4OjgWiZiJmDpFHPbj4pWFT44RQ6 =ygpo -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Dec 20, 2016 at 10:44 AM, Anton Aylward
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
Without coding.
That's being ridiculous.
Leap 42.2 and factory have prewikka. I think prewikka can be used for advanced log viewing. I know it works with syslog. I don't know about journald logs. Anyone used prewikka? Anyone know if it can pull in logs for journald Thanks Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 21:08, Greg Freemyer wrote:
Leap 42.2 and factory have prewikka. I think prewikka can be used for advanced log viewing.
I know it works with syslog. I don't know about journald logs.
Anyone used prewikka? Anyone know if it can pull in logs for journald
Seems to be too complex. I installed it, could not start it. Requires being root. And: cer@Isengard:~> rpm -ql prewikka | grep bin/ /usr/sbin/prewikka-httpd cer@Isengard:~> it appears to be a daemon. Removing. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhZr/MACgkQja8UbcUWM1x+fQD/WVNrhgPP4iBFrpU8A6PObPoS dPw6VTTj/5GLU7w9/m0BAIaJgNvsau9+pfDTbxY2A2mhnfkX2CEYO68wmvLaSlKk =AbGR -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
20.12.2016 23:08, Greg Freemyer пишет:
On Tue, Dec 20, 2016 at 10:44 AM, Anton Aylward
wrote: I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
Without coding.
That's being ridiculous.
Leap 42.2 and factory have prewikka. I think prewikka can be used for advanced log viewing.
Is it not a frontend to prelude? Just to be sure we mean the same thing?
I know it works with syslog. I don't know about journald logs.
Anyone used prewikka? Anyone know if it can pull in logs for journald
If this is indeed frontend to prelude, we need to look for prelude driver (or is it called sensor) for journald. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Dec 20, 2016 at 10:44 PM, Andrei Borzenkov
20.12.2016 23:08, Greg Freemyer пишет:
On Tue, Dec 20, 2016 at 10:44 AM, Anton Aylward
wrote: I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
Without coding.
That's being ridiculous.
Leap 42.2 and factory have prewikka. I think prewikka can be used for advanced log viewing.
Is it not a frontend to prelude? Just to be sure we mean the same thing?
The prelude SIEM user interface, yes.
I know it works with syslog. I don't know about journald logs.
Anyone used prewikka? Anyone know if it can pull in logs for journald
If this is indeed frontend to prelude, we need to look for prelude driver (or is it called sensor) for journald.
Agreed. prelude-lml is the syslog sensor. https://www.prelude-siem.org/projects/prelude/wiki/PreludeLml I don't know if it can also handle journald. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
to be objective and constructive: 1- could someone articulate clearly what we want to achieve. (except journal selective editing) 2- there seem to be rants about viewing, filtering, template-like alerts?, forwarding logs to server? whats the use case of oocalc? 3- ALL the tools are already available, the code would be TRIVIAL. (from a quick view prelude etc look convoluted) 4- there is already a (toy) journal viewer in yast, if this is the kind of thing wanted, what is missing? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-21 08:03, nicholas cunliffe wrote:
to be objective and constructive: 1- could someone articulate clearly what we want to achieve. (except journal selective editing) 2- there seem to be rants about viewing, filtering, template-like alerts?, forwarding logs to server? whats the use case of oocalc?
calc can import databases, and there is also a database component on LO/OO. With that you can read directly the syslog mysql database as generated by modern daemons. Once displayed in LO, you can apply filters, sorts, the lot of things you can do in a database with a powerful database viewer.
3- ALL the tools are already available, the code would be TRIVIAL. (from a quick view prelude etc look convoluted)
I'm reading about that. As I'm no longer being payed to examine logs, I will not develop code to inspect the journal, or invest much time on learning complex tools, but I'm interested in knowing if the tools exist. Tools ready made, not code to be done.
4- there is already a (toy) journal viewer in yast, if this is the kind of thing wanted, what is missing?
It seems a text viewer, does not separate fields in columns, no sort. I did not try the filter. It can not follow trails. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhaWjoACgkQja8UbcUWM1yDOQEAg45tG9Gn02dJG3K0DflJ2SzV ZEeGFWVQmzsYeIjNlscA/juh98hIKc4hfEBrSMy5sA1BhQag7YoHCPXIeYHX03wE =ANz8 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Dec 20, 2016 at 10:44 AM, Anton Aylward
On 12/20/2016 09:30 AM, Carlos E. R. wrote:
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
Without coding.
That's being ridiculous.
But if you want you could take the JSON output, read it into a MySQL database and use the very power phpMyAdmin took, which has lots of filter and slicing and dicing and querying capabilities.
Reality is that OOCalc is just a spreadsheet. Even if you're using it to view syslog generated logs then its not going to as good as a dedicated (aka 'coded' for the job) logfile parser such as can be purchased. Those do import into a MySQL or Oracle database and let you manipulate and query it properly.
We have good databases in openSUSE. We have at least one log aggregator in openSUSE (libprelude). We have at least 2 log correlation engines in openSUSE (Prelude SIEM and Suricata (only in OBS)). Again, I have no first hand experience, but it is time for some experimenting I do believe. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 21:11, Greg Freemyer wrote:
We have good databases in openSUSE.
We have at least one log aggregator in openSUSE (libprelude).
We have at least 2 log correlation engines in openSUSE (Prelude SIEM and Suricata (only in OBS)).
Again, I have no first hand experience, but it is time for some experimenting I do believe.
If I were still paid to do log analysis, yes, I would. As it is, I do not see those fantastic tools to see the journal, which would be the main justification for having binary logs. The tools have not been developed, you have to write your own. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhZsLAACgkQja8UbcUWM1wXZAD/T+gZ1OK5MQX+csJCzFCpqtEs v3S8oQQbY9NCxWBqacAA/3CY9PhNCX1th8xQjHh5YVWoKjvPqLbukH1bA5FwyP7p =drHG -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2016-12-20 08:53, Per Jessen wrote:
Carlos E. R. wrote:
I don't know. Without a database viewer, like an automatic import to LibreOffice
Way too inefficient, *Office is not suited for processing large amounts(*) of data. We rotate all logs at midnight, then rsync them to a central archive and run daily analysis during the night. All command line tools.
(*) I am assuming large amounts because small amounts don't require any tools, really.
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
We don't use the journal, it doesn't add anything for us. I thought there were such options though. -- Per Jessen, Zürich (1.6°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2016-12-20 08:53, Per Jessen wrote:
Carlos E. R. wrote:
I don't know. Without a database viewer, like an automatic import to LibreOffice
Way too inefficient, *Office is not suited for processing large amounts(*) of data. We rotate all logs at midnight, then rsync them to a central archive and run daily analysis during the night. All command line tools.
(*) I am assuming large amounts because small amounts don't require any tools, really.
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
grep -v ? -- Per Jessen, Zürich (1.6°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-20 17:00, Per Jessen wrote:
Carlos E. R. wrote:
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
grep -v ?
Of course, that's what I use, but on syslog, not on the journal. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2016-12-20 17:00, Per Jessen wrote:
Carlos E. R. wrote:
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such. Do you know some viewer for the journal, where I can, say, filter out of display all nntp messages?
grep -v ?
Of course, that's what I use, but on syslog, not on the journal.
It'll work on the journal too. -- Per Jessen, Zürich (0.9°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-20 21:03, Per Jessen wrote:
Carlos E. R. wrote:
grep -v ?
Of course, that's what I use, but on syslog, not on the journal.
It'll work on the journal too.
Of course. But first you have to expand it, and this operation can take minutes on rotating rust. And then there is the problem that due to the debug messages in the logs, the journal for a year can be really huge, so I don't keep it. In this machine I disabled completely the journal: minas-tirith:~ # journalctl No journal files were found. minas-tirith:~ # /etc/systemd/journald.conf: [Journal] #CER Storage=none <========== SystemMaxUse=100M RuntimeMaxUse=30M MaxLevelStore=notice This works with rsyslog, but not with syslog-ng, I believe. No duplication of space, no time spent writing to disk twice the same messages. I did it as a test, but I like it, I see no problem. Well, that "systemctl status" will not show log entries - but it wouldn't anyway: with the volume of debug messages in my system, they get rotated out pretty soon. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhZtAAACgkQja8UbcUWM1yzSAD/RWMnyM/n1i+db85s6VGgZGIS J3QSQu/U2eB0JzvtwE0BAKC9vKbUBbx1bwtSzZRVZo3bIG/uUv1JKS3s16pOp9m7 =cNcg -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2016-12-20 21:03, Per Jessen wrote:
Carlos E. R. wrote:
grep -v ?
Of course, that's what I use, but on syslog, not on the journal.
It'll work on the journal too.
Of course. But first you have to expand it, and this operation can take minutes on rotating rust.
Apparently yes, but the solution works nonetheless. As for a log viewer - I don't have a need myself, but I feel certain there are tools out there that will let you filter adhoc, colourise lines etc. Not a text-processor like *office, a log-processor. I don't think they belong as part of the journal though. I googled it - how about "colourtail", "multitail", "grc" or "view" (with vi syntax-highlighting) ?
And then there is the problem that due to the debug messages in the logs, the journal for a year can be really huge, so I don't keep it.
Yes, I do see that as a problem, especially when other messages are discarded when it grows too big. Same thing happens with the kernel ring buffer - firewall logging will often make it wrap. I'm implementing ulog, I just haven't quite got it working yet. If you have or want these copious amounts of logging and you want to use the journal, I see no other solution but making the space available. Or maybe direct those debug logs to disk?
In this machine I disabled completely the journal: This works with rsyslog, but not with syslog-ng, I believe.
I expect it would work with both, I'm sure they use the same interface. Hmm, I have just changed to storage=none on a test system, and syslog-ng does indeed not work. Interesting, I wonder why that is. Well, not a real problem, we use the default journal setup anyway. -- Per Jessen, Zürich (0.9°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Dec 21, 2016 at 10:22 AM, Per Jessen
In this machine I disabled completely the journal: This works with rsyslog, but not with syslog-ng, I believe.
I expect it would work with both, I'm sure they use the same interface.
Hmm, I have just changed to storage=none on a test system, and syslog-ng does indeed not work. Interesting, I wonder why that is.
Default configuration of syslog-ng is using system() source and system() source on Linux adds systemd-journal() source if it detect that it runs under systemd. systemd-journal() source actively pulls log entries from journal files. "storage=none" means no journal files - no log entries for syslog-ng source as well. Actually I'm rather puzzled why rsyslog works for you at all :) On test TW installation I do not see any journal-specific part in rsyslog configuration and ForwardToSyslog apparently defaults to "no" so rsyslog should not receive data even via socket ... indeed, after installing and starting rsyslog I do not get any test message via it. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-21 10:04, Andrei Borzenkov wrote:
On Wed, Dec 21, 2016 at 10:22 AM, Per Jessen
wrote: In this machine I disabled completely the journal: This works with rsyslog, but not with syslog-ng, I believe.
I expect it would work with both, I'm sure they use the same interface.
Hmm, I have just changed to storage=none on a test system, and syslog-ng does indeed not work. Interesting, I wonder why that is.
It reads and tracks the files.
Default configuration of syslog-ng is using system() source and system() source on Linux adds systemd-journal() source if it detect that it runs under systemd. systemd-journal() source actively pulls log entries from journal files. "storage=none" means no journal files - no log entries for syslog-ng source as well.
Actually I'm rather puzzled why rsyslog works for you at all :) On test TW installation I do not see any journal-specific part in rsyslog configuration and ForwardToSyslog apparently defaults to "no" so rsyslog should not receive data even via socket ... indeed, after installing and starting rsyslog I do not get any test message via it.
You need to have both initially, journal and rsyslog. Once rsyslog is working, disable storage on journal. This works on 13.1 at least, I have not tested in recent versions. It works because it doesn't pull the messages from the files. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhaW18ACgkQja8UbcUWM1zdOwEAg7opIfQloPVrQi7NLVmnSGyM nSxAWlfzcttOBGRhA84A/3y86m2gu9JpyAHcEvRN5tXQJYjhaZ+DmKZCWU+fzj00 =4pNw -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-21 10:04, Andrei Borzenkov wrote:
On Wed, Dec 21, 2016 at 10:22 AM, Per Jessen <> wrote:
In this machine I disabled completely the journal: This works with rsyslog, but not with syslog-ng, I believe.
I expect it would work with both, I'm sure they use the same interface.
Hmm, I have just changed to storage=none on a test system, and syslog-ng does indeed not work. Interesting, I wonder why that is.
Default configuration of syslog-ng is using system() source and system() source on Linux adds systemd-journal() source if it detect that it runs under systemd. systemd-journal() source actively pulls log entries from journal files. "storage=none" means no journal files - no log entries for syslog-ng source as well.
Actually I'm rather puzzled why rsyslog works for you at all :) On test TW installation I do not see any journal-specific part in rsyslog configuration and ForwardToSyslog apparently defaults to "no" so rsyslog should not receive data even via socket ... indeed, after installing and starting rsyslog I do not get any test message via it.
Well, quick test: I rebooted to 42.2 and applied the change. Firstly I installed rsyslog, enabled it (with some problems), checked that it was getting messages, then configured storage=none in journal. Rebooted, and syslog was still getting messages. :-) -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-22 14:56, Carlos E. R. wrote:
On 2016-12-21 10:04, Andrei Borzenkov wrote:
On Wed, Dec 21, 2016 at 10:22 AM, Per Jessen <> wrote:
In this machine I disabled completely the journal: This works with rsyslog, but not with syslog-ng, I believe.
I expect it would work with both, I'm sure they use the same interface.
Hmm, I have just changed to storage=none on a test system, and syslog-ng does indeed not work. Interesting, I wonder why that is.
Default configuration of syslog-ng is using system() source and system() source on Linux adds systemd-journal() source if it detect that it runs under systemd. systemd-journal() source actively pulls log entries from journal files. "storage=none" means no journal files - no log entries for syslog-ng source as well.
Actually I'm rather puzzled why rsyslog works for you at all :) On test TW installation I do not see any journal-specific part in rsyslog configuration and ForwardToSyslog apparently defaults to "no" so rsyslog should not receive data even via socket ... indeed, after installing and starting rsyslog I do not get any test message via it.
Well, quick test: I rebooted to 42.2 and applied the change. Firstly I installed rsyslog, enabled it (with some problems), checked that it was getting messages, then configured storage=none in journal. Rebooted, and syslog was still getting messages.
:-)
Journal says: cer@Minas-Anor:~> journalctl --follow -- Logs begin at Sun 2016-12-11 17:25:41 CET. -- Dec 22 14:06:34 Minas-Anor systemd[1]: Stopped Create Static Device Nodes in /dev. Dec 22 14:06:34 Minas-Anor systemd[1]: Stopped Create list of required static device nodes for the current kernel. Dec 22 14:06:34 Minas-Anor systemd[1]: Closed udev Kernel Socket. Dec 22 14:06:34 Minas-Anor systemd[1]: Closed udev Control Socket. Dec 22 14:06:34 Minas-Anor systemd[1]: Starting Cleanup udevd DB... Dec 22 14:06:34 Minas-Anor systemd[1]: Started Cleanup udevd DB. Dec 22 14:06:34 Minas-Anor systemd[1]: Reached target Switch Root. Dec 22 14:06:34 Minas-Anor systemd[1]: Starting Switch Root... Dec 22 14:06:34 Minas-Anor systemd[1]: Switching root. Dec 22 14:06:34 Minas-Anor systemd-journald[116]: Journal stopped and syslog says: 2016-12-22T14:01:16.180279+01:00 Minas-Anor rsyslogd: [origin software="rsyslogd" swVersion="8.4.0" x-pid="3986" x-info="http://www.rsyslog.com"] start 2016-12-22T14:01:16.181997+01:00 Minas-Anor systemd[1]: Listening on Syslog Socket. 2016-12-22T14:01:16.182028+01:00 Minas-Anor systemd[1]: Starting System Logging Service... 2016-12-22T14:01:16.182438+01:00 Minas-Anor systemd[1]: Started System Logging Service. 2016-12-22T14:01:16.184725+01:00 Minas-Anor kernel: [ 0.000000] microcode: CPU0 microcode updated early to revision 0xa0b, date = 2010-09-28 2016-12-22T14:01:16.184737+01:00 Minas-Anor kernel: [ 0.000000] Initializing cgroup subsys cpuset 2016-12-22T14:01:16.184739+01:00 Minas-Anor kernel: [ 0.000000] Initializing cgroup subsys cpu 2016-12-22T14:01:16.184741+01:00 Minas-Anor kernel: [ 0.000000] Initializing cgroup subsys cpuacct So it captured the current boot or close to it. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Dec 22, 2016 at 5:39 PM, Carlos E. R.
and syslog says:
2016-12-22T14:01:16.180279+01:00 Minas-Anor rsyslogd: [origin software="rsyslogd" swVersion="8.4.0" x-pid="3986" x-info="http://www.rsyslog.com"] start 2016-12-22T14:01:16.181997+01:00 Minas-Anor systemd[1]: Listening on Syslog Socket. 2016-12-22T14:01:16.182028+01:00 Minas-Anor systemd[1]: Starting System Logging Service... 2016-12-22T14:01:16.182438+01:00 Minas-Anor systemd[1]: Started System Logging Service. 2016-12-22T14:01:16.184725+01:00 Minas-Anor kernel: [ 0.000000] microcode: CPU0 microcode updated early to revision 0xa0b, date = 2010-09-28 2016-12-22T14:01:16.184737+01:00 Minas-Anor kernel: [ 0.000000] Initializing cgroup subsys cpuset 2016-12-22T14:01:16.184739+01:00 Minas-Anor kernel: [ 0.000000] Initializing cgroup subsys cpu 2016-12-22T14:01:16.184741+01:00 Minas-Anor kernel: [ 0.000000] Initializing cgroup subsys cpuacct
So it captured the current boot or close to it.
That's only kernel messages that it likely gets directly from kmsg. Try logger - does it appear in /var/log/messages? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-22 15:55, Andrei Borzenkov wrote:
On Thu, Dec 22, 2016 at 5:39 PM, Carlos E. R. <> wrote:
So it captured the current boot or close to it.
That's only kernel messages that it likely gets directly from kmsg. Try logger - does it appear in /var/log/messages?
Sure :-) cer@Minas-Anor:~> logger hello ^C cer@Minas-Anor:~> 2016-12-22T17:32:36.704131+01:00 Minas-Anor dbus[821]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' 2016-12-22T17:32:36.704728+01:00 Minas-Anor systemd[1]: Started Network Manager Script Dispatcher Service. 2016-12-22T17:33:14.696189+01:00 Minas-Anor cer: hello -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
22.12.2016 19:34, Carlos E. R. пишет:
On 2016-12-22 15:55, Andrei Borzenkov wrote:
On Thu, Dec 22, 2016 at 5:39 PM, Carlos E. R. <> wrote:
So it captured the current boot or close to it.
That's only kernel messages that it likely gets directly from kmsg. Try logger - does it appear in /var/log/messages?
Sure :-)
cer@Minas-Anor:~> logger hello ^C cer@Minas-Anor:~>
2016-12-22T17:32:36.704131+01:00 Minas-Anor dbus[821]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' 2016-12-22T17:32:36.704728+01:00 Minas-Anor systemd[1]: Started Network Manager Script Dispatcher Service. 2016-12-22T17:33:14.696189+01:00 Minas-Anor cer: hello
Yes, on Leap 42.2 ForwardToSylog default to "yes". On TW it is disabled. TW actually follows upstream default, although I could not find any changelog describing it. Arguably default rsyslog configuration on TW should then include journald module. Anyone cares enough to open bug report? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 18:19, Andrei Borzenkov wrote:
22.12.2016 19:34, Carlos E. R. пишет:
On 2016-12-22 15:55, Andrei Borzenkov wrote:
On Thu, Dec 22, 2016 at 5:39 PM, Carlos E. R. <> wrote:
So it captured the current boot or close to it.
That's only kernel messages that it likely gets directly from kmsg. Try logger - does it appear in /var/log/messages?
Sure :-)
cer@Minas-Anor:~> logger hello ^C cer@Minas-Anor:~>
2016-12-22T17:32:36.704131+01:00 Minas-Anor dbus[821]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' 2016-12-22T17:32:36.704728+01:00 Minas-Anor systemd[1]: Started Network Manager Script Dispatcher Service. 2016-12-22T17:33:14.696189+01:00 Minas-Anor cer: hello
Yes, on Leap 42.2 ForwardToSylog default to "yes". On TW it is disabled. TW actually follows upstream default, although I could not find any changelog describing it.
Arguably default rsyslog configuration on TW should then include journald module. Anyone cares enough to open bug report?
I don't use TW... It will have to be you ;-) I can report it, but I can not easily run tests. Only on an old virtual install. I'd have first to try reproduce the issue. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdCikACgkQja8UbcUWM1zQQwEAiUPbR8RWkxzEvA5FHn49ZdKk vHpLaGSZt1l7W6ZOxLcA/0bg/btgLUxGcuegIfjoHBxb8zMtVr7g2m0fef56Rvng =LkxY -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/20/2016 11:22 PM, Per Jessen wrote:
And then there is the problem that due to the debug messages in the logs, the journal for a year can be really huge, so I don't keep it.
Yes, I do see that as a problem, especially when other messages are discarded when it grows too big. Same thing happens with the kernel ring buffer - firewall logging will often make it wrap. I'm implementing ulog, I just haven't quite got it working yet.
The solution here is to set the logging level in journald.conf to avoid most of the debug logs. man journald.conf But also, someone with some influence over the KDE developers needs to point out to those guys they they simply MUST STOP sending every useless debug message out with "warning" or higher severity levels. They (KDE devs) are by far the most egregious offenders, and I have actually had some email exchanges directly with these developers trying to convince them of their folly, and had exactly zero success. The've never listened to end users on any subject whatsoever, but they might listen to Opensuse. If the End user can't do anything about the cause of the message then the message should be set absolutely no higher than debug level. Side note: Wrapping logs is NOT a journald specific problem. It also plagued old school logs. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 21.12.2016 19:50, John Andersen wrote:
But also, someone with some influence over the KDE developers needs to point out to those guys they they simply MUST STOP sending every useless debug message out with "warning" or higher severity levels.
They (KDE devs) are by far the most egregious offenders, and I have actually had some email exchanges directly with these developers trying to convince them of their folly, and had exactly zero success. The've never listened to end users on any subject whatsoever, but they might listen to Opensuse.
I could only find https://bugs.kde.org/show_bug.cgi?id=368357, which was actually caused by a version mismatch on your system. Could you please list the other bug numbers in bugs.kde.org? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-21 19:50, John Andersen wrote:
On 12/20/2016 11:22 PM, Per Jessen wrote:
And then there is the problem that due to the debug messages in the logs, the journal for a year can be really huge, so I don't keep it.
Yes, I do see that as a problem, especially when other messages are discarded when it grows too big. Same thing happens with the kernel ring buffer - firewall logging will often make it wrap. I'm implementing ulog, I just haven't quite got it working yet.
The solution here is to set the logging level in journald.conf to avoid most of the debug logs. man journald.conf
But also, someone with some influence over the KDE developers needs to point out to those guys they they simply MUST STOP sending every useless debug message out with "warning" or higher severity levels.
Good grief, yes, and not only them. I have pages of filters on rsyslog to purge them out. There are no filters to journal...
They (KDE devs) are by far the most egregious offenders, and I have actually had some email exchanges directly with these developers trying to convince them of their folly, and had exactly zero success. The've never listened to end users on any subject whatsoever, but they might listen to Opensuse.
If the End user can't do anything about the cause of the message then the message should be set absolutely no higher than debug level.
Yep. I complained or asked about those messages (not kde) and got zero response, too.
Side note: Wrapping logs is NOT a journald specific problem. It also plagued old school logs.
But not syslog types. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhbLBwACgkQja8UbcUWM1zBCwD+KUPIalTLHJ+oSQl7g2C2ombW NzehJNbHHcLdglmdn00A/RPjk4bLgFkf2AjFtwTlWtmarwzBSORJO3ul5UfYjMlq =ILga -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-21 08:22, Per Jessen wrote:
Carlos E. R. wrote:
grep -v ?
Of course, that's what I use, but on syslog, not on the journal.
It'll work on the journal too.
Of course. But first you have to expand it, and this operation can take minutes on rotating rust.
Apparently yes, but the solution works nonetheless.
If that's what I must use, I prefer working on syslog than on journal.
As for a log viewer - I don't have a need myself, but I feel certain there are tools out there that will let you filter adhoc, colourise lines etc. Not a text-processor like *office,
I did not suggest to use a text processor, but a database or a table processor.
a log-processor. I don't think they belong as part of the journal though.
I googled it - how about "colourtail", "multitail", "grc" or "view" (with vi syntax-highlighting) ?
I do not know. I may try them, time permitting. :-)
And then there is the problem that due to the debug messages in the logs, the journal for a year can be really huge, so I don't keep it.
Yes, I do see that as a problem, especially when other messages are discarded when it grows too big. Same thing happens with the kernel ring buffer - firewall logging will often make it wrap. I'm implementing ulog, I just haven't quite got it working yet. If you have or want these copious amounts of logging and you want to use the journal, I see no other solution but making the space available. Or maybe direct those debug logs to disk?
My solution is to use rsyslog, not journal. Problem solved.
In this machine I disabled completely the journal: This works with rsyslog, but not with syslog-ng, I believe.
I expect it would work with both, I'm sure they use the same interface.
No, they don't. At least, not on 13.1
Hmm, I have just changed to storage=none on a test system, and syslog-ng does indeed not work. Interesting, I wonder why that is. Well, not a real problem, we use the default journal setup anyway.
And Andrei said that it does not work for him on TW with rsyslog. I want to know if this is a change somewhere that makes my trick impossible, or some config error that can be solved. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhbLW8ACgkQja8UbcUWM1wFEAD+IPaZ8CFnf4aFz1Qk/KZzpV9/ 2/ODnHhvYAtMTe09nbAA/R/2Ffls3y6+cFNB5GGxl7pUCm86afYDoa+7gyMD+QMR =AbWQ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2016-12-21 08:22, Per Jessen wrote:
Carlos E. R. wrote:
grep -v ?
Of course, that's what I use, but on syslog, not on the journal.
It'll work on the journal too.
Of course. But first you have to expand it, and this operation can take minutes on rotating rust.
Apparently yes, but the solution works nonetheless.
If that's what I must use, I prefer working on syslog than on journal.
Ditto.
As for a log viewer - I don't have a need myself, but I feel certain there are tools out there that will let you filter adhoc, colourise lines etc. Not a text-processor like *office,
I did not suggest to use a text processor, but a database or a table processor.
I thought you mentioned *office, that's a text-processor in my book. It's basically not meant for handling tens of thousands of lines.
Hmm, I have just changed to storage=none on a test system, and syslog-ng does indeed not work. Interesting, I wonder why that is. Well, not a real problem, we use the default journal setup anyway.
And Andrei said that it does not work for him on TW with rsyslog. want to know if this is a change somewhere that makes my trick impossible, or some config error that can be solved.
There's something for you to do over the Xmas holidays. :-) -- Per Jessen, Zürich (-0.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 08:36, Per Jessen wrote:
Carlos E. R. wrote:
I did not suggest to use a text processor, but a database or a table processor.
I thought you mentioned *office, that's a text-processor in my book. It's basically not meant for handling tens of thousands of lines.
No no. Libre Office is a suite, has different programs. One is the text processor, but another is a calc program, similar to Excel, which is capable of importing databases into a table. Another is "lobase", a database eumm... processor.
Hmm, I have just changed to storage=none on a test system, and syslog-ng does indeed not work. Interesting, I wonder why that is. Well, not a real problem, we use the default journal setup anyway.
And Andrei said that it does not work for him on TW with rsyslog. want to know if this is a change somewhere that makes my trick impossible, or some config error that can be solved.
There's something for you to do over the Xmas holidays. :-)
Yes, but I need access to my main computer, to create virtual machines into which experiment. On the laptop, 42.2 means rebooting to the test partition. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhbyI8ACgkQja8UbcUWM1ypTQEAh1vlgGLBR7YFxR+11qG/+oBQ UGxaYCtPx0yrPFoKUQ8A/3xeyFGQbbDIdN3/P70uPU0YHcx2gc29dkf4VRXK02I8 =9z09 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wednesday, 21 December 2016 08:22:20 GMT Per Jessen wrote:
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2016-12-20 21:03, Per Jessen wrote:
Carlos E. R. wrote:
grep -v ?
Of course, that's what I use, but on syslog, not on the journal.
It'll work on the journal too.
Of course. But first you have to expand it, and this operation can take minutes on rotating rust.
Apparently yes, but the solution works nonetheless. As for a log viewer - I don't have a need myself, but I feel certain there are tools out there that will let you filter adhoc, colourise lines etc. Not a text-processor like *office, a log-processor. I don't think they belong as part of the journal though.
I googled it - how about "colourtail", "multitail", "grc" or "view" (with vi syntax-highlighting) ?
And then there is the problem that due to the debug messages in the logs, the journal for a year can be really huge, so I don't keep it.
Yes, I do see that as a problem, especially when other messages are discarded when it grows too big. Same thing happens with the kernel ring buffer - firewall logging will often make it wrap. I'm implementing ulog, I just haven't quite got it working yet. If you have or want these copious amounts of logging and you want to use the journal, I see no other solution but making the space available. Or> maybe direct those debug logs to disk?
I'm obviously not an admin but i'm interested in expanding my knowledge so you'll have to excuse my ignorance/inexperience and as the thread has got long, i might have forgotten or missed an explanation. I think the journal keeping all the records is the only way, sure its storage heavy but it'll have to be done. I don't see how you can prove that your logs are secure i.e. not tampered with (by outsiders or insiders), to an independent reviewer if you do not keep every single record. But if you are in an industry that does not legally mandate keeping every record then i guess you can do what you like with the data as its irrelevant to anyone else until you are trying to find a rogue employee fiddling with the system.
In this machine I disabled completely the journal: This works with rsyslog, but not with syslog-ng, I believe.
I expect it would work with both, I'm sure they use the same interface.
Hmm, I have just changed to storage=none on a test system, and syslog-ng does indeed not work. Interesting, I wonder why that is. Well, not a real problem, we use the default journal setup anyway.
-- opensuse:tumbleweed:20161219 Qt: 5.7.0 KDE Frameworks: 5.29.0 KDE Plasma: 5.8.4 kwin5-5.8.4-171.1.x86_64 Kernel: 4.8.14-1-default Nouveau: 1.0.13_2.1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
ianseeks wrote:
On Wednesday, 21 December 2016 08:22:20 GMT Per Jessen wrote:
Yes, I do see that as a problem, especially when other messages are discarded when it grows too big. Same thing happens with the kernel ring buffer - firewall logging will often make it wrap. I'm implementing ulog, I just haven't quite got it working yet. If you have or want these copious amounts of logging and you want to use the journal, I see no other solution but making the space available. Or> maybe direct those debug logs to disk?
I'm obviously not an admin but i'm interested in expanding my knowledge so you'll have to excuse my ignorance/inexperience and as the thread has got long, i might have forgotten or missed an explanation. I think the journal keeping all the records is the only way, sure its storage heavy but it'll have to be done.
I would tend to agree. If you want to use journal in a situation like Carlos', just make sure there is plenty of space. Disk is cheap.
I don't see how you can prove that your logs are secure i.e. not tampered with (by outsiders or insiders), to an independent reviewer if you do not keep every single record.
Even if you do not filter the logs, you'll have trouble proving that. Non-repudiation is an entirely different topic. -- Per Jessen, Zürich (-0.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-22 08:45, Per Jessen wrote:
ianseeks wrote:
On Wednesday, 21 December 2016 08:22:20 GMT Per Jessen wrote:
I'm obviously not an admin but i'm interested in expanding my knowledge so you'll have to excuse my ignorance/inexperience and as the thread has got long, i might have forgotten or missed an explanation. I think the journal keeping all the records is the only way, sure its storage heavy but it'll have to be done.
I would tend to agree. If you want to use journal in a situation like Carlos', just make sure there is plenty of space. Disk is cheap.
Is it reliable? Can we trust the journal to work and not loose anything in, say, two years time, even after upgrades?
I don't see how you can prove that your logs are secure i.e. not tampered with (by outsiders or insiders), to an independent reviewer if you do not keep every single record.
Even if you do not filter the logs, you'll have trouble proving that. Non-repudiation is an entirely different topic.
rsyslog has a cryptographic extension or module. And it can store the logs in biinary format (mysql, for instance), so instantly readable by database viewers. -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/22/2016 09:07 AM, Carlos E. R. wrote:
Is it reliable? Can we trust the journal to work and not loose anything in, say, two years time, even after upgrades?
You've descended into the stage of a ridiculous argument, Carlos. Once you start hypothesising that something can be unreliable you can hypothesise that anything and everything can be unrealisable up to an including the kernel. After all its designed by humans, implemented by humans, coded by humans, tested by humans, and we've seen bugs that have defied 'many eyes' for years, haven't we. Humans are fallible. Even the automated stuff, well tat was designed and coded by humans too, so it could have errors that propagate into the 'machine-designed code'. What's to say that there isn't a syndromic bug in ng-syslog that only emerges under certain conditions by causes a runaway write that consumes all disk space? I can hypothesise that just as easily as you can hypothesise a hypothetical about the result of an upgrade to to the journal in two years. Or four years. Or ten years. You've descended into the stage of a ridiculous argument, Carlos. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-22 15:19, Anton Aylward wrote:
On 12/22/2016 09:07 AM, Carlos E. R. wrote:
Is it reliable? Can we trust the journal to work and not loose anything in, say, two years time, even after upgrades?
You've descended into the stage of a ridiculous argument, Carlos.
I don't think so. A binary store is prone to breakage, it has happened. If you think that way, I'll think then that you are a zealot that negates any possible problem. It is a reasonable question. Will I be able to read it in 2 years time, in any machine? -- Cheers/Saludos Carlos E. R. (testing openSUSE Leap 42.2, at Minas-Anor) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/22/2016 11:29 AM, Carlos E. R. wrote:
On 2016-12-22 15:19, Anton Aylward wrote:
On 12/22/2016 09:07 AM, Carlos E. R. wrote:
Is it reliable? Can we trust the journal to work and not loose anything in, say, two years time, even after upgrades?
You've descended into the stage of a ridiculous argument, Carlos.
I don't think so. A binary store is prone to breakage, it has happened. If you think that way, I'll think then that you are a zealot that negates any possible problem.
It is a reasonable question. Will I be able to read it in 2 years time, in any machine
I say again: You've descended into the stage of a ridiculous argument, Carlos The moment you start saying "any machine" you bring in a gunnysack of things like transmission protocols and that means the possibility of transmission errors, which I've touched on before. Different machine architectures, byte ordering and more. The belief that text or rather ACII-7 is a cure for this is misguided. that's why we have hamming codes, Reed-Sullivan codes, even for thins that we think of as plain text. The suite we refer to as "TCP Architecture" deal with the byte ordering problem a long time ago. We have as reliable binary transmission as we choose to pay for, as evidenced by downloading of DVDs or network based installs. We have transmission between machines of different architecture, word size, byte ordering. I can download the image for a RasberryPi that runs a version of Linux on a ARM with a 32 bit architecture on an old 16-bit Wndows/95 PC and burn it onto a card that's on a carrier plugged into a USB2 socket all quite compatibly. This isn't the 1960s. The really issue isn't the data or even the program, its the hardware. I have UNIX V6 and V7 magnetic tapes that were written with TAR. I _could_ read that data and run those programs, if I had a tape unit and PDP-11. The problem isn't the program, modern TAR can still read those old tapes. The problem is having the hardware to do anything with it. Oh, there's one exception to this: its legal and involves Digital Rights Management. "Will I be able to read those ebooks I bought in two years time?" is a realistic question, but it is of a very different nature. As Amazon has already demonstrated! -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Dec 22, 2016 at 11:58 AM, Anton Aylward
On 12/22/2016 11:29 AM, Carlos E. R. wrote:
On 2016-12-22 15:19, Anton Aylward wrote:
On 12/22/2016 09:07 AM, Carlos E. R. wrote:
Is it reliable? Can we trust the journal to work and not loose anything in, say, two years time, even after upgrades?
You've descended into the stage of a ridiculous argument, Carlos.
I don't think so. A binary store is prone to breakage, it has happened. If you think that way, I'll think then that you are a zealot that negates any possible problem.
It is a reasonable question. Will I be able to read it in 2 years time, in any machine
I say again: You've descended into the stage of a ridiculous argument, Carlos
I 100% disagree, and reading logs is part of my professional day job. (Never journald yet professionally. Always Windows event logs or linux text logs to date.). With syslog logs, there are lots of tools. I use X-Ways and Splunk in particular to read logs off of computers that have been compromised. -> X-Ways is my main day-to-day tool. It has zero support of journald binary logs. I would have to export the log files and run them through journalctl then add the text logs back into my exam. -> Splunk can also natively parse Windows binary event logs from the last 15 or so years. When I just googled for Splunk and systemd I found this comment: "Now that Splunk is running and has a place to live, we need to feed data to it. To do this, I setup another service, which uses the journalctl tool to export the journal to a text file. This is required as Splunk can’t read the native JournalD binary format" From: http://blogs.splunk.com/2015/04/30/integrating-splunk-with-docker-coreos-and... So for this professional centralized logging and analysis tool, I have to use text files as the stepping stone. That obviously works okay for day 1 centralizing of logs. But, what if I'm doing an examination of a server 5 years after the fact. (I really do things like that on occasion.) I haven't seen a spec for the journald binary format. I haven't seen a commitment that journalctl from 2020 will have backward compatibility to read 2016 journald logs. I'm not saying those don't exist, but without that commitment at a minimum systemd loses much of its value for me as an incident response responder. And unfortunately I'm not normally part of the planning process for this. I get called in when the incident is identified, not years earlier when the logging system is established. I interpret Carlos's question to relate to that commitment. It is a very important question. I truly hope the commitment has been made, but to claim it is unimportant is simply wrong. Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 22/12/2016 à 19:29, Greg Freemyer a écrit :
But, what if I'm doing an examination of a server 5 years after the fact. (I really do things like that on occasion.)
same for databases. Did you notice mysql backups are in text form? It's not surprising than a system that is pretty new don't have all the tools an old one have. If we happen to have no more electricity, will' have to resort to printed material... That's a very good reason to keep old and new for a long period of time. There are still kde3 users down here :-) this is no reason not to use modern systems, with reason. The interest of present discussion is to make clear what is the fundamental of the problem. It's certainly not what is written in the subject. Better werite it as: * how to manage useful flood in logs? * how to manage very long term logs archival and may be some others jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
22.12.2016 21:29, Greg Freemyer пишет:
I haven't seen a spec for the journald binary format. I haven't seen a commitment that journalctl from 2020 will have backward compatibility to read 2016 journald logs.
https://www.freedesktop.org/wiki/Software/systemd/InterfacePortabilityAndSta... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/22/2016 01:29 PM, Greg Freemyer wrote:
I haven't seen a spec for the journald binary format. I haven't seen a commitment that journalctl from 2020 will have backward compatibility to read 2016 journald logs.
In my googling around I did see a API spec for the journal database. No, I don't think I specifically bookmarked it. I did mention scripting and yes there is an API for journal in scripting languages: https://pypi.python.org/pypi/pyjournalctl/ So they must have drawn on a published API. Lets see, googling for "systemd journal api" returne this at the top of the list https://www.freedesktop.org/software/systemd/man/sd-journal.html "APIs for submitting and querying log entries to and from the journal" As for compatible commitment, can you show me a similar statement for other binary media such as eBooks or the binary format used by the Linux link-loader? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 19:57, Anton Aylward wrote:
On 12/22/2016 01:29 PM, Greg Freemyer wrote:
As for compatible commitment, can you show me a similar statement for other binary media such as eBooks or the binary format used by the Linux link-loader?
Greg mentioned a tool for Windows that specifies being able to read 15 year old logs. As for ebooks, that's a proprietary format, not opensource. There are many complains about that format. What many people do is break the protection and save the backup as unprotected ebook, to negate the control to the companies and keep it in the hand of the person that bought the ebook or his inheritors. As to the link loader, it remains compatible. I still run binaries that were built circa yr 2000. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdDDUACgkQja8UbcUWM1z0ggD+JF7p4oQbBYDjDbfmYcLJ+EDR Zaw3JIFTNvuur4IQsd0BAJNtdodd5rCjPOC8/n03I4xXzENnRMnbsBe49MKID9pl =b/Io -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 23/12/16 11:36, Carlos E. R. wrote:
On 2016-12-22 19:57, Anton Aylward wrote:
On 12/22/2016 01:29 PM, Greg Freemyer wrote:
As for compatible commitment, can you show me a similar statement for other binary media such as eBooks or the binary format used by the Linux link-loader?
Greg mentioned a tool for Windows that specifies being able to read 15 year old logs.
As for ebooks, that's a proprietary format, not opensource. There are many complains about that format. What many people do is break the protection and save the backup as unprotected ebook, to negate the control to the companies and keep it in the hand of the person that bought the ebook or his inheritors.
What's wrong with proprietary formats? What matters is that they are DOCUMENTED formats. And actually, I believe journald's predecessor format wasn't even *defined*, let alone documented (other than the *assumption* it was ascii text - even that wasn't a given, I believe!)
As to the link loader, it remains compatible. I still run binaries that were built circa yr 2000.
And I can (if I could only get the program to run) use a certain DOS program from 1994 to read files created by its latest incarnation. The WordPerfect file format has remained unchanged, and both backwards- AND forwards-compatible since WP6 was released in 1994. And the only reason the file format changed from the WP5 format was because of Microsoft. This ever-changing file-formats roundabout - I won't say it was *created* by them - has been made the norm by Microsoft as a revenue-gouging technique. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-24 02:00, Wols Lists wrote:
On 23/12/16 11:36, Carlos E. R. wrote:
As for ebooks, that's a proprietary format, not opensource. There are many complains about that format. What many people do is break the protection and save the backup as unprotected ebook, to negate the control to the companies and keep it in the hand of the person that bought the ebook or his inheritors.
What's wrong with proprietary formats? What matters is that they are DOCUMENTED formats.
This was a comment about the epub format with DRM. The fact that it is proprietary takes it out of our concern in our context, because we can not do anything about it, it is not under community control.
And actually, I believe journald's predecessor format wasn't even *defined*, let alone documented (other than the *assumption* it was ascii text - even that wasn't a given, I believe!)
No, that's not so. There are documented standards. You can send the log entries as they are generated on a machine (say, a router) to any other machine running a syslog server, with different operating system. How they are formatted when saved to disk may differ, but there are also some standards. The message itself is entirely free form text, though. Same in systemd journal, as it is up to each program that saves messages to the log, not to systemd or syslog. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdzFEACgkQja8UbcUWM1yd6wD7BAIWIj8bZZV7mC2uZY4Z2FTs amOxCIaU3V17wG59t64BAJDsXRrOCBC2N+0Zl1+BcPlh2r/XRYyHnwJqAKAd/Id/ =TfcT -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 19:29, Greg Freemyer wrote:
On Thu, Dec 22, 2016 at 11:58 AM, Anton Aylward
wrote: On 12/22/2016 11:29 AM, Carlos E. R. wrote:
But, what if I'm doing an examination of a server 5 years after the fact. (I really do things like that on occasion.)
I haven't seen a spec for the journald binary format. I haven't seen a commitment that journalctl from 2020 will have backward compatibility to read 2016 journald logs.
I'm not saying those don't exist, but without that commitment at a minimum systemd loses much of its value for me as an incident response responder. And unfortunately I'm not normally part of the planning process for this. I get called in when the incident is identified, not years earlier when the logging system is established.
I interpret Carlos's question to relate to that commitment. It is a very important question. I truly hope the commitment has been made, but to claim it is unimportant is simply wrong.
Yes, absolutely. I was thinking of that. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdCucACgkQja8UbcUWM1x0rwD/QiNHttIJLVsc/E7FbGShcRiG uXUKnPCbHVRQXgNDuaEA/RyPgNhXpcNt05cA1cDASnGdn8MQyTjgGkiHXzFqhlkt =YEV5 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2016-12-22 15:19, Anton Aylward wrote:
On 12/22/2016 09:07 AM, Carlos E. R. wrote:
Is it reliable? Can we trust the journal to work and not loose anything in, say, two years time, even after upgrades?
You've descended into the stage of a ridiculous argument, Carlos.
I don't think so. A binary store is prone to breakage, it has happened.
That it has happened at least once is perhaps not quite the same as "being prone to". Otherwise my mysql databases are prone to breaking. -- Per Jessen, Zürich (0.1°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 19:12, Per Jessen wrote:
Carlos E. R. wrote:
I don't think so. A binary store is prone to breakage, it has happened.
That it has happened at least once is perhaps not quite the same as "being prone to". Otherwise my mysql databases are prone to breaking.
Well, that's proof that mysql is reliable :-) Are there tools to recover and rebuild a broken/corrupted journal? - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdDLYACgkQja8UbcUWM1z+5gD/diZCgO+cxi/19DFvmNgf1fu4 7BBva0LlY4VNS7IfkwUA/jznZtHcaxDgg1ynPeyQZWGafQZ0/ZdIxGYvmsJzrExP =xXmu -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 23/12/2016 à 12:38, Carlos E. R. a écrit :
Are there tools to recover and rebuild a broken/corrupted journal?
that's a good question jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 12/22/2016 09:07 AM, Carlos E. R. wrote:
Is it reliable? Can we trust the journal to work and not loose anything in, say, two years time, even after upgrades?
You've descended into the stage of a ridiculous argument, Carlos.
--- Not really. He points out that ng-syslogd already logs to binary in a way that can be read by current DB viewers. So far, journeld has no tool support as it was designed from scratch providing no or limited compatibility. For example, someone may want to keep logs related to logins going back 3 years, but not everything else. I have specific logs going back insanely log... squid logs back to 2010, for example. I see some old spamd logs from 2009 -- gotta clean these out.. But they are all still readable using "more"... To ask that such info be readable only 2 years in the future, is a *ridiculously* small amount. To call wanting such, ridiculous by any measure is just naive, ridiculous or senile (having forgotten that such basic needs were met ages ago by the previous log system, but now need to be re-invented w/a new log system). Remember, by default journald was designed for volatile storage, w/working store on SSD's. Not your typical logging system. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 21:53, L A Walsh wrote:
Anton Aylward wrote:
On 12/22/2016 09:07 AM, Carlos E. R. wrote:
Is it reliable? Can we trust the journal to work and not loose anything in, say, two years time, even after upgrades?
You've descended into the stage of a ridiculous argument, Carlos.
--- Not really. He points out that ng-syslogd already logs to binary in a way that can be read by current DB viewers.
So far, journeld has no tool support as it was designed from scratch providing no or limited compatibility. For example, someone may want to keep logs related to logins going back 3 years, but not everything else.
I have specific logs going back insanely log... squid logs back to 2010, for example. I see some old spamd logs from 2009 -- gotta clean these out..
But they are all still readable using "more"...
To ask that such info be readable only 2 years in the future, is a *ridiculously* small amount. To call wanting such, ridiculous by any measure is just naive, ridiculous or senile (having forgotten that such basic needs were met ages ago by the previous log system, but now need to be re-invented w/a new log system). Remember, by default journald was designed for volatile storage, w/working store on SSD's. Not your typical logging system.
Thanks. Yes, that is so. Journal brings wanted features, like data integrity (I doubt it, but nevertheless lets accepts it does have it). Ie, it guarantees that the data has not being altered and that it was issued by the parties it says, not faked. Well, to be of use to a party that needs to keep logs (legally or not), it also has to prove that it can be accessed in a number of years and that it can be backed up. If the data has to be dumped to text the integrity is broken. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdDhgACgkQja8UbcUWM1z9kwD9Ee6NROY6BOwmAHeOC6p3ILe7 JTknWwedYgApQoU4uG0A+QFekHye8+95o0USPtraUpPx4bXhQsbuDqSZr4p5s4Gu =brSQ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/23/2016 03:44 AM, Carlos E. R. wrote:
On 2016-12-22 21:53, L A Walsh wrote:
Anton Aylward wrote:
On 12/22/2016 09:07 AM, Carlos E. R. wrote:
Is it reliable? Can we trust the journal to work and not loose anything in, say, two years time, even after upgrades?
You've descended into the stage of a ridiculous argument, Carlos.
--- Not really. He points out that ng-syslogd already logs to binary in a way that can be read by current DB viewers.
So far, journeld has no tool support as it was designed from scratch providing no or limited compatibility. For example, someone may want to keep logs related to logins going back 3 years, but not everything else.
I have specific logs going back insanely log... squid logs back to 2010, for example. I see some old spamd logs from 2009 -- gotta clean these out..
But they are all still readable using "more"...
To ask that such info be readable only 2 years in the future, is a *ridiculously* small amount. To call wanting such, ridiculous by any measure is just naive, ridiculous or senile (having forgotten that such basic needs were met ages ago by the previous log system, but now need to be re-invented w/a new log system). Remember, by default journald was designed for volatile storage, w/working store on SSD's. Not your typical logging system.
Thanks. Yes, that is so.
Journal brings wanted features, like data integrity (I doubt it, but nevertheless lets accepts it does have it). Ie, it guarantees that the data has not being altered and that it was issued by the parties it says, not faked. Well, to be of use to a party that needs to keep logs (legally or not), it also has to prove that it can be accessed in a number of years and that it can be backed up. If the data has to be dumped to text the integrity is broken.
But journald is just the first destination of log messages Carlos, as you have pointed out in other posts. It is not intended, nor does it pretend to provide a single solution. You and others have pointed out how to short circuit it to any other logging engine, for those special case situations where ISPs or such need legal logs. So once again TEMPEST IN A TEAPOT. And no, there is no legal requirement in any country that states logs lose authenticity simply because they were dumped to text. You made that up!! Old logs are already compressed routinely, and decompressing them and printing them out has nevee cause a judge or Magistrate to toss them out of court. Find me ONE case law example of this happening in any jurisdiction in the world!!! If SLES wasn't so ridiculously expensive nobody running a business would use opensuse in any capacity that had legal requirements. -- After all is said and done, more is said than done.
On Fri, Dec 23, 2016 at 3:40 PM, John Andersen
Old logs are already compressed routinely, and decompressing them and printing them out has nevee cause a judge or Magistrate to toss them out of court. Find me ONE case law example of this happening in any jurisdiction in the world!!!
John, I do this for a living. For records of any kind to be used in court, someone has to provide a foundation as to why the records are accurate and trustworthy. Logs are no different. The court starts with the assumption that all potential evidence is unreliable. That's why all records introduced in court have to be introduced by someone who can provide a foundation as to their reliability. 5 examples: 1) Once I was asked to certify phone billing logs for a client. They gave me a print-out of the billing records they wanted certified. I refused and said I could only certify the billing records if I got them straight from the original source (the phone company website) and printed them myself. When I did, there were key changes. My client had downloaded the text billing logs and edited them to get rid of the evidence, then wanted me to swear they were reliable. If I'd have done so, I'd need to find a new career. 2) In a UNIX (pre-linux) work environment I once had to evaluate who had conducted certain actions. The logs showed it was person A. We didn't trust those text logs and reviewed the shell history files for all the users to see if we could find any contradictory evidence. We found where an admin (root level user) had used vi to edit the relevant log files shortly after the key event took place. If he would have edited his own shell history file, we would have never known. Just goes to show how untrustworthy text logs can be. 3) Modern malware is known to attempt to cover its tracks. It first gets root/admin privileges, then does its dirty work, then attempts delete the relevant logs. With text logs, that can be very targeted. With Windows binary event logs, it has to delete the entire event log file. I for one would be unwilling in general to swear text logs are reliable. I would merely be willing to swear the logs being presented represented the logs as preserved by me on a given date and time. 4) There was a bankruptcy case about 5 years ago where American Express was trying to recover a lot of money that had been charged on their cards (millions of $s as I recall). The party going bankrupt argued that AmEx'es bills were not a reliable source of information about what charges were made to the card. I'm sure American Express could have satisfied the court as to why their bills were reliable and that the charges they reflected were actually made. Instead AmEx sent a non-technical representative to testify to the court about why the billing records were reliable. The judge refused to listen to the non-technically knowledgeable testimony. He told American Express they had to produce a person knowledgeable about their computer systems and architecture to explain how the charges were collected and managed and why they should be considered reliable. American Express made 2 additional attempts to send someone to certify those billing records as reliable. In both cases the judge refused to accept their testimony due to their lack of foundation about how the process worked. The billing records were therefore not allowed into evidence and AmEx recovered none of their money. fyi: This was a famous case at the time. 5) I worked for the defense in a criminal case where the financial records being used by the FBI came out of an accounting system. The records clearly showed that financial filings for grants had false information in them. As part of my work, I reviewed years of trouble tickets and system engineers notes about the specific system in question. It was clear the system had been extremely poorly maintained and the records couldn't be trusted. === The reality is all evidence used in court has to sworn to as being accurate and what their foundation is for swearing to that.. The opposing side is allowed to argue that documents aren't reliable and . text logs are not going to be considered reliable in a lot of court cases. I'd be more than happy to be hired to argue about having them thrown out of court. Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/23/2016 01:46 PM, Greg Freemyer wrote:
On Fri, Dec 23, 2016 at 3:40 PM, John Andersen
wrote: Old logs are already compressed routinely, and decompressing them and printing them out has nevee cause a judge or Magistrate to toss them out of court. Find me ONE case law example of this happening in any jurisdiction in the world!!!
John,
I do this for a living. For records of any kind to be used in court, someone has to provide a foundation as to why the records are accurate and trustworthy. Logs are no different. The court starts with the assumption that all potential evidence is unreliable. That's why all records introduced in court have to be introduced by someone who can provide a foundation as to their reliability.
5 examples:
1) Once I was asked to certify phone billing logs for a client. They gave me a print-out of the billing records they wanted certified.
I refused and said I could only certify the billing records if I got them straight from the original source (the phone company website) and printed them myself.
When I did, there were key changes. My client had downloaded the text billing logs and edited them to get rid of the evidence, then wanted me to swear they were reliable. If I'd have done so, I'd need to find a new career.
2) In a UNIX (pre-linux) work environment I once had to evaluate who had conducted certain actions. The logs showed it was person A. We didn't trust those text logs and reviewed the shell history files for all the users to see if we could find any contradictory evidence. We found where an admin (root level user) had used vi to edit the relevant log files shortly after the key event took place.
If he would have edited his own shell history file, we would have never known. Just goes to show how untrustworthy text logs can be.
3) Modern malware is known to attempt to cover its tracks. It first gets root/admin privileges, then does its dirty work, then attempts delete the relevant logs. With text logs, that can be very targeted. With Windows binary event logs, it has to delete the entire event log file.
I for one would be unwilling in general to swear text logs are reliable. I would merely be willing to swear the logs being presented represented the logs as preserved by me on a given date and time.
4) There was a bankruptcy case about 5 years ago where American Express was trying to recover a lot of money that had been charged on their cards (millions of $s as I recall). The party going bankrupt argued that AmEx'es bills were not a reliable source of information about what charges were made to the card.
I'm sure American Express could have satisfied the court as to why their bills were reliable and that the charges they reflected were actually made. Instead AmEx sent a non-technical representative to testify to the court about why the billing records were reliable.
The judge refused to listen to the non-technically knowledgeable testimony. He told American Express they had to produce a person knowledgeable about their computer systems and architecture to explain how the charges were collected and managed and why they should be considered reliable.
American Express made 2 additional attempts to send someone to certify those billing records as reliable. In both cases the judge refused to accept their testimony due to their lack of foundation about how the process worked.
The billing records were therefore not allowed into evidence and AmEx recovered none of their money.
fyi: This was a famous case at the time.
5) I worked for the defense in a criminal case where the financial records being used by the FBI came out of an accounting system. The records clearly showed that financial filings for grants had false information in them.
As part of my work, I reviewed years of trouble tickets and system engineers notes about the specific system in question. It was clear the system had been extremely poorly maintained and the records couldn't be trusted.
=== The reality is all evidence used in court has to sworn to as being accurate and what their foundation is for swearing to that.. The opposing side is allowed to argue that documents aren't reliable and .
text logs are not going to be considered reliable in a lot of court cases. I'd be more than happy to be hired to argue about having them thrown out of court.
Greg -- Greg Freemyer
None of that is at question here. What is at question is that carlos is arguing that the simple act of dumping a binary log to a text one renders it unacceptable. I think he is also arguing (unstated) that passing logs through journald on their way to oldschool syslog also renders them unacceptable by mere extension of the above. Of course there is no way for a HUMAN to read the binary logs, so the case he is building is that journald is basically unusable, because even the act of reading with journalctl renders the binary log to text. At that point I am lost as to what he would prefer and what he would trust, and where is train of though is going. No, the argument is not that log tampering is impossible, but rather the concept that rendering logs to text suddenly and by itself renders them no longer authoritative. The binary logs are certainly closer to something that is far more tamper proof. The format and structure is documented, and will be as readable in 2020 or 2120 as they are today. But if you don't trust the process of printing them in human readable form then all is for naught. Again, the thread has devolved into something begging to be put out of its own misery. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-23 23:05, John Andersen wrote:
On 12/23/2016 01:46 PM, Greg Freemyer wrote:
None of that is at question here.
What is at question is that carlos is arguing that the simple act of dumping a binary log to a text one renders it unacceptable.
As you see from Greg response, that is so. It is only acceptable if /he/ does it and certifies in court that he did it in a controlled environment. This requires that we have tools to extract the text data several years later, and that those binary logs can be kept reliably for many years in the running machines.
I think he is also arguing (unstated) that passing logs through journald on their way to oldschool syslog also renders them unacceptable by mere extension of the above.
I did not say so, but the systemd developer himself would say so. It is obvious. One of the goals of the systemd journal is authenticity and integrity. The journal can not make this guarantee if a second party is in the mix this way. To be valid in a legal environment the binary logs have to arrive intact to the people doing the investigation. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdokEACgkQja8UbcUWM1yG0gD+P9BMEar8QObM47M0eF9XxUez +o5ceRarXQTRbu5RrpIBAIW3Fgq+zgYjBMvMc5KTBnYSfRvQ+mVg4YlEpTKPbAk5 =xSJX -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/23/2016 02:16 PM, Carlos E. R. wrote:
To be valid in a legal environment the binary logs have to arrive intact to the people doing the investigation.
That would be pointless, if, as you insisted, the mere act of dumping them to text renders them suspect. Judges will just need to learn to read binary, and compare it to the binary catalog apparently. -- After all is said and done, more is said than done.
On Fri, Dec 23, 2016 at 5:19 PM, John Andersen
On 12/23/2016 02:16 PM, Carlos E. R. wrote:
To be valid in a legal environment the binary logs have to arrive intact to the people doing the investigation.
That would be pointless, if, as you insisted, the mere act of dumping them to text renders them suspect. Judges will just need to learn to read binary, and compare it to the binary catalog apparently.
Judges don't do that stage of analysis. People like me do. I hope to learn that journald has a high-level of integrity so I can be the one that takes possession of a journald binary log and converts it to a text log and prints it. I would then testify to the fact I did so as a trusted party. Any pointers to why journald logs have a high-level of integrity? Why they can't be easily manipulated after the fact? I understand that was a design goal, but I've yet to read how it was accomplished. Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 23/12/16 23:19, Greg Freemyer wrote:
On Fri, Dec 23, 2016 at 5:19 PM, John Andersen
wrote: On 12/23/2016 02:16 PM, Carlos E. R. wrote:
To be valid in a legal environment the binary logs have to arrive intact to the people doing the investigation.
That would be pointless, if, as you insisted, the mere act of dumping them to text renders them suspect. Judges will just need to learn to read binary, and compare it to the binary catalog apparently.
Judges don't do that stage of analysis.
People like me do. I hope to learn that journald has a high-level of integrity so I can be the one that takes possession of a journald binary log and converts it to a text log and prints it. I would then testify to the fact I did so as a trusted party.
Any pointers to why journald logs have a high-level of integrity? Why they can't be easily manipulated after the fact?
I understand that was a design goal, but I've yet to read how it was accomplished.
Simple chaining of checksums I believe. If each record contains the checksum of the previous record, in order to modify one old record you then have to re-checksum the ENTIRE log from that point on. And any competent sysadmin should do what I did for an accounting system years ago. Every night, I ran an integrity check which spat out a hard copy double-entry summary. It didn't add up to zero, but the point is that total should NEVER change. And every morning, the accountant got the summary off the printer, checked it against the previous night's summary, and filed it. I heard of a legal office that did the same sort of thing - every night at close of business, they got day's final checksum off their document management system, and sent it to the local newspaper as a tiny legal notice ad to be printed next day. So you can't necessarily detect tampering, but any attempt to alter the historic record across one of those boundaries will be tamper-evident. And because systemd doesn't have to store the log on the machine that generated it, an intruder could easily find he has no access to the logs that need to be altered. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Dec 23, 2016 at 5:05 PM, John Andersen
The binary logs are certainly closer to something that is far more tamper proof. The format and structure is documented, and will be as readable in 2020 or 2120 as they are today. But if you don't trust the process of printing them in human readable form then all is for naught.
The tamper proof nature is what I hoped this thread would be about. I was hoping that people would provide reasons (or point at relevant web resources) why journald binary logs were preferred to traditional text logs. Instead the whole discussion went sideways. Personally, I expect journald logs to be better than syslog text logs, but little has been said to truly support that. Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/23/2016 02:30 PM, Greg Freemyer wrote:
Personally, I expect journald logs to be better than syslog text logs, but little has been said to truly support that.
I think little CAN be said to support that! journald logs are just obfuscated text logs. We all know that Security Through Obscurity doesn't exist. I'd think that the only way logs would be completely trusted by a technologically aware court would be to somehow cryptographically hash the entries with timestamps and run-time authorization keys of some sort. BTW, I've seen requirements for logs to be protected from alteration by requiring two separate authenticators. This would imply that root wouldn't have access, but it would require two separate smartcard or tokens for access. Can this kind of a thing even exist in the UNIX/Linux world? Windows AD maybe? Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-23 23:30, Greg Freemyer wrote:
On Fri, Dec 23, 2016 at 5:05 PM, John Andersen
wrote: The binary logs are certainly closer to something that is far more tamper proof. The format and structure is documented, and will be as readable in 2020 or 2120 as they are today. But if you don't trust the process of printing them in human readable form then all is for naught.
The tamper proof nature is what I hoped this thread would be about.
True.
I was hoping that people would provide reasons (or point at relevant web resources) why journald binary logs were preferred to traditional text logs.
Ok, lets think. A plain text log can not be tamperproof, because as you know, it can be edited. It needs a cryptographic signature, and this has to be added on each line as it gets written, not later as a postprocess. It can be a checksum per line, then a paragraph signature now and then that signs a number of lines preceding the signature block. Plain text can be signed, yes. We do it on email. But the instant we do that, that text becomes a binary: change a single "bit", change a single letter, and the signature fails. Add a space, reformat, and it fails. On normal text, it does not "change" by doing some minor edit. Anyway. A binary format has it easier to add checksums and validity checks to the fields. It is easier to detect tampering. However, I do not know if these checksums and signatures have been added already to the journal. I read something about it long ago, but I have no links, only vague recollections. Thinking aloud, each log record could also have a signature block or a checksum. Then I guess that every now and then a bunch of records would need a big signature block, stored as a another record, but calculated and written perhaps many minutes after they were written. rsyslog has cryptographic modules. I do not know if for signing or for encryption of logs. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdyNQACgkQja8UbcUWM1wuiAEAoQ7uv+qG/magiZkDWs7Tyei1 alYcC0uD9yVzCGJ02BsA/0D5Kqirqp+Qlbu2V9LtR3aMC31VzC8igv6BCGKX8jNX =HrXc -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-23 21:40, John Andersen wrote:
On 12/23/2016 03:44 AM, Carlos E. R. wrote:
On 2016-12-22 21:53, L A Walsh wrote:
Anton Aylward wrote:
On 12/22/2016 09:07 AM, Carlos E. R. wrote:
But they are all still readable using "more"...
To ask that such info be readable only 2 years in the future, is a *ridiculously* small amount. To call wanting such, ridiculous by any measure is just naive, ridiculous or senile (having forgotten that such basic needs were met ages ago by the previous log system, but now need to be re-invented w/a new log system). Remember, by default journald was designed for volatile storage, w/working store on SSD's. Not your typical logging system.
Thanks. Yes, that is so.
Journal brings wanted features, like data integrity (I doubt it, but nevertheless lets accepts it does have it). Ie, it guarantees that the data has not being altered and that it was issued by the parties it says, not faked. Well, to be of use to a party that needs to keep logs (legally or not), it also has to prove that it can be accessed in a number of years and that it can be backed up. If the data has to be dumped to text the integrity is broken.
But journald is just the first destination of log messages Carlos, as you have pointed out in other posts.
No, not so. I use another destination because journal doesn't fill my bill, which negates the advantages of the journal.
It is not intended, nor does it pretend to provide a single solution. You and others have pointed out how to short circuit it to any other logging engine, for those special case situations where ISPs or such need legal logs.
So once again TEMPEST IN A TEAPOT.
And no, there is no legal requirement in any country that states logs lose authenticity simply because they were dumped to text. You made that up!!
You see in Greg response how wrong you are in this, so I will not bother. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhdn4wACgkQja8UbcUWM1z36gD+Kh3T4sOALxH4UJS2/WNPk7va mRoG3/htCzM2BsvfuzQA/0+K2g6M/tvfpKPc9mHtYLJ9D65gn8/IICcSEBXgb9gv =HXaT -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/23/2016 02:05 PM, Carlos E. R. wrote:
And no, there is no legal requirement in any country that states logs lose authenticity simply because they were dumped to text. You made that up!! You see in Greg response how wrong you are in this, so I will not bother.
I see no such thing. I do notice some apparent comprehension difficulties on your part. Greg talked about intentional fraud. He never implied that simply dumping logs to text was sufficient to render logs un-acceptable to a court. Nor did he mention how it fraud in logs was discovered without dumping logs to text by the investigators. -- After all is said and done, more is said than done.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-22 08:18, ianseeks wrote:
On Wednesday, 21 December 2016 08:22:20 GMT Per Jessen wrote:
Carlos E. R. wrote:
I'm obviously not an admin but i'm interested in expanding my knowledge so you'll have to excuse my ignorance/inexperience and as the thread has got long, i might have forgotten or missed an explanation. I think the journal keeping all the records is the only way, sure its storage heavy but it'll have to be done. I don't see how you can prove that your logs are secure i.e. not tampered with (by outsiders or insiders), to an independent reviewer if you do not keep every single record. But if you are in an industry that does not legally mandate keeping every record then i guess you can do what you like with the data as its irrelevant to anyone else until you are trying to find a rogue employee fiddling with the system.
Yes, exactly. If you are not mandated to keep all logs, you can not tune which you keep. There is no filtering capacity to the journal. In that case, syslog is better. As for the rogue employee, you simply have to keep logs in a separate machine with no access by anybody. If he has access to the machine, he can do things, like erasing all logs. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhbydgACgkQja8UbcUWM1xP9QD/SYPHKqQRRaT5bpbyfFbi8Xb1 CeNdAIRUhf6z+QEEzS0BAJBH1ySeEdB03wVB9ihB9QSr2EeTgMzWPi1O5flsv9yj =pz4K -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
This sums the situation up well, I believe. On 12/22/2016 02:18 AM, ianseeks wrote:
I think the journal keeping all the records is the only way, sure its storage heavy but it'll have to be done. I don't see how you can prove that your logs are secure i.e. not tampered with (by outsiders or insiders), to an independent reviewer if you do not keep every single record.
In order to be "bank Grade", that is to meet regulations such as COSO, HHIPC and many others, Suse ABSOLUTELY MUST supply this high integrity logging system.
But if you are in an industry that does not legally mandate keeping every record then i guess you can do what you like with the data as its irrelevant to anyone else
As Carlos makes clear, you can turn it off and drop back and edit/filter your logs if those legal constraints are not present. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
ianseeks wrote:
I think the journal keeping all the records is the only way, sure its storage heavy but it'll have to be done. I don't see how you can prove that your logs are secure i.e. not tampered with (by outsiders or insiders), to an independent reviewer if you do not keep every single record.
Why would an opensuse user (vs. corporate user using Novell's SuSE) need such things? Opensuse is supposed to be mostly for users and maybe small businesses not expecting official support. Certainly, they would not be businesses needed secure or tamper-proof logs. If they were, they wouldn't be recording to local disks were things could be edited anyway, but to an external recording device or signed-network-secure device.
But if you are in an industry that does not legally mandate keeping every record then i guess you can do what you like with the data as its irrelevant to anyone else until you are trying to find a rogue employee fiddling with the system.
That's the assumption here. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Tue, 20 Dec 2016, Carlos E. R. wrote:
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such.
Looked at gnumeric? -dnh -- printk(KERN_DEBUG "%s: I'm broken.\n", dev->name); linux-2.6.6/drivers/net/plip.c -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-21 04:30, David Haller wrote:
Hello,
On Tue, 20 Dec 2016, Carlos E. R. wrote:
I mention LO because that's the only powerful table viewer I know in Linux that can apply filters and such.
Looked at gnumeric?
Not recently, no. I found it lacking but nice. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhaW6IACgkQja8UbcUWM1zzjAD+OhvaWBXpY1mkAHtreNYNhwW7 bncQFFZQKCNrrtGu3swA/3+MFdu9TvXfJ145tuKPD7t87GaaXYrjbY1ruDTRNrs2 =ydSI -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-12-18 20:21, Greg Freemyer wrote:
I'd like to know more about this. In particular from a authenticity and integrity perspective.
Comments about those, or pointers to discussion about same would be appreciated.
==== Here is Lennart Poettering's list of problems with what journald replaces:
1) The message data is generally not authenticated, every local process can claim to be Apache under PID 4711, and syslog will believe that and store it on disk.
I'm not aware that this has changed with journal. I can use "logger" in a script to send entries to syslog and journald, and they are more or less free form.
2) The data logged is very free-form. Automated log-analyzers need to parse human language strings to a) identify message types, and b) parse parameters from them. This results in regex horrors, and a steady need to play catch-up with upstream developers who might tweak the human language log strings in new versions of their software. Effectively, in a away, in order not to break user-applied regular expressions all log messages become ABI of the software generating them, which is usually not intended by the developer.
I don't see this as changed. The application or system programmer chooses what to send to the log and in which format.
3) The timestamps generally do not carry timezone information, even though some newer specifications define support for it.
I understand the timestamp is a binary, and the syslog application formats it as the administrator defines. There is, I understand, one timestamp as sent by the application, and another as stamped by the syslog daemon.
4) Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.
It is up to the programmers to use syslog or not. This is not one ring to rule them all.
5) Reading log files is simple but very inefficient. Many key log operations have a complexity of O(n). Indexing is generally not available.
6) The syslog network protocol is very simple, but also very limited. Since it generally supports only a push transfer model, and does not employ store-and-forward, problems such as Thundering Herd or packet loss severely hamper its use.
7) Log files are easily manipulable by attackers, providing easy ways to hide attack information from the administrator
Yes. But. With some syslog daemons you can choose to save into a database instead than as text.
8) Access control is non-existent. Unless manually scripted by the administrator a user either gets full access to the log files, or no access at all.
Partly. The administrator may give access to one log file or others, as he wishes, with standard *nix type file permissions and acls.
9) The meta data stored for log entries is limited, and lacking key bits of information, such as service name, audit session or monotonic timestamps.
10) Automatic rotation of log files is available, but less than ideal in most implementations: instead of watching disk usage continuously to enforce disk usage limits rotation is only attempted in fixed time intervals, thus leaving the door open to many DoS attacks.
On the other hand, I have seen journald to grow excessively, like gigabytes in days, because one particular service logs a lot. Say a busy mail server or a busy news server. There is no provision with journald to purge selectively a class of logs that is not wanted to be kept for long. Also, accessing the journal can be very slow unless the machine has an SSD. An order of magnitude slower than traditional syslog.
11) Rate limiting is available in some implementations, however, generally does not take the disk usage or service assignment into account, which is highly advisable.
12) Compression in the log structure on disk is generally available but usually only as effect of rotation and has a negative effect on the already bad complexity behaviour of many key log operations.
Perhaps not if syslog is writing to a database. It is then up to the database engine.
13) Classic Syslog traditionally is not useful to handle early boot or late shutdown logging, even though recent improvements (for example in systemd) made this work.
No, early boot could be logged to syslog. When the syslog service started, the kernel log entries could be logged to syslog, to another log file, or both. The openSUSE implementation was to another file, and as far as I remember, everything was included. As to halt messages, nothing after the syslog daemon closes can be stored, obviously. Same with journal. Does it stop later? I don't know, but at some point it has to stop.
14) Binary data cannot be logged, which in some cases is essential (Examples: ATA SMART blobs or SCSI sense data, firmware dumps)
Better into separate files, IMHO. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlhYCCoACgkQja8UbcUWM1w7FQD+LPoYzX3UdJs+nqP4hQze1mLb FaMKJ5/KuNyrASyLvS0A/0ijYgF8wlUoL/j7tbIvE5JL/uBiJWX+2OkKNcMRhAXa =kci2 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (18)
-
Andrei Borzenkov
-
Anton Aylward
-
Carlos E. R.
-
Christoph Feck
-
Damon Register
-
David Haller
-
Felix Miata
-
Greg Freemyer
-
ianseeks
-
jdd
-
John Andersen
-
L A Walsh
-
Lew Wolfgang
-
nicholas cunliffe
-
Patrick Shanahan
-
Per Jessen
-
Peter Ragosch
-
Wols Lists