Feature changed by: Kay Sievers (kay_sievers) Feature #312466, revision 3 Title: Separate user and kernel log messages openSUSE Distribution: New Priority Requester: Important Requested by: Andreas Jaeger (a_jaeger) Product Manager: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Partner organization: openSUSE.org Description: SUSE used blogd to separate kernel and user log messages during boot. With systemd now everything is written to the kernel ring buffer. We should separate those again. Relations: - syslog (novell/bugzilla/id: 696963) https://bugzilla.novell.com/show_bug.cgi?id=696963 + Discussion: + #1: Kay Sievers (kay_sievers) (2011-05-31 15:13:14) + They are separated by the facility value in the syslog prefix. The + kernel was changed to handle this separation just fine. This is + actually feature not something to work around. Proper syslog + implementations can separate the messages just fine. + Early boot tools should use the same log target as the kernel. Today, + there is no reason to distinguish kernel- from userspace during early + boot. + As soon as the real syslog is running, the kernel buffer is not used + anymore. -- openSUSE Feature: https://features.opensuse.org/312466

Feature changed by: Andreas Jaeger (a_jaeger) Feature #312466, revision 5 Title: Separate user and kernel log messages openSUSE Distribution: New Priority Requester: Important + Info Provider: (Novell) Requested by: Andreas Jaeger (a_jaeger) Product Manager: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Partner organization: openSUSE.org Description: SUSE used blogd to separate kernel and user log messages during boot. With systemd now everything is written to the kernel ring buffer. We should separate those again. Relations: - syslog (novell/bugzilla/id: 696963) https://bugzilla.novell.com/show_bug.cgi?id=696963 Discussion: #1: Kay Sievers (kay_sievers) (2011-05-31 15:13:14) They are separated by the facility value in the syslog prefix. The kernel was changed to handle this separation just fine. This is actually feature not something to work around. Proper syslog implementations can separate the messages just fine. Early boot tools should use the same log target as the kernel. Today, there is no reason to distinguish kernel- from userspace during early boot. As soon as the real syslog is running, the kernel buffer is not used anymore. #2: Dr. Werner Fink (wernerfink) (2011-05-31 16:18:11) (reply to #1) # echo Test printk > /dev/kmsg ; sleep 1; tail /var/log/messages | grep printk May 31 16:06:16 shannon kernel: [1743439.436249] Test printk ... linux/drivers/char/mem.c ... kmsg_write() ... linux/kernel/printk.c ... DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */ ... linux/include/linux/kernel.h ... KERN_WARNING + #3: Andreas Jaeger (a_jaeger) (2011-05-31 16:40:01) (reply to #2) + Kay, how does this separation work? With 2.6.39 and the echo that + Werner used, I do not see it at all... -- openSUSE Feature: https://features.opensuse.org/312466

Feature changed by: Andreas Jaeger (a_jaeger) Feature #312466, revision 7 Title: Separate user and kernel log messages openSUSE Distribution: New Priority Requester: Important - Info Provider: (Novell) Requested by: Andreas Jaeger (a_jaeger) Product Manager: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Partner organization: openSUSE.org Description: SUSE used blogd to separate kernel and user log messages during boot. With systemd now everything is written to the kernel ring buffer. We should separate those again. Relations: - syslog (novell/bugzilla/id: 696963) https://bugzilla.novell.com/show_bug.cgi?id=696963 Discussion: #1: Kay Sievers (kay_sievers) (2011-05-31 15:13:14) They are separated by the facility value in the syslog prefix. The kernel was changed to handle this separation just fine. This is actually feature not something to work around. Proper syslog implementations can separate the messages just fine. Early boot tools should use the same log target as the kernel. Today, there is no reason to distinguish kernel- from userspace during early boot. As soon as the real syslog is running, the kernel buffer is not used anymore. #2: Dr. Werner Fink (wernerfink) (2011-05-31 16:18:11) (reply to #1) # echo Test printk > /dev/kmsg ; sleep 1; tail /var/log/messages | grep printk May 31 16:06:16 shannon kernel: [1743439.436249] Test printk ... linux/drivers/char/mem.c ... kmsg_write() ... linux/kernel/printk.c ... DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */ ... linux/include/linux/kernel.h ... KERN_WARNING #3: Andreas Jaeger (a_jaeger) (2011-05-31 16:40:01) (reply to #2) Kay, how does this separation work? With 2.6.39 and the echo that Werner used, I do not see it at all... #4: Kay Sievers (kay_sievers) (2011-05-31 16:51:44) (reply to #3) The echo needs the proper syslog prefix. The kernel uses the defaults if its missing. # echo '<30> LOG_INFO+LOG_DAEMON' > /dev/kmsg # dmesg -r | grep LOG_DAEMON <30>[172420.523276] LOG_INFO+LOG_DAEMON -- openSUSE Feature: https://features.opensuse.org/312466

Feature changed by: Dr. Werner Fink (WernerFink) Feature #312466, revision 9 Title: Separate user and kernel log messages openSUSE Distribution: New Priority Requester: Important Requested by: Andreas Jaeger (a_jaeger) Product Manager: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Partner organization: openSUSE.org Description: SUSE used blogd to separate kernel and user log messages during boot. With systemd now everything is written to the kernel ring buffer. We should separate those again. Relations: - syslog (novell/bugzilla/id: 696963) https://bugzilla.novell.com/show_bug.cgi?id=696963 Discussion: #1: Kay Sievers (kay_sievers) (2011-05-31 15:13:14) They are separated by the facility value in the syslog prefix. The kernel was changed to handle this separation just fine. This is actually feature not something to work around. Proper syslog implementations can separate the messages just fine. Early boot tools should use the same log target as the kernel. Today, there is no reason to distinguish kernel- from userspace during early boot. As soon as the real syslog is running, the kernel buffer is not used anymore. #2: Dr. Werner Fink (wernerfink) (2011-05-31 16:18:11) (reply to #1) # echo Test printk > /dev/kmsg ; sleep 1; tail /var/log/messages | grep printk May 31 16:06:16 shannon kernel: [1743439.436249] Test printk ... linux/drivers/char/mem.c ... kmsg_write() ... linux/kernel/printk.c ... DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */ ... linux/include/linux/kernel.h ... KERN_WARNING #3: Andreas Jaeger (a_jaeger) (2011-05-31 16:40:01) (reply to #2) Kay, how does this separation work? With 2.6.39 and the echo that Werner used, I do not see it at all... #4: Kay Sievers (kay_sievers) (2011-05-31 16:51:44) (reply to #3) The echo needs the proper syslog prefix. The kernel uses the defaults if its missing. # echo '<30> LOG_INFO+LOG_DAEMON' > /dev/kmsg # dmesg -r | grep LOG_DAEMON <30>[172420.523276] LOG_INFO+LOG_DAEMON #5: Kay Sievers (kay_sievers) (2011-05-31 18:03:29) (reply to #4) Maybe this is easier to read: # logger test; dmesg -r | grep test [ 41.342824] root[418]: test <13>[ 41.342824] root[418]: test The 13 is LOG_USER LOG_NOTICE, which is 'logger' default. The kernel extracs the LOG_NOTICE level for its own log-level processing. The kernel will use LOG_KERNEL==0 for its messages, systemd used LOG_DAEMON==6 for the messages. So all userspace messages sent by syslog() will be LOG_DAEMON unless otherwise specified. Mixing them during early boot is actually a wanted feature, it makes things easier to follow if things go wrong. There is no information lost, syslog can sort or separate the messages in any way it wants. + #6: Dr. Werner Fink (wernerfink) (2011-05-31 18:31:25) (reply to #5) + Now suppose the system is running (that is rsyslogd is up) and I've + used the lines # echo '<30>LOG_INFO+LOG_DAEMON Test' > /dev/kmsg # echo + '<30>6+24 Das ist ein Test' > /dev/kmsg # echo '<30>Das ist ein Test' > + /dev/kmsg the question rises why I do not see anything in + /var/log/messages. + The next question rises for e.g. mainframe and the e.g. setup of the + system for e.g. SAP environment before rsyslogd is up and running. -- openSUSE Feature: https://features.opensuse.org/312466

Feature changed by: Dr. Werner Fink (WernerFink) Feature #312466, revision 11 Title: Separate user and kernel log messages openSUSE Distribution: New Priority Requester: Important Requested by: Andreas Jaeger (a_jaeger) Product Manager: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Partner organization: openSUSE.org Description: SUSE used blogd to separate kernel and user log messages during boot. With systemd now everything is written to the kernel ring buffer. We should separate those again. Relations: - syslog (novell/bugzilla/id: 696963) https://bugzilla.novell.com/show_bug.cgi?id=696963 Discussion: #1: Kay Sievers (kay_sievers) (2011-05-31 15:13:14) They are separated by the facility value in the syslog prefix. The kernel was changed to handle this separation just fine. This is actually feature not something to work around. Proper syslog implementations can separate the messages just fine. Early boot tools should use the same log target as the kernel. Today, there is no reason to distinguish kernel- from userspace during early boot. As soon as the real syslog is running, the kernel buffer is not used anymore. #2: Dr. Werner Fink (wernerfink) (2011-05-31 16:18:11) (reply to #1) # echo Test printk > /dev/kmsg ; sleep 1; tail /var/log/messages | grep printk May 31 16:06:16 shannon kernel: [1743439.436249] Test printk ... linux/drivers/char/mem.c ... kmsg_write() ... linux/kernel/printk.c ... DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */ ... linux/include/linux/kernel.h ... KERN_WARNING #3: Andreas Jaeger (a_jaeger) (2011-05-31 16:40:01) (reply to #2) Kay, how does this separation work? With 2.6.39 and the echo that Werner used, I do not see it at all... #4: Kay Sievers (kay_sievers) (2011-05-31 16:51:44) (reply to #3) The echo needs the proper syslog prefix. The kernel uses the defaults if its missing. # echo '<30> LOG_INFO+LOG_DAEMON' > /dev/kmsg # dmesg -r | grep LOG_DAEMON <30>[172420.523276] LOG_INFO+LOG_DAEMON #5: Kay Sievers (kay_sievers) (2011-05-31 18:03:29) (reply to #4) Maybe this is easier to read: # logger test; dmesg -r | grep test [ 41.342824] root[418]: test <13>[ 41.342824] root[418]: test The 13 is LOG_USER LOG_NOTICE, which is 'logger' default. The kernel extracs the LOG_NOTICE level for its own log-level processing. The kernel will use LOG_KERNEL==0 for its messages, systemd used LOG_DAEMON==6 for the messages. So all userspace messages sent by syslog() will be LOG_DAEMON unless otherwise specified. Mixing them during early boot is actually a wanted feature, it makes things easier to follow if things go wrong. There is no information lost, syslog can sort or separate the messages in any way it wants. #6: Dr. Werner Fink (wernerfink) (2011-05-31 18:31:25) (reply to #5) Now suppose the system is running (that is rsyslogd is up) and I've used the lines # echo '<30>LOG_INFO+LOG_DAEMON Test' > /dev/kmsg # echo '<30>6+24 Das ist ein Test' > /dev/kmsg # echo '<30>Das ist ein Test' > /dev/kmsg the question rises why I do not see anything in /var/log/messages. The next question rises for e.g. mainframe and the e.g. setup of the system for e.g. SAP environment before rsyslogd is up and running. #7: Kay Sievers (kay_sievers) (2011-05-31 18:47:53) (reply to #6) That's a question how syslog works. Nothing systemd can do about, if syslog does not properly read the kernel log buffer. SUSE's syslog packages do (did) not work properly with systemd because it did not ship the upstream provided service files, but hacked-up init scripts which try to use/invent features which are not supported by systemd. That might be the reason syslog does not work for you. The 'mainframe' issue is not different from the general issue of 'how to see the kernel's messages'. The same way you get to the kernel's messages, you get to the early syslog() messages, because they are just stored in the kernel. Any system that can see the kernel's messages, can see the early syslog() messages from the systemd bridge. + #8: Dr. Werner Fink (wernerfink) (2011-05-31 18:53:45) (reply to #7) + The parser of the rsyslogd has nothing todo how the rsyslogd is started + and you should know that rsyslogd is able to be the end of the systemd + logging bridge. In fact all three syslog deamons can be used with + systemd as all three do have the appropiate changes for serving the end + of systemd logging bridge. -- openSUSE Feature: https://features.opensuse.org/312466

Feature changed by: Dr. Werner Fink (WernerFink) Feature #312466, revision 13 Title: Separate user and kernel log messages openSUSE Distribution: New Priority Requester: Important Requested by: Andreas Jaeger (a_jaeger) Product Manager: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Partner organization: openSUSE.org Description: SUSE used blogd to separate kernel and user log messages during boot. With systemd now everything is written to the kernel ring buffer. We should separate those again. Relations: - syslog (novell/bugzilla/id: 696963) https://bugzilla.novell.com/show_bug.cgi?id=696963 Discussion: #1: Kay Sievers (kay_sievers) (2011-05-31 15:13:14) They are separated by the facility value in the syslog prefix. The kernel was changed to handle this separation just fine. This is actually feature not something to work around. Proper syslog implementations can separate the messages just fine. Early boot tools should use the same log target as the kernel. Today, there is no reason to distinguish kernel- from userspace during early boot. As soon as the real syslog is running, the kernel buffer is not used anymore. #2: Dr. Werner Fink (wernerfink) (2011-05-31 16:18:11) (reply to #1) # echo Test printk > /dev/kmsg ; sleep 1; tail /var/log/messages | grep printk May 31 16:06:16 shannon kernel: [1743439.436249] Test printk ... linux/drivers/char/mem.c ... kmsg_write() ... linux/kernel/printk.c ... DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */ ... linux/include/linux/kernel.h ... KERN_WARNING #3: Andreas Jaeger (a_jaeger) (2011-05-31 16:40:01) (reply to #2) Kay, how does this separation work? With 2.6.39 and the echo that Werner used, I do not see it at all... #4: Kay Sievers (kay_sievers) (2011-05-31 16:51:44) (reply to #3) The echo needs the proper syslog prefix. The kernel uses the defaults if its missing. # echo '<30> LOG_INFO+LOG_DAEMON' > /dev/kmsg # dmesg -r | grep LOG_DAEMON <30>[172420.523276] LOG_INFO+LOG_DAEMON #5: Kay Sievers (kay_sievers) (2011-05-31 18:03:29) (reply to #4) Maybe this is easier to read: # logger test; dmesg -r | grep test [ 41.342824] root[418]: test <13>[ 41.342824] root[418]: test The 13 is LOG_USER LOG_NOTICE, which is 'logger' default. The kernel extracs the LOG_NOTICE level for its own log-level processing. The kernel will use LOG_KERNEL==0 for its messages, systemd used LOG_DAEMON==6 for the messages. So all userspace messages sent by syslog() will be LOG_DAEMON unless otherwise specified. Mixing them during early boot is actually a wanted feature, it makes things easier to follow if things go wrong. There is no information lost, syslog can sort or separate the messages in any way it wants. #6: Dr. Werner Fink (wernerfink) (2011-05-31 18:31:25) (reply to #5) Now suppose the system is running (that is rsyslogd is up) and I've used the lines # echo '<30>LOG_INFO+LOG_DAEMON Test' > /dev/kmsg # echo '<30>6+24 Das ist ein Test' > /dev/kmsg # echo '<30>Das ist ein Test' > /dev/kmsg the question rises why I do not see anything in /var/log/messages. The next question rises for e.g. mainframe and the e.g. setup of the system for e.g. SAP environment before rsyslogd is up and running. #7: Kay Sievers (kay_sievers) (2011-05-31 18:47:53) (reply to #6) That's a question how syslog works. Nothing systemd can do about, if syslog does not properly read the kernel log buffer. SUSE's syslog packages do (did) not work properly with systemd because it did not ship the upstream provided service files, but hacked-up init scripts which try to use/invent features which are not supported by systemd. That might be the reason syslog does not work for you. The 'mainframe' issue is not different from the general issue of 'how to see the kernel's messages'. The same way you get to the kernel's messages, you get to the early syslog() messages, because they are just stored in the kernel. Any system that can see the kernel's messages, can see the early syslog() messages from the systemd bridge. #8: Dr. Werner Fink (wernerfink) (2011-05-31 18:53:45) (reply to #7) The parser of the rsyslogd has nothing todo how the rsyslogd is started and you should know that rsyslogd is able to be the end of the systemd logging bridge. In fact all three syslog deamons can be used with systemd as all three do have the appropiate changes for serving the end of systemd logging bridge. #9: Kay Sievers (kay_sievers) (2011-05-31 19:24:27) (reply to #8) None of SUSE's current syslog can work properly with systemd, because they don't use native service files. SYSV scripts can not use socket activation/passing, which is the key feature needed to do a seamless transition from the early-syslog-bridge to the real syslog. + #10: Dr. Werner Fink (wernerfink) (2011-05-31 20:31:21) (reply to #9) + The parser works even with systemd just check but not for your special + case of redirecting system messages to the kernel printk buffer (as + Andreas said he is using native service file). + Beside there was *no* need to break a working setup that is that the + deamons could do their work if you wouldn't have broken it in malicious + manner. And the maintainers of the sysloggers have do their job, that + is first L3, then SLES11-SP2, and at last but not least openSUSE. -- openSUSE Feature: https://features.opensuse.org/312466

Feature changed by: Dr. Werner Fink (WernerFink) Feature #312466, revision 15 Title: Separate user and kernel log messages openSUSE Distribution: New Priority Requester: Important Requested by: Andreas Jaeger (a_jaeger) Product Manager: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Partner organization: openSUSE.org Description: SUSE used blogd to separate kernel and user log messages during boot. With systemd now everything is written to the kernel ring buffer. We should separate those again. Relations: - syslog (novell/bugzilla/id: 696963) https://bugzilla.novell.com/show_bug.cgi?id=696963 Discussion: #1: Kay Sievers (kay_sievers) (2011-05-31 15:13:14) They are separated by the facility value in the syslog prefix. The kernel was changed to handle this separation just fine. This is actually feature not something to work around. Proper syslog implementations can separate the messages just fine. Early boot tools should use the same log target as the kernel. Today, there is no reason to distinguish kernel- from userspace during early boot. As soon as the real syslog is running, the kernel buffer is not used anymore. #2: Dr. Werner Fink (wernerfink) (2011-05-31 16:18:11) (reply to #1) # echo Test printk > /dev/kmsg ; sleep 1; tail /var/log/messages | grep printk May 31 16:06:16 shannon kernel: [1743439.436249] Test printk ... linux/drivers/char/mem.c ... kmsg_write() ... linux/kernel/printk.c ... DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */ ... linux/include/linux/kernel.h ... KERN_WARNING #3: Andreas Jaeger (a_jaeger) (2011-05-31 16:40:01) (reply to #2) Kay, how does this separation work? With 2.6.39 and the echo that Werner used, I do not see it at all... #4: Kay Sievers (kay_sievers) (2011-05-31 16:51:44) (reply to #3) The echo needs the proper syslog prefix. The kernel uses the defaults if its missing. # echo '<30> LOG_INFO+LOG_DAEMON' > /dev/kmsg # dmesg -r | grep LOG_DAEMON <30>[172420.523276] LOG_INFO+LOG_DAEMON #5: Kay Sievers (kay_sievers) (2011-05-31 18:03:29) (reply to #4) Maybe this is easier to read: # logger test; dmesg -r | grep test [ 41.342824] root[418]: test <13>[ 41.342824] root[418]: test The 13 is LOG_USER LOG_NOTICE, which is 'logger' default. The kernel extracs the LOG_NOTICE level for its own log-level processing. The kernel will use LOG_KERNEL==0 for its messages, systemd used LOG_DAEMON==6 for the messages. So all userspace messages sent by syslog() will be LOG_DAEMON unless otherwise specified. Mixing them during early boot is actually a wanted feature, it makes things easier to follow if things go wrong. There is no information lost, syslog can sort or separate the messages in any way it wants. #6: Dr. Werner Fink (wernerfink) (2011-05-31 18:31:25) (reply to #5) Now suppose the system is running (that is rsyslogd is up) and I've used the lines # echo '<30>LOG_INFO+LOG_DAEMON Test' > /dev/kmsg # echo '<30>6+24 Das ist ein Test' > /dev/kmsg # echo '<30>Das ist ein Test' > /dev/kmsg the question rises why I do not see anything in /var/log/messages. The next question rises for e.g. mainframe and the e.g. setup of the system for e.g. SAP environment before rsyslogd is up and running. #7: Kay Sievers (kay_sievers) (2011-05-31 18:47:53) (reply to #6) That's a question how syslog works. Nothing systemd can do about, if syslog does not properly read the kernel log buffer. SUSE's syslog packages do (did) not work properly with systemd because it did not ship the upstream provided service files, but hacked-up init scripts which try to use/invent features which are not supported by systemd. That might be the reason syslog does not work for you. The 'mainframe' issue is not different from the general issue of 'how to see the kernel's messages'. The same way you get to the kernel's messages, you get to the early syslog() messages, because they are just stored in the kernel. Any system that can see the kernel's messages, can see the early syslog() messages from the systemd bridge. #8: Dr. Werner Fink (wernerfink) (2011-05-31 18:53:45) (reply to #7) The parser of the rsyslogd has nothing todo how the rsyslogd is started and you should know that rsyslogd is able to be the end of the systemd logging bridge. In fact all three syslog deamons can be used with systemd as all three do have the appropiate changes for serving the end of systemd logging bridge. #9: Kay Sievers (kay_sievers) (2011-05-31 19:24:27) (reply to #8) None of SUSE's current syslog can work properly with systemd, because they don't use native service files. SYSV scripts can not use socket activation/passing, which is the key feature needed to do a seamless transition from the early-syslog-bridge to the real syslog. #10: Dr. Werner Fink (wernerfink) (2011-05-31 20:31:21) (reply to #9) The parser works even with systemd just check but not for your special case of redirecting system messages to the kernel printk buffer (as Andreas said he is using native service file). Beside there was *no* need to break a working setup that is that the deamons could do their work if you wouldn't have broken it in malicious manner. And the maintainers of the sysloggers have do their job, that is first L3, then SLES11-SP2, and at last but not least openSUSE. #11: Kay Sievers (kay_sievers) (2011-05-31 20:54:46) (reply to #10) The SUSE maintainers of the syslog just need to package an *upstream* provided file, instead of inventing features for systemd bootups which do not fit into the model. That takes almost zero time, unlike the dirty hacks they tried to push. It was their decision to waste this time, don't act surprised now, I told you from the beginning that this does not fit, and will not work out. There was no working, in the sense of working-with-systemd setup broken. Syslog is special for systemd, needs tight integration because of systemd's own syslog bridge, and therefore makes limitations for legacy SYSV compatibility. Advanced init features like socket activation are not available for any SYSV scripts. The systemd authors just don't share your view of not using the kernel buffer for early boot tools, nor do they share your view how the default setup of syslog needs to behave on a systemd system. Other than that, there is nothing malicious going on here. + #12: Dr. Werner Fink (wernerfink) (2011-06-01 09:14:22) (reply to #11) + Does this mean as you're a upstream developer of systemd you do not + care about the needs of our enterprise customers? The openSUSE is the + base of the enterprise and also vice versa. -- openSUSE Feature: https://features.opensuse.org/312466

Feature changed by: Dr. Werner Fink (WernerFink) Feature #312466, revision 17 Title: Separate user and kernel log messages openSUSE Distribution: New Priority Requester: Important Requested by: Andreas Jaeger (a_jaeger) Product Manager: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Technical Contact: (Novell) Partner organization: openSUSE.org Description: SUSE used blogd to separate kernel and user log messages during boot. With systemd now everything is written to the kernel ring buffer. We should separate those again. Relations: - syslog (novell/bugzilla/id: 696963) https://bugzilla.novell.com/show_bug.cgi?id=696963 Discussion: #1: Kay Sievers (kay_sievers) (2011-05-31 15:13:14) They are separated by the facility value in the syslog prefix. The kernel was changed to handle this separation just fine. This is actually feature not something to work around. Proper syslog implementations can separate the messages just fine. Early boot tools should use the same log target as the kernel. Today, there is no reason to distinguish kernel- from userspace during early boot. As soon as the real syslog is running, the kernel buffer is not used anymore. #2: Dr. Werner Fink (wernerfink) (2011-05-31 16:18:11) (reply to #1) # echo Test printk > /dev/kmsg ; sleep 1; tail /var/log/messages | grep printk May 31 16:06:16 shannon kernel: [1743439.436249] Test printk ... linux/drivers/char/mem.c ... kmsg_write() ... linux/kernel/printk.c ... DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */ ... linux/include/linux/kernel.h ... KERN_WARNING #3: Andreas Jaeger (a_jaeger) (2011-05-31 16:40:01) (reply to #2) Kay, how does this separation work? With 2.6.39 and the echo that Werner used, I do not see it at all... #4: Kay Sievers (kay_sievers) (2011-05-31 16:51:44) (reply to #3) The echo needs the proper syslog prefix. The kernel uses the defaults if its missing. # echo '<30> LOG_INFO+LOG_DAEMON' > /dev/kmsg # dmesg -r | grep LOG_DAEMON <30>[172420.523276] LOG_INFO+LOG_DAEMON #5: Kay Sievers (kay_sievers) (2011-05-31 18:03:29) (reply to #4) Maybe this is easier to read: # logger test; dmesg -r | grep test [ 41.342824] root[418]: test <13>[ 41.342824] root[418]: test The 13 is LOG_USER LOG_NOTICE, which is 'logger' default. The kernel extracs the LOG_NOTICE level for its own log-level processing. The kernel will use LOG_KERNEL==0 for its messages, systemd used LOG_DAEMON==6 for the messages. So all userspace messages sent by syslog() will be LOG_DAEMON unless otherwise specified. Mixing them during early boot is actually a wanted feature, it makes things easier to follow if things go wrong. There is no information lost, syslog can sort or separate the messages in any way it wants. #6: Dr. Werner Fink (wernerfink) (2011-05-31 18:31:25) (reply to #5) Now suppose the system is running (that is rsyslogd is up) and I've used the lines # echo '<30>LOG_INFO+LOG_DAEMON Test' > /dev/kmsg # echo '<30>6+24 Das ist ein Test' > /dev/kmsg # echo '<30>Das ist ein Test' > /dev/kmsg the question rises why I do not see anything in /var/log/messages. The next question rises for e.g. mainframe and the e.g. setup of the system for e.g. SAP environment before rsyslogd is up and running. #7: Kay Sievers (kay_sievers) (2011-05-31 18:47:53) (reply to #6) That's a question how syslog works. Nothing systemd can do about, if syslog does not properly read the kernel log buffer. SUSE's syslog packages do (did) not work properly with systemd because it did not ship the upstream provided service files, but hacked-up init scripts which try to use/invent features which are not supported by systemd. That might be the reason syslog does not work for you. The 'mainframe' issue is not different from the general issue of 'how to see the kernel's messages'. The same way you get to the kernel's messages, you get to the early syslog() messages, because they are just stored in the kernel. Any system that can see the kernel's messages, can see the early syslog() messages from the systemd bridge. #8: Dr. Werner Fink (wernerfink) (2011-05-31 18:53:45) (reply to #7) The parser of the rsyslogd has nothing todo how the rsyslogd is started and you should know that rsyslogd is able to be the end of the systemd logging bridge. In fact all three syslog deamons can be used with systemd as all three do have the appropiate changes for serving the end of systemd logging bridge. #9: Kay Sievers (kay_sievers) (2011-05-31 19:24:27) (reply to #8) None of SUSE's current syslog can work properly with systemd, because they don't use native service files. SYSV scripts can not use socket activation/passing, which is the key feature needed to do a seamless transition from the early-syslog-bridge to the real syslog. #10: Dr. Werner Fink (wernerfink) (2011-05-31 20:31:21) (reply to #9) The parser works even with systemd just check but not for your special case of redirecting system messages to the kernel printk buffer (as Andreas said he is using native service file). Beside there was *no* need to break a working setup that is that the deamons could do their work if you wouldn't have broken it in malicious manner. And the maintainers of the sysloggers have do their job, that is first L3, then SLES11-SP2, and at last but not least openSUSE. #11: Kay Sievers (kay_sievers) (2011-05-31 20:54:46) (reply to #10) The SUSE maintainers of the syslog just need to package an *upstream* provided file, instead of inventing features for systemd bootups which do not fit into the model. That takes almost zero time, unlike the dirty hacks they tried to push. It was their decision to waste this time, don't act surprised now, I told you from the beginning that this does not fit, and will not work out. There was no working, in the sense of working-with-systemd setup broken. Syslog is special for systemd, needs tight integration because of systemd's own syslog bridge, and therefore makes limitations for legacy SYSV compatibility. Advanced init features like socket activation are not available for any SYSV scripts. The systemd authors just don't share your view of not using the kernel buffer for early boot tools, nor do they share your view how the default setup of syslog needs to behave on a systemd system. Other than that, there is nothing malicious going on here. #12: Dr. Werner Fink (wernerfink) (2011-06-01 09:14:22) (reply to #11) Does this mean as you're a upstream developer of systemd you do not care about the needs of our enterprise customers? The openSUSE is the base of the enterprise and also vice versa. #13: Kay Sievers (kay_sievers) (2011-06-01 12:42:13) (reply to #12) Blogd was needed with SYSV, to make sure the early boot message got captured before they got lost. That's not needed or wanted with systemd, it's all fully covered. It has nothing to do with enterprise or not. + #14: Dr. Werner Fink (wernerfink) (2011-06-01 13:46:14) (reply to #13) + blogd? ... syslog-ng or syslogd or rsyslogd depending on the + requirements and the resulting choise of the enterprise customer. This + choise will be ignored by systemd as it ignores /etc/sysconfig/ + Beside this blogd is not only for logging but also for reporting an all + devices used for the system console not only for the primary device .. + AFAIK systemd does not do this. blogd has nothing todo with system + message logger, its a console message logger. -- openSUSE Feature: https://features.opensuse.org/312466

Feature changed by: Matthias Eckermann (mge1512) Feature #312466, revision 22 Title: Separate user and kernel log messages - openSUSE Distribution: New + openSUSE Distribution: Rejected by Matthias Eckermann (mge1512) + reject date: 2013-08-09 15:03:00 + reject reason: Infrastructure change based on SystemD . Priority Requester: Important Requested by: Andreas Jaeger (a_jaeger) Partner organization: openSUSE.org Description: SUSE used blogd to separate kernel and user log messages during boot. With systemd now everything is written to the kernel ring buffer. We should separate those again. Relations: - syslog (novell/bugzilla/id: 696963) https://bugzilla.novell.com/show_bug.cgi?id=696963 Discussion: #1: Kay Sievers (kay_sievers) (2011-05-31 15:13:14) They are separated by the facility value in the syslog prefix. The kernel was changed to handle this separation just fine. This is actually feature not something to work around. Proper syslog implementations can separate the messages just fine. Early boot tools should use the same log target as the kernel. Today, there is no reason to distinguish kernel- from userspace during early boot. As soon as the real syslog is running, the kernel buffer is not used anymore. #2: Dr. Werner Fink (wernerfink) (2011-05-31 16:18:11) (reply to #1) # echo Test printk > /dev/kmsg ; sleep 1; tail /var/log/messages | grep printk May 31 16:06:16 shannon kernel: [1743439.436249] Test printk ... linux/drivers/char/mem.c ... kmsg_write() ... linux/kernel/printk.c ... DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */ ... linux/include/linux/kernel.h ... KERN_WARNING #3: Andreas Jaeger (a_jaeger) (2011-05-31 16:40:01) (reply to #2) Kay, how does this separation work? With 2.6.39 and the echo that Werner used, I do not see it at all... #4: Kay Sievers (kay_sievers) (2011-05-31 16:51:44) (reply to #3) The echo needs the proper syslog prefix. The kernel uses the defaults if its missing. # echo '<30> LOG_INFO+LOG_DAEMON' > /dev/kmsg # dmesg -r | grep LOG_DAEMON <30>[172420.523276] LOG_INFO+LOG_DAEMON #5: Kay Sievers (kay_sievers) (2011-05-31 18:03:29) (reply to #4) Maybe this is easier to read: # logger test; dmesg -r | grep test [ 41.342824] root[418]: test <13>[ 41.342824] root[418]: test The 13 is LOG_USER LOG_NOTICE, which is 'logger' default. The kernel extracs the LOG_NOTICE level for its own log-level processing. The kernel will use LOG_KERNEL==0 for its messages, systemd used LOG_DAEMON==6 for the messages. So all userspace messages sent by syslog() will be LOG_DAEMON unless otherwise specified. Mixing them during early boot is actually a wanted feature, it makes things easier to follow if things go wrong. There is no information lost, syslog can sort or separate the messages in any way it wants. #6: Dr. Werner Fink (wernerfink) (2011-05-31 18:31:25) (reply to #5) Now suppose the system is running (that is rsyslogd is up) and I've used the lines # echo '<30>LOG_INFO+LOG_DAEMON Test' > /dev/kmsg # echo '<30>6+24 Das ist ein Test' > /dev/kmsg # echo '<30>Das ist ein Test' > /dev/kmsg the question rises why I do not see anything in /var/log/messages. The next question rises for e.g. mainframe and the e.g. setup of the system for e.g. SAP environment before rsyslogd is up and running. #7: Kay Sievers (kay_sievers) (2011-05-31 18:47:53) (reply to #6) That's a question how syslog works. Nothing systemd can do about, if syslog does not properly read the kernel log buffer. SUSE's syslog packages do (did) not work properly with systemd because it did not ship the upstream provided service files, but hacked-up init scripts which try to use/invent features which are not supported by systemd. That might be the reason syslog does not work for you. The 'mainframe' issue is not different from the general issue of 'how to see the kernel's messages'. The same way you get to the kernel's messages, you get to the early syslog() messages, because they are just stored in the kernel. Any system that can see the kernel's messages, can see the early syslog() messages from the systemd bridge. #8: Dr. Werner Fink (wernerfink) (2011-05-31 18:53:45) (reply to #7) The parser of the rsyslogd has nothing todo how the rsyslogd is started and you should know that rsyslogd is able to be the end of the systemd logging bridge. In fact all three syslog deamons can be used with systemd as all three do have the appropiate changes for serving the end of systemd logging bridge. #9: Kay Sievers (kay_sievers) (2011-05-31 19:24:27) (reply to #8) None of SUSE's current syslog can work properly with systemd, because they don't use native service files. SYSV scripts can not use socket activation/passing, which is the key feature needed to do a seamless transition from the early-syslog-bridge to the real syslog. #10: Dr. Werner Fink (wernerfink) (2011-05-31 20:31:21) (reply to #9) The parser works even with systemd just check but not for your special case of redirecting system messages to the kernel printk buffer (as Andreas said he is using native service file). Beside there was *no* need to break a working setup that is that the deamons could do their work if you wouldn't have broken it in malicious manner. And the maintainers of the sysloggers have do their job, that is first L3, then SLES11-SP2, and at last but not least openSUSE. #11: Kay Sievers (kay_sievers) (2011-05-31 20:54:46) (reply to #10) The SUSE maintainers of the syslog just need to package an *upstream* provided file, instead of inventing features for systemd bootups which do not fit into the model. That takes almost zero time, unlike the dirty hacks they tried to push. It was their decision to waste this time, don't act surprised now, I told you from the beginning that this does not fit, and will not work out. There was no working, in the sense of working-with-systemd setup broken. Syslog is special for systemd, needs tight integration because of systemd's own syslog bridge, and therefore makes limitations for legacy SYSV compatibility. Advanced init features like socket activation are not available for any SYSV scripts. The systemd authors just don't share your view of not using the kernel buffer for early boot tools, nor do they share your view how the default setup of syslog needs to behave on a systemd system. Other than that, there is nothing malicious going on here. #12: Dr. Werner Fink (wernerfink) (2011-06-01 09:14:22) (reply to #11) Does this mean as you're a upstream developer of systemd you do not care about the needs of our enterprise customers? The openSUSE is the base of the enterprise and also vice versa. #13: Kay Sievers (kay_sievers) (2011-06-01 12:42:13) (reply to #12) Blogd was needed with SYSV, to make sure the early boot message got captured before they got lost. That's not needed or wanted with systemd, it's all fully covered. It has nothing to do with enterprise or not. #14: Dr. Werner Fink (wernerfink) (2011-06-01 13:46:14) (reply to #13) blogd? ... syslog-ng or syslogd or rsyslogd depending on the requirements and the resulting choise of the enterprise customer. This choise will be ignored by systemd as it ignores /etc/sysconfig/ Beside this blogd is not only for logging but also for reporting an all devices used for the system console not only for the primary device .. AFAIK systemd does not do this. blogd has nothing todo with system message logger, its a console message logger. -- openSUSE Feature: https://features.opensuse.org/312466