[opensuse] SysRq output on serial console?
Hi all, to hunt down occasional kernel oopses I attached a serial console at my linux box. So I can get oopses even though the kernel can't write log files any more. But sometimes the kernel is not completely fucked up and SysRq still works. But the kernel can't write /var/log/messages any more, and I can't switch to my X11 console (Ctrl-Alt-F10). But SysRq writes header lines only to my serial console, e.g. "SysRq : Show Memory". The real interesting part, memory dumps, backtraces, locks, etc. are written to /var/log/messages only and thus are lost :-( I need them on the serial console. Anybody out there a clue at which facility and level SysRq dumps? Assuming the facility is kernel I edited syslog-ng.conf to ... filter f_console { level(debug) and facility(kern) and not filter(f_iptables) or level(err) and not facility(authpriv); }; ... but this didn't help either. Thanks for any help, Winfried -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hi all again, I forgot to mention I'm running openSuSE 10.2, current patchlevel. Sorry for the noise... Winfried -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Winfried Huber wrote:
But SysRq writes header lines only to my serial console, e.g. "SysRq : Show Memory". The real interesting part, memory dumps, backtraces, locks, etc. are written to /var/log/messages only and thus are lost :-( I need them on the serial console.
try sysrq-9 first (sets loglevel to highest, i.e. debug).
cheers,
Gerd
--
Gerd Hoffmann
Am Montag, 15. Januar 2007 14:00 schrieb Gerd Hoffmann:
But SysRq writes header lines only to my serial console, e.g. "SysRq : Show Memory". The real interesting part, memory dumps, backtraces, locks, etc. are written to /var/log/messages only and thus are lost :-( I need them on the serial console.
try sysrq-9 first (sets loglevel to highest, i.e. debug).
Doesn't help either ... dumps are written to /var/log/messages, but not to the serial console :-( Winfried -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Winfried Huber wrote:
Am Montag, 15. Januar 2007 14:00 schrieb Gerd Hoffmann:
But SysRq writes header lines only to my serial console, e.g. "SysRq : Show Memory". The real interesting part, memory dumps, backtraces, locks, etc. are written to /var/log/messages only and thus are lost :-( I need them on the serial console.
try sysrq-9 first (sets loglevel to highest, i.e. debug).
Doesn't help either ... dumps are written to /var/log/messages, but not to the serial console :-(
Did you try adding "kdb=on console=ttyS0 console=tty0,38400,8n1" to {lilo|elilo|grub}.conf append= line? Regards, Tore -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Am Montag, 15. Januar 2007 14:27 schrieb Tore H. Larsen:
Did you try adding "kdb=on console=ttyS0 console=tty0,38400,8n1" to {lilo|elilo|grub}.conf append= line?
Hi Tore, my /boot/grub/menu.lst entry reads... title openSUSE 10.2 - Serielle Konsole root (hd0,1) kernel /boot/vmlinuz-2.6.18.2-34-default root=/dev/sda2 vga=0x31a \ resume=/dev/sda1 splash=silent showopts \ console=ttyS0,38400n8r console=tty1 initrd /boot/initrd-2.6.18.2-34-default This should be correct, I think, and the other usual stuff (kernel oops etc.) gets written to the serial console. But the SysRq-Body lines are missing. But I did not "kdb=on", just added that. Will see whether this helps! Thanks, Winfried -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Am Montag, 15. Januar 2007 14:27 schrieb Tore H. Larsen:
Did you try adding "kdb=on console=ttyS0 console=tty0,38400,8n1" to {lilo|elilo|grub}.conf append= line?
I added "kbd=on" to my /boot/grub/menu.lst line, did not make any difference. Perhaps this does the same thing as "echo 1 > /proc/sys/kernel/sysrq" which I run in boot.local. Thanks, Winfried -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Am Montag, 15. Januar 2007 11:15 schrieb Winfried Huber:
to hunt down occasional kernel oopses I attached a serial console at my linux box. So I can get oopses even though the kernel can't write log files any more.
But sometimes the kernel is not completely fucked up and SysRq still works. But the kernel can't write /var/log/messages any more, and I can't switch to my X11 console (Ctrl-Alt-F10).
But SysRq writes header lines only to my serial console, e.g. "SysRq : Show Memory". The real interesting part, memory dumps, backtraces, locks, etc. are written to /var/log/messages only and thus are lost :-( I need them on the serial console.
Got it! The value of "KERNEL_LOGLEVEL" in /etc/sysconfig/syslog was set to 1 (the default value). Setting this to 7 solved the problem. This can be done either by manually editing the file (and running SuSEconfig afterwards) or using YAST2 -> system -> editor for /etc/sysconfig files -> System -> Logging -> KERNEL_LOGLEVEL. Now my console is pretty chatty... Thanks! Winfried -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Winfried Huber wrote:
Am Montag, 15. Januar 2007 11:15 schrieb Winfried Huber:
to hunt down occasional kernel oopses I attached a serial console at my linux box. So I can get oopses even though the kernel can't write log files any more.
But sometimes the kernel is not completely fucked up and SysRq still works. But the kernel can't write /var/log/messages any more, and I can't switch to my X11 console (Ctrl-Alt-F10).
But SysRq writes header lines only to my serial console, e.g. "SysRq : Show Memory". The real interesting part, memory dumps, backtraces, locks, etc. are written to /var/log/messages only and thus are lost :-( I need them on the serial console.
Got it! The value of "KERNEL_LOGLEVEL" in /etc/sysconfig/syslog was set to 1 (the default value). Setting this to 7 solved the problem.
This can be done either by manually editing the file (and running SuSEconfig afterwards) or using YAST2 -> system -> editor for /etc/sysconfig files -> System -> Logging -> KERNEL_LOGLEVEL.
Now my console is pretty chatty...
Thanks! Winfried
Suse set the kernel log level when starting the klogd daemon. I would prefer that klogd not modify the log log level. If a change is to be made to the default log level, why not make the change in sysctl.conf? While on the subject, there is an error in /etc/init.d/syslog. The line that sets the parameters for the klogd daemon include the -x option, which is also defined in /etc/sysconfig/syslog. The result is the duplication of the -x option. This is easy to see with ps -f $(pgrep klogd). Bill Anderson -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hi Bill, Am Donnerstag, 18. Januar 2007 20:55 schrieb Bill Anderson:
Got it! The value of "KERNEL_LOGLEVEL" in /etc/sysconfig/syslog was set to 1 (the default value). Setting this to 7 solved the problem.
This can be done either by manually editing the file (and running SuSEconfig afterwards) or using YAST2 -> system -> editor for /etc/sysconfig files -> System -> Logging -> KERNEL_LOGLEVEL.
Now my console is pretty chatty...
Thanks! Winfried
Suse set the kernel log level when starting the klogd daemon. I would prefer that klogd not modify the log log level. If a change is to be made to the default log level, why not make the change in sysctl.conf?
adding KERNEL_LOGLEVEL = "7" to /etc/sysctl.conf does not work for me (openSuSE 10.2) - this line seems to be simply ignored. The value in /etc/sysconfig/syslog gets used, and if it's commented out the default value of 1 is used. A value of "5" turned out to be "chatty enough" for the sysctl message bodies. Why do you think setting KERNEL_LOGLEVEL in /etc/sysconfig/syslog is evil? I think tuning the system is the purpose of /etc/sysconfig/* files? Winfried -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Winfried Huber wrote:
Hi Bill,
Am Donnerstag, 18. Januar 2007 20:55 schrieb Bill Anderson:
Got it! The value of "KERNEL_LOGLEVEL" in /etc/sysconfig/syslog was set to 1 (the default value). Setting this to 7 solved the problem.
This can be done either by manually editing the file (and running SuSEconfig afterwards) or using YAST2 -> system -> editor for /etc/sysconfig files -> System -> Logging -> KERNEL_LOGLEVEL.
Now my console is pretty chatty...
Thanks! Winfried
Suse set the kernel log level when starting the klogd daemon. I would prefer that klogd not modify the log log level. If a change is to be made to the default log level, why not make the change in sysctl.conf?
adding KERNEL_LOGLEVEL = "7" to /etc/sysctl.conf does not work for me (openSuSE 10.2) - this line seems to be simply ignored. The value in /etc/sysconfig/syslog gets used, and if it's commented out the default value of 1 is used. A value of "5" turned out to be "chatty enough" for the sysctl message bodies.
Why do you think setting KERNEL_LOGLEVEL in /etc/sysconfig/syslog is evil? I think tuning the system is the purpose of /etc/sysconfig/* files?
Winfried
You are correct in that setting a value in sysctl.conf is changed by the klogd parameter. The purpose of klogd is to extract kernel messages from the kernel buffer and pass them to the syslog-ng daemon. It is not its purpose to tune kernel parameters. The purpose of the sysctl command, and sysctl.conf at boot time, is to tune the writable parameters shown in /proc/sys. While many people use the echo command, you can modify the kernel log level at any time with the command sysctl -w kernel.printk=8. The log levels are 0 through 7, and the specified log level says print all message that are less than this value. This is different than syslog configuration, where the log level specifies values that are equal to or less than. Another reason for using sysctl.conf is that you control the log level earlier in the boot process. This may be important if you are looking for debug messages, which are log level 7. Bill Anderson WW7BA -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Am Freitag, 19. Januar 2007 20:36 schrieb Bill Anderson:
Winfried Huber wrote:
Hi Bill,
Am Donnerstag, 18. Januar 2007 20:55 schrieb Bill Anderson:
Suse set the kernel log level when starting the klogd daemon. I would prefer that klogd not modify the log log level. If a change is to be made to the default log level, why not make the change in sysctl.conf?
adding KERNEL_LOGLEVEL = "7" to /etc/sysctl.conf does not work for me (openSuSE 10.2) - this line seems to be simply ignored. The value in /etc/sysconfig/syslog gets used, and if it's commented out the default value of 1 is used. A value of "5" turned out to be "chatty enough" for the sysctl message bodies.
Why do you think setting KERNEL_LOGLEVEL in /etc/sysconfig/syslog is evil? I think tuning the system is the purpose of /etc/sysconfig/* files?
Winfried
You are correct in that setting a value in sysctl.conf is changed by the klogd parameter. The purpose of klogd is to extract kernel messages from the kernel buffer and pass them to the syslog-ng daemon. It is not its purpose to tune kernel parameters. The purpose of the sysctl command, and sysctl.conf at boot time, is to tune the writable parameters shown in /proc/sys. While many people use the echo command, you can modify the kernel log level at any time with the command sysctl -w kernel.printk=8. The log levels are 0 through 7, and the specified log level says print all message that are less than this value. This is different than syslog configuration, where the log level specifies values that are equal to or less than.
Another reason for using sysctl.conf is that you control the log level earlier in the boot process. This may be important if you are looking for debug messages, which are log level 7.
Bill Anderson
Thanks for lifting the fog... I now use both places: sysctl.conf to make the boot process chatty, and /etc/sysconfig/syslog to set the level once the machine has come up. I will check if I can omit this after the next reboot. Using echo or sysctl to tune the kernel log level during run time is not an option for me because this is no more possible once the machine is frozen or crashed. Thanks again, Bill! Winfried Huber -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Winfried Huber wrote:
Am Freitag, 19. Januar 2007 20:36 schrieb Bill Anderson:
Winfried Huber wrote:
Hi Bill,
Am Donnerstag, 18. Januar 2007 20:55 schrieb Bill Anderson:
Suse set the kernel log level when starting the klogd daemon. I would prefer that klogd not modify the log log level. If a change is to be made to the default log level, why not make the change in sysctl.conf?
adding KERNEL_LOGLEVEL = "7" to /etc/sysctl.conf does not work for me (openSuSE 10.2) - this line seems to be simply ignored. The value in /etc/sysconfig/syslog gets used, and if it's commented out the default value of 1 is used. A value of "5" turned out to be "chatty enough" for the sysctl message bodies.
Why do you think setting KERNEL_LOGLEVEL in /etc/sysconfig/syslog is evil? I think tuning the system is the purpose of /etc/sysconfig/* files?
Winfried
You are correct in that setting a value in sysctl.conf is changed by the klogd parameter. The purpose of klogd is to extract kernel messages from the kernel buffer and pass them to the syslog-ng daemon. It is not its purpose to tune kernel parameters. The purpose of the sysctl command, and sysctl.conf at boot time, is to tune the writable parameters shown in /proc/sys. While many people use the echo command, you can modify the kernel log level at any time with the command sysctl -w kernel.printk=8. The log levels are 0 through 7, and the specified log level says print all message that are less than this value. This is different than syslog configuration, where the log level specifies values that are equal to or less than.
Another reason for using sysctl.conf is that you control the log level earlier in the boot process. This may be important if you are looking for debug messages, which are log level 7.
Bill Anderson
Thanks for lifting the fog... I now use both places: sysctl.conf to make the boot process chatty, and /etc/sysconfig/syslog to set the level once the machine has come up. I will check if I can omit this after the next reboot. Using echo or sysctl to tune the kernel log level during run time is not an option for me because this is no more possible once the machine is frozen or crashed.
Thanks again, Bill!
Winfried Huber
Happy to help. I just want to mention one more detail. The default log level for kernel printk messages is 7, which is everything but debug messages. This value is set in printk.c. It is the operative value, until modified by sysctl.conf, klogd, dmesg, or modifying the value in /proc/sys/kernel/printk. Bill Anderson -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (4)
-
Bill Anderson -
Gerd Hoffmann -
Tore H. Larsen -
Winfried Huber