+++================================ netconsole. Kernel logging on remote machine. Date: Fri, 23 May 2014 19:19:15 -0400 From: Cristian Rodríguez <> Reply-To: OS-en To: opensuse@opensuse.org Subject: Re: [opensuse] Kernel crash on multiple file write on reiserfs GPT partition. ... P D O .. that means: "P" --> propietary module loaded, developers will most likely ignore your report if it comes in this form. "D" --> the kernel has oopsed before, that means what you are showing in the picture is a secondary oops, not the actual problem. "O" -> "Out of tree module" is loaded, good luck with getting that fixed. ... https://www.kernel.org/doc/Documentation/networking/netconsole.txt ... Ah. Ok it appears to be the same as in "/usr/share/doc/packages/netconsole-tools/netlogging.txt" ... The documentation is obsolete. The correct syntax appears to be: modprobe netconsole 6666@192.168.1.14/eth0,6666@192.168.1.15 which I got from "http://www.cyberciti.biz/tips/linux-netconsole-log-management-tutorial.html". not modprobe netconsole netconsole="... ... > Try with the section "dynamic configuration" from the netconsole.txt doc. Telcontar:~ # modprobe netconsole Telcontar:~ # cd /sys/kernel/config/netconsole/ Telcontar:/sys/kernel/config/netconsole # ls Telcontar:/sys/kernel/config/netconsole # mkdir target1 Telcontar:/sys/kernel/config/netconsole # ls target1 Telcontar:/sys/kernel/config/netconsole # cd target1/ Telcontar:/sys/kernel/config/netconsole/target1 # ls dev_name enabled local_ip local_mac local_port remote_ip remote_mac remote_port Telcontar:/sys/kernel/config/netconsole/target1 # cat local_ local_ip local_mac local_port Telcontar:/sys/kernel/config/netconsole/target1 # cat local_ip 0.0.0.0 Telcontar:/sys/kernel/config/netconsole/target1 # echo 192.168.1.14 > local_ip Telcontar:/sys/kernel/config/netconsole/target1 # echo 6666 > local_port but Telcontar:/sys/kernel/config/netconsole/target1 # echo "00:21:85:16:2D:0B" > local_mac -bash: local_mac: Permission denied Telcontar:/sys/kernel/config/netconsole/target1 # cat local_mac ff:ff:ff:ff:ff: weird. Telcontar:/sys/kernel/config/netconsole/target1 # echo "00:03:0D:05:17:FC" > remote_mac Telcontar:/sys/kernel/config/netconsole/target1 # echo 6666 > remote_port Telcontar:/sys/kernel/config/netconsole/target1 # echo 192.168.1.15 > remote_ip Telcontar:/sys/kernel/config/netconsole/target1 # cat dev_name eth0 Telcontar:/sys/kernel/config/netconsole/target1 # echo 1 > enabled It is apparently started: Telcontar:/sys/kernel/config/netconsole/target1 # tail /var/log/messages <3.6> 2014-05-24 13:23:01 Telcontar systemd 1 - - Starting Session 78 of user news. <3.6> 2014-05-24 13:25:01 Telcontar systemd 1 - - Starting Session 79 of user news. <3.6> 2014-05-24 13:28:01 Telcontar systemd 1 - - Starting Session 80 of user news. <0.6> 2014-05-24 13:28:06 Telcontar kernel - - - [10768.603827] netpoll: netconsole: local port 6666 <0.6> 2014-05-24 13:28:06 Telcontar kernel - - - [10768.609384] netpoll: netconsole: local IPv4 address 192.168.1.14 <0.6> 2014-05-24 13:28:06 Telcontar kernel - - - [10768.614762] netpoll: netconsole: interface 'eth0' <0.6> 2014-05-24 13:28:06 Telcontar kernel - - - [10768.620095] netpoll: netconsole: remote port 6666 <0.6> 2014-05-24 13:28:06 Telcontar kernel - - - [10768.625373] netpoll: netconsole: remote IPv4 address 192.168.1.15 <0.6> 2014-05-24 13:28:06 Telcontar kernel - - - [10768.630545] netpoll: netconsole: remote ethernet address 00:03:0d:05:17:fc <0.6> 2014-05-24 13:28:06 Telcontar kernel - - - [10768.635653] netconsole: network logging started On the receiving computer, I have: netcat -u -l 6666 | tee -a remote_log I plugged a usb stick, and got the messages on the remote, so good! Now I go for testing and crashing the machine again. Nvidia is not in the list. Wish me luck! modprobe netconsole cd /sys/kernel/config/netconsole/ ls mkdir target1 ls cd target1/ ls cat * echo 192.168.1.14 > local_ip echo 6666 > local_port echo "00:03:0D:05:17:FC" > remote_mac echo 6666 > remote_port echo 192.168.1.15 > remote_ip cat dev_name echo 1 > enabled ------ 2015-11-22 6666 - Local port 192.168.1.5 - Local system IP eth0 - Local system interface 514 - Remote syslogd udp port 192.168.1.100 - Remote syslogd IP 00:19:D1:2A:BA:A8 - Remote syslogd Mac You can add above modprobe line to /etc/rc.local to load module automatically. Another recommend option is create /etc/modprobe.d/netconsole file and append following text: # echo 'options netconsole netconsole=6666@192.168.1.5/eth0,514@192.168.1.100/00:19:D1:2A:BA:A8 '> /etc/modprobe.d/netconsole echo 'options netconsole netconsole=6666@192.168.1.14/eth0,514@192.168.1.15/00:03:0d:05:17:fc '> /etc/modprobe.d/netconsole Log shows: <0.6> 2015-11-22 14:38:04 Telcontar kernel - - - [ 3495.081911] netpoll: netconsole: local port 6666 <0.6> 2015-11-22 14:38:04 Telcontar kernel - - - [ 3495.081920] netpoll: netconsole: local IPv4 address 192.168.1.14 <0.6> 2015-11-22 14:38:04 Telcontar kernel - - - [ 3495.081921] netpoll: netconsole: interface 'eth0' <0.6> 2015-11-22 14:38:04 Telcontar kernel - - - [ 3495.081922] netpoll: netconsole: remote port 514 <0.6> 2015-11-22 14:38:04 Telcontar kernel - - - [ 3495.081923] netpoll: netconsole: remote IPv4 address 192.168.1.15 <0.6> 2015-11-22 14:38:04 Telcontar kernel - - - [ 3495.081925] netpoll: netconsole: remote ethernet address 00:03:0d:05:17:fc <0.6> 2015-11-22 14:38:04 Telcontar kernel - - - [ 3495.081949] console [netcon0] enabled <0.6> 2015-11-22 14:38:04 Telcontar kernel - - - [ 3495.081949] netconsole: network logging started 2015-11-22 On the other side I get nothing, once I open the port. Nada. This does work: modprobe netconsole cd /sys/kernel/config/netconsole/ ls mkdir target1 ls cd target1/ ls cat * echo 192.168.1.14 > local_ip echo 6666 > local_port echo "00:03:0D:05:17:FC" > remote_mac echo 6666 > remote_port echo 192.168.1.15 > remote_ip cat dev_name echo 1 > enabled But not on port 514. This does work. echo 'options netconsole netconsole=6666@192.168.1.14/eth0,6666@192.168.1.15/00:03:0d:05:17:fc '> /etc/modprobe.d/netconsole.conf Ie, el syslog doesn't work. netcat -u -l 6666 | tee -a remote_log Very good: ================================---