Our server running SLES8 for AMD64 crashed this morning. It restarted fine, but looking at the logs I cannot find anything that suggests a problem. The server runs Apache and Tomcat. I have looked at: /var/log/messages /var/log/apache2/* My Tomcat logs Where else should I be looking? I included my /var/log/messags below from just before the crash and the startup afterwareds - maybe I'm missing something. It crashed at 11:29, finally got the coloc operator to restart it at 11:54. Thanks in advance - Richard Aug 31 11:29:28 kingfish pop3d[6519]: mystore: starting txn 2147513745 Aug 31 11:29:28 kingfish pop3d[6519]: mystore: committing txn 2147513745 Aug 31 11:29:28 kingfish pop3d[6519]: starttls: TLSv1 with cipher RC4-MD5 (128/128 bits new) no authentication Aug 31 11:29:28 kingfish saslauthd[704]: pam_sm_authenticate called. ... (removed lines reporting on apm authentication with usernames) ... Aug 31 11:29:28 kingfish saslauthd[704]: returning 0. Aug 31 11:29:28 kingfish saslauthd[704]: pam_mysql: acct_mgmt called but not implemented. Dont panic though :) Aug 31 11:29:28 kingfish pop3d[6519]: login: [130.13.89.148] rnmixon.fromltoj.com plaintext+TLS Aug 31 11:29:28 kingfish pop3d[6519]: accepted connection Aug 31 11:29:29 kingfish pop3d[6519]: starttls: TLSv1 with cipher RC4-MD5 (128/128 bits reused) no authentication Aug 31 11:54:09 kingfish syslogd 1.4.1: restart. Aug 31 11:54:10 kingfish /etc/hotplug/usb.rc[463]: loaded HCD: usb-ohci Aug 31 11:54:14 kingfish kernel: klogd 1.4.1, log source = /proc/kmsg started. Aug 31 11:54:14 kingfish kernel: Inspecting /boot/System.map-2.4.21-241-smp Aug 31 11:54:14 kingfish kernel: Loaded 14971 symbols from /boot/System.map-2.4.21-241-smp. Aug 31 11:54:14 kingfish kernel: Symbols match kernel version 2.4.21. Aug 31 11:54:14 kingfish kernel: Error seeking in /dev/kmem Aug 31 11:54:14 kingfish kernel: Symbol #st, value a011f000 Aug 31 11:54:14 kingfish kernel: Error adding kernel module table entry. Aug 31 11:54:14 kingfish kernel: ip_tables: (C) 2000-2002 Netfilter core team Aug 31 11:54:14 kingfish kernel: ip_conntrack version 2.1 (8192 buckets, 65536 max) - 448 bytes per conntrack Aug 31 11:54:14 kingfish kernel: Broadcom Gigabit Ethernet Driver bcm5700 with Broadcom NIC Extension (NICE) ver. 6.2.17 (07/14/03) Aug 31 11:54:14 kingfish kernel: eth0: Broadcom BCM5704 1000Base-T found at mem fc9c0000, IRQ 24, node addr 00e081527800 Aug 31 11:54:14 kingfish kernel: eth0: Broadcom BCM5704 Integrated Copper transceiver found Aug 31 11:54:14 kingfish kernel: eth0: Scatter-gather ON, 64-bit DMA ON, Tx Checksum ON, Rx Checksum ON, 802.1Q VLAN ON Aug 31 11:54:14 kingfish kernel: eth1: Broadcom BCM5704 1000Base-T found at mem fc9f0000, IRQ 25, node addr 00e081527801 Aug 31 11:54:14 kingfish kernel: eth1: Broadcom BCM5704 Integrated Copper transceiver found Aug 31 11:54:14 kingfish kernel: eth1: Scatter-gather ON, 64-bit DMA ON, Tx Checksum ON, Rx Checksum ON, 802.1Q VLAN ON Aug 31 11:54:14 kingfish kernel: usb.c: registered new driver usbdevfs Aug 31 11:54:14 kingfish kernel: usb.c: registered new driver hub Aug 31 11:54:14 kingfish kernel: usb-uhci.c: $Revision: 1.275 $ time 14:46:30 Aug 10 2004 Aug 31 11:54:14 kingfish kernel: usb-uhci.c: High bandwidth mode enabled Aug 31 11:54:14 kingfish kernel: usb-uhci.c: v1.275:USB Universal Host Controller Interface driver Aug 31 11:54:14 kingfish kernel: uhci.c: USB Universal Host Controller Interface driver v1.1 Aug 31 11:54:14 kingfish kernel: usb-ohci.c: USB OHCI at membase 0xffffff0000eaf000, IRQ 19 Aug 31 11:54:14 kingfish kernel: usb-ohci.c: usb-03:00.0, PCI device 1022:7464 Aug 31 11:54:14 kingfish kernel: usb.c: new USB bus registered, assigned bus number 1 Aug 31 11:54:14 kingfish kernel: hub.c: USB hub found Aug 31 11:54:14 kingfish kernel: hub.c: 3 ports detected Aug 31 11:54:14 kingfish kernel: usb-ohci.c: USB OHCI at membase 0xffffff0000ed6000, IRQ 19 Aug 31 11:54:14 kingfish kernel: usb-ohci.c: usb-03:00.1, PCI device 1022:7464 Aug 31 11:54:14 kingfish kernel: usb.c: new USB bus registered, assigned bus number 2 Aug 31 11:54:14 kingfish kernel: hub.c: USB hub found Aug 31 11:54:14 kingfish kernel: hub.c: 3 ports detected Aug 31 11:54:14 kingfish kernel: bcm5700: eth1 NIC Link is UP, 100 Mbps half duplex Aug 31 11:54:15 kingfish master[682]: setrlimit: Unable to set file descriptors limit to -1: Operation not permitted Aug 31 11:54:15 kingfish master[682]: retrying with 1024 (current max) Aug 31 11:54:15 kingfish master[682]: process started Aug 31 11:54:15 kingfish master[683]: about to exec /usr/lib/cyrus/bin/ctl_cyrusdb Aug 31 11:54:15 kingfish master[682]: process started Aug 31 11:54:15 kingfish master[683]: about to exec /usr/lib/cyrus/bin/ctl_cyrusdb Aug 31 11:54:15 kingfish ctl_cyrusdb[683]: recovering cyrus databases Aug 31 11:54:15 kingfish saslauthd[702]: START: saslauthd 2.1.12 Aug 31 11:54:15 kingfish saslauthd[710]: master PID is: 710 Aug 31 11:54:15 kingfish saslauthd[710]: daemon started, listening on /var/run/sasl2//mux Aug 31 11:54:16 kingfish kernel: IPsec Security Association Database (SADB): initialized. Aug 31 11:54:16 kingfish kernel: IPsec Security Policy Database (SPD): initialized. Aug 31 11:54:16 kingfish kernel: IPsec PF_KEY V2: initialized Aug 31 11:54:16 kingfish kernel: IPv6 v0.8 (usagi-cvs/IPsec6 based StS) for NET4.0 Aug 31 11:54:16 kingfish kernel: IPv6 over IPv4 tunneling driver Aug 31 11:54:16 kingfish sshd[730]: Server listening on :: port 22. Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/in.rshd is not executable [line=100] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=100] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/in.rlogind is not executable [line=111] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=111] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/in.fingerd is not executable [line=123] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=123] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/imapd is not executable [line=135] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=135] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/imapd is not executable [line=148] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=148] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/ipop3d is not executable [line=161] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=161] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/ipop3d is not executable [line=173] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=173] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/in.talkd is not executable [line=195] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=195] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/in.talkd is not executable [line=205] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=205] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/in.identd is not executable [line=340] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=340] Aug 31 11:54:19 kingfish xinetd[842]: Server /usr/sbin/leafnode is not executable [line=349] Aug 31 11:54:19 kingfish xinetd[842]: Error parsing attribute server - DISABLING SERVICE [line=349] Aug 31 11:54:19 kingfish xinetd[842]: removing shell Aug 31 11:54:19 kingfish xinetd[842]: removing login Aug 31 11:54:19 kingfish xinetd[842]: removing finger Aug 31 11:54:19 kingfish xinetd[842]: removing imap Aug 31 11:54:19 kingfish xinetd[842]: removing imaps Aug 31 11:54:19 kingfish xinetd[842]: removing pop3 Aug 31 11:54:19 kingfish xinetd[842]: removing pop3s Aug 31 11:54:19 kingfish xinetd[842]: removing ntalk Aug 31 11:54:19 kingfish xinetd[842]: removing talk Aug 31 11:54:19 kingfish xinetd[842]: removing ident Aug 31 11:54:19 kingfish xinetd[842]: removing nntp Aug 31 11:54:19 kingfish xinetd[842]: xinetd Version 20030122 started with libwrap loadavg options compiled in. Aug 31 11:54:19 kingfish xinetd[842]: Started working: 2 available services Aug 31 11:54:22 kingfish kernel: keyboard: Timeout - AT keyboard not present?(f4) Aug 31 11:54:25 kingfish last message repeated 2 times Aug 31 11:54:25 kingfish /etc/hotplug/net.agent[757]: No HW description found ... exiting Aug 31 11:54:26 kingfish kernel: eth1: no IPv6 routers present Aug 31 11:54:26 kingfish kernel: keyboard: Timeout - AT keyboard not present?(f4) Aug 31 11:54:28 kingfish ctl_cyrusdb[683]: skiplist: recovered /var/lib/imap/mailboxes.db (43 records, 16796 bytes) in 0 seconds Aug 31 11:54:28 kingfish ctl_cyrusdb[683]: done recovering cyrus databases Aug 31 11:54:28 kingfish master[682]: ready for work Aug 31 11:54:28 kingfish master[995]: about to exec /usr/lib/cyrus/bin/tls_prune Aug 31 11:54:28 kingfish master[996]: about to exec /usr/lib/cyrus/bin/ctl_deliver Aug 31 11:54:28 kingfish master[997]: about to exec /usr/lib/cyrus/bin/ctl_cyrusdb Aug 31 11:54:28 kingfish ctl_cyrusdb[997]: checkpointing cyrus databases Aug 31 11:54:28 kingfish ctl_cyrusdb[997]: archiving database file: /var/lib/imap/mailboxes.db Aug 31 11:54:28 kingfish ctl_deliver[996]: duplicate_prune: pruning back 3 days Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483651 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483651 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483654 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483654 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483655 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483655 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483656 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483656 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483657 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483657 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483658 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483657 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483658 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483658 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483659 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483659 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483660 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483660 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483662 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483662 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483661 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483663 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483661 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483663 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483664 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483665 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483664 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483665 Aug 31 11:54:28 kingfish ctl_cyrusdb[997]: archiving log file: /var/lib/imap/db/log.0000000005 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483666 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483667 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483666 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483667 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: starting txn 2147483668 Aug 31 11:54:28 kingfish ctl_cyrusdb[997]: done checkpointing cyrus databases Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483669 Aug 31 11:54:28 kingfish ctl_deliver[996]: mydelete: committing txn 2147483668 Aug 31 11:54:28 kingfish master[682]: process 997 exited, status 0 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483669 Aug 31 11:54:28 kingfish ctl_deliver[996]: duplicate_prune: purged 12 out of 165 entries Aug 31 11:54:28 kingfish master[682]: process 996 exited, status 0 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483670 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483670 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483671 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483671 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483672 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483672 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483673 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483673 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483674 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483674 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483675 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483675 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483676 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483676 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483677 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483677 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: starting txn 2147483678 Aug 31 11:54:28 kingfish tls_prune[995]: mydelete: committing txn 2147483678 Aug 31 11:54:28 kingfish tls_prune[995]: tls_prune: purged 14 out of 90 entries Aug 31 11:54:28 kingfish master[682]: process 995 exited, status 0 Aug 31 11:54:29 kingfish kernel: keyboard: Timeout - AT keyboard not present?(f4) Aug 31 11:54:32 kingfish last message repeated 2 times Aug 31 11:54:33 kingfish kernel: ide-floppy driver 0.99.newide Aug 31 11:54:33 kingfish kernel: hda: attached ide-cdrom driver. Aug 31 11:54:33 kingfish kernel: hda: ATAPI 52X CD-ROM drive, 120kB Cache Aug 31 11:54:33 kingfish kernel: Uniform CD-ROM driver Revision: 3.12 Aug 31 11:54:39 kingfish kernel: cdrom: open failed. Aug 31 11:54:52 kingfish /usr/sbin/cron[1251]: (CRON) STARTUP (fork ok) Aug 31 11:55:36 kingfish sshd[1341]: Could not reverse map address 130.13.89.148. Aug 31 11:56:10 kingfish master[1346]: about to exec /usr/lib/cyrus/bin/pop3d Aug 31 11:56:10 kingfish pop3s[1346]: executed Aug 31 11:56:10 kingfish pop3d[1346]: accepted connection Aug 31 11:56:12 kingfish pop3d[1346]: starttls: TLSv1 with cipher RC4-MD5 (128/128 bits reused) no authentication Aug 31 11:56:12 kingfish saslauthd[711]: pam_sm_authenticate called.