On 2006-01-08 09:48 Anders Norrbring wrote:
On 2006-01-05 03:34 Scott Leighton wrote:
On Wednesday 04 January 2006 12:39 am, Anders Norrbring wrote:
I have a server running SuSE 9.0 that dies almost every morning around 5.15.. I would need some help in isolating the error so I can fix it.
This is gonna be a long mail with log file extracts, please bare with me..
Any ideas on how to track this down would be greatly appreciated!
If you haven't already, I'd check these areas.
1) cron, see what, if anything, fires off around 5:15am
2) your apache error and access logs, see if there's anything in them around that time.
I'd probably also try to capture the ps output at around 5:14am to see if the process mentioned in the log entry provides any clue.
Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9
Hopefully someone else more experienced will have some better ideas for you, but the above is where I'd start if it were my issue.
Scott
Now it happened again... This time it looks as the problem begins here:
Jan 8 04:34:25 iris pop3d[5739]: login: 812161634-SE-KEMAB.host.songnetworks.se[81.216.16.34] mpb0011 plaintext Jan 8 04:35:35 iris master[1029]: process 5739 exited, status 0 Jan 8 iris kernel: Loaded 21361 symbols from /boot/System.map-2.4.21-297-default. Jan 8 04:39:43 iris kernel: Symbols match kernel version 2.4.21. Jan 8 04:40:29 iris master[5760]: about to exec /usr/lib/cyrus/bin/pop3d Jan 8 04:40:59 iris kernel: Loaded 499 symbols from 22 modules. Jan 8 04:42:19 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 8 04:42:19 iris pop3[5760]: executed Jan 8 04:42:19 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 8 04:42:21 iris master[5767]: about to exec /usr/lib/cyrus/bin/imapd Jan 8 04:43:41 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 8 04:52:19 iris syslogd: /var/log/mail.warn: Cannot allocate memory
In the mail.warn log, I see this: Jan 8 04:39:15 iris postfix/master[1713]: warning: process /usr/lib/postfix/bounce pid 5763 killed by signal 9 Jan 8 04:39:16 iris postfix/master[1713]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling And in the warn log, I grep'ed everything with the word 'warning:' and came up with this: Jan 8 04:34:53 iris postfix/smtpd[5607]: warning: 24-151-7-191.static.nwtn.ct.charter.com[24.151.7.191] sent non-SMTP command: From: "Tiul fjrfmnlzedouh Vsfwekahel" <azpkb@elchavo.com> Jan 8 04:39:02 iris postfix/smtpd[5572]: warning: pcp05206190pcs.sanarb01.mi.comcast.net[68.42.214.215] sent non-SMTP command: From: "Gunz berg"<rogert@yahoo.com> Jan 8 04:39:15 iris postfix/master[1713]: warning: process /usr/lib/postfix/bounce pid 5763 killed by signal 9 Jan 8 04:39:16 iris postfix/master[1713]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling Jan 8 04:39:55 iris postfix/smtpd[5607]: warning: smtpd_peer_init: 24.144.39.75: hostname dhcp39-75.cable.conwaycorp.net verification fail ed: Name or service not known Jan 8 04:42:21 iris postfix/master[1713]: warning: process /usr/lib/postfix/smtpd pid 5764 killed by signal 9 Jan 8 04:45:57 iris postfix/master[1713]: warning: /usr/lib/postfix/smtpd: bad command startup -- throttling Jan 8 04:57:38 iris postfix/master[1713]: warning: process /usr/lib/postfix/smtpd pid 5784 killed by signal 9 Jan 8 04:57:38 iris postfix/master[1713]: warning: /usr/lib/postfix/smtpd: bad command startup -- throttling Jan 8 04:57:38 iris postfix/master[1713]: warning: process /usr/lib/postfix/bounce pid 5766 killed by signal 9 Jan 8 04:57:38 iris postfix/master[1713]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling Jan 8 04:57:38 iris postfix/master[1713]: warning: process /usr/lib/postfix/pickup pid 3703 exit status 1 Jan 8 05:08:19 iris postfix/master[1713]: warning: process /usr/lib/postfix/trivial-rewrite pid 5787 killed by signal 9 Jan 8 05:08:29 iris postfix/master[1713]: warning: /usr/lib/postfix/trivial-rewrite: bad command startup -- throttling Jan 8 05:22:16 iris postfix/master[1713]: warning: process /usr/lib/postfix/bounce pid 5806 killed by signal 9 Jan 8 05:31:00 iris postfix/master[1713]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling Jan 8 05:37:23 iris postfix/lmtp[5671]: warning: timeout on private/defer socket while reading input attribute name Jan 8 05:37:24 iris postfix/master[1713]: warning: process /usr/lib/postfix/smtp pid 5811 killed by signal 9 Jan 8 05:37:25 iris postfix/lmtp[5671]: warning: 7957C49D9: defer service failure Jan 8 05:38:42 iris postfix/smtp[5660]: warning: timeout on private/defer socket while reading input attribute name Jan 8 05:44:37 iris postfix/smtp[5660]: warning: BC5CE49B4: defer service failure Jan 8 05:48:24 iris postfix/lmtp[5702]: warning: timeout on private/defer socket while reading input attribute name Jan 8 05:59:17 iris postfix/lmtp[5702]: warning: 1B23B45D7: defer service failure Jan 8 05:59:38 iris postfix/smtpd[5716]: warning: timeout on public/cleanup socket while reading input attribute name Jan 8 06:02:17 iris postfix/smtpd[5572]: warning: timeout on public/cleanup socket while reading input attribute name Jan 8 06:03:30 iris postfix/smtp[5798]: warning: premature end-of-input on private/defer socket while reading input attribute name Jan 8 06:03:30 iris postfix/lmtp[5671]: warning: premature end-of-input on public/flush socket while reading input attribute name Jan 8 06:03:30 iris postfix/smtp[5798]: warning: 00DC7495F: defer service failure Jan 8 06:03:30 iris postfix/lmtp[5671]: warning: 7957C49D9: flush service failure Jan 8 06:03:31 iris postfix/lmtp[5702]: warning: 1B23B45D7: flush service failure After that, the box was dead hung. -- Anders Norrbring Norrbring Consulting