I have a server running SuSE 9.0 that dies almost every morning around 5.15.. I would need some help in isolating the error so I can fix it. This is gonna be a long mail with log file extracts, please bare with me.. Any ideas on how to track this down would be greatly appreciated! Here's today's "warn" log file: ........ Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9 Jan 4 05:23:24 iris postfix/master[2376]: warning: unix_trigger_event: read timeout for service public/flush Jan 4 05:23:28 iris kernel: ldt allocation failed Jan 4 05:23:28 iris kernel: VM: killing process httpd2-prefork Jan 4 05:28:06 iris master[1030]: process 17688 exited, signaled to death by 9 Jan 4 05:30:26 iris postfix/master[2376]: warning: process /usr/lib/postfix/flush pid 29324 killed by signal 9 Jan 4 05:30:26 iris postfix/master[2376]: warning: /usr/lib/postfix/flush: bad command startup -- throttling Jan 4 05:30:26 iris postfix/master[2376]: warning: process /usr/lib/postfix/smtpd pid 29322 killed by signal 9 Jan 4 05:30:26 iris postfix/master[2376]: warning: /usr/lib/postfix/smtpd: bad command startup -- throttling Jan 4 05:32:34 iris postfix/master[2376]: warning: process /usr/lib/postfix/qmgr pid 2383 killed by signal 9 Jan 4 05:35:50 iris postfix/master[2376]: warning: unix_trigger_event: read timeout for service public/flush Jan 4 05:49:23 iris master[1030]: process 29326 exited, signaled to death by 9 Jan 4 05:52:30 iris postfix/master[2376]: warning: unix_trigger_event: read timeout for service public/flush Jan 4 05:59:06 iris master[1030]: process 29374 exited, signaled to death by 9 Jan 4 05:59:35 iris master[1030]: process 29327 exited, signaled to death by 9 Jan 4 06:01:17 iris master[1030]: process 29369 exited, signaled to death by 9 Jan 4 06:03:03 iris master[1030]: process 29325 exited, signaled to death by 9 Jan 4 06:06:00 iris postfix/master[2376]: warning: process /usr/lib/postfix/proxymap pid 29375 killed by signal 9 Jan 4 06:06:00 iris postfix/master[2376]: warning: /usr/lib/postfix/proxymap: bad command startup -- throttling Jan 4 06:10:40 iris master[29377]: fatal: master_spawn: exec /usr/lib/postfix/trivial-rewrite: Cannot allocate memory Jan 4 06:11:51 iris postfix/master[2376]: warning: process /usr/lib/postfix/flush pid 29372 killed by signal 9 Jan 4 06:11:51 iris postfix/master[2376]: warning: /usr/lib/postfix/flush: bad command startup -- throttling Jan 4 06:12:25 iris postfix/master[2376]: warning: unix_trigger_event: read timeout for service public/flush Jan 4 06:17:31 iris postfix/master[2376]: warning: process /usr/lib/postfix/flush pid 29381 killed by signal 9 Jan 4 06:17:31 iris postfix/master[2376]: warning: /usr/lib/postfix/flush: bad command startup -- throttling Jan 4 06:17:31 iris postfix/master[2376]: warning: process /usr/lib/postfix/pickup pid 29379 killed by signal 9 Jan 4 06:17:31 iris postfix/master[2376]: warning: /usr/lib/postfix/pickup: bad command startup -- throttling Jan 4 06:17:59 iris postfix/master[2376]: warning: process /usr/lib/postfix/trivial-rewrite pid 29377 killed by signal 9 Jan 4 06:17:59 iris postfix/master[2376]: warning: /usr/lib/postfix/trivial-rewrite: bad command startup -- throttling Jan 4 06:20:29 iris master[29383]: fatal: master_spawn: exec /usr/lib/postfix/pickup: Cannot allocate memory Jan 4 06:23:08 iris postfix/qmgr[29348]: fatal: watchdog timeout Jan 4 06:24:08 iris postfix/lmtp[29239]: warning: premature end-of-input on public/flush socket while reading input attribute name Jan 4 06:24:08 iris postfix/lmtp[29239]: warning: 058A2C67F: flush service failure Jan 4 06:25:43 iris postfix/smtpd[29346]: warning: premature end-of-input on private/proxymap socket while reading input attribute name Jan 4 06:25:43 iris postfix/smtpd[29346]: warning: private/proxymap socket: service dict_proxy_open: Connection reset by peer Jan 4 06:25:44 iris postfix/smtpd[29346]: warning: connect #1 to subsystem private/proxymap: Connection refused Jan 4 06:25:45 iris kernel: VM: killing process imapd Jan 4 06:25:45 iris kernel: VM: killing process python Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process nmbd Jan 4 06:25:45 iris kernel: VM: killing process flush Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process smtpd Jan 4 06:25:45 iris kernel: VM: killing process imapd Jan 4 06:25:45 iris kernel: VM: killing process cron Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process cron Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process cron Jan 4 06:25:45 iris kernel: VM: killing process qmgr Jan 4 06:25:45 iris kernel: VM: killing process java Jan 4 06:25:45 iris kernel: VM: killing process cron Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process webalizer Jan 4 06:25:45 iris kernel: VM: killing process python Jan 4 06:25:45 iris kernel: ldt allocation failed Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process amavis-stats Jan 4 06:25:45 iris kernel: VM: killing process python Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process amavis-stats Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process imapd Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process webalizer Jan 4 06:25:45 iris kernel: VM: killing process python Jan 4 06:25:45 iris kernel: VM: killing process master Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process pop3d Jan 4 06:25:45 iris kernel: VM: killing process amavis-stats Jan 4 06:25:45 iris kernel: VM: killing process ctl_cyrusdb Jan 4 06:25:45 iris kernel: VM: killing process webalizer Jan 4 06:25:45 iris kernel: VM: killing process ctl_cyrusdb Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process proxymap Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: VM: killing process webalizer Jan 4 06:25:45 iris kernel: VM: killing process flush Jan 4 06:25:45 iris kernel: VM: killing process pickup Jan 4 06:25:45 iris kernel: VM: killing process master Jan 4 06:25:45 iris last message repeated 3 times Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:48 iris kernel: VM: killing process webalizer Jan 4 06:25:48 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:54 iris postfix/smtpd[29346]: warning: connect #2 to subsystem private/proxymap: Connection refused Jan 4 06:26:04 iris postfix/smtpd[29346]: warning: connect #3 to subsystem private/proxymap: Connection refused Jan 4 06:26:14 iris postfix/smtpd[29346]: warning: connect #4 to subsystem private/proxymap: Connection refused Jan 4 06:29:13 iris syslogd: /var/log/mail.info: Cannot allocate memory Jan 4 06:26:24 iris postfix/smtpd[29346]: warning: connect #5 to subsystem private/proxymap: Connection refused Jan 4 06:26:34 iris postfix/smtpd[29346]: warning: connect #6 to subsystem private/proxymap: Connection refused Jan 4 06:26:44 iris postfix/smtpd[29346]: warning: connect #7 to subsystem private/proxymap: Connection refused Jan 4 06:26:54 iris postfix/smtpd[29346]: warning: connect #8 to subsystem private/proxymap: Connection refused Jan 4 06:27:04 iris postfix/smtpd[29346]: warning: connect #9 to subsystem private/proxymap: Connection refused Jan 4 06:27:14 iris postfix/smtpd[29346]: warning: connect #10 to subsystem private/proxymap: Connection refused Jan 4 06:27:24 iris postfix/smtpd[29346]: fatal: connect #11 to subsystem private/proxymap: Connection refused Jan 4 06:27:47 iris kernel: VM: killing process webalizer Jan 4 06:29:08 iris kernel: VM: killing process httpd2-prefork Jan 4 06:31:04 iris kernel: VM: killing process httpd2-prefork Jan 4 06:31:43 iris kernel: VM: killing process python Jan 4 06:33:08 iris kernel: VM: killing process httpd2-prefork Jan 4 06:43:48 iris kernel: VM: killing process nod32d Jan 4 06:44:38 iris kernel: VM: killing process webalizer Jan 4 06:55:38 iris kernel: VM: killing process master Jan 4 07:01:40 iris kernel: VM: killing process nod32d Jan 4 07:02:03 iris kernel: VM: killing process webalizer Jan 4 07:04:51 iris kernel: VM: killing process ctl_cyrusdb Jan 4 07:12:33 iris kernel: VM: killing process httpd2-prefork Jan 4 07:13:35 iris syslogd: /var/log/messages: Cannot allocate memory Jan 4 07:17:26 iris kernel: VM: killing process httpd2-prefork Jan 4 07:28:06 iris kernel: VM: killing process webalizer Jan 4 07:30:23 iris kernel: VM: killing process webalizer Jan 4 07:36:19 iris kernel: VM: killing process webalizer Jan 4 07:41:17 iris kernel: VM: killing process webalizer Jan 4 07:47:36 iris kernel: VM: killing process webalizer Jan 4 07:47:57 iris kernel: VM: killing process httpd2-prefork ................. And then from todays messages: ....... Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9 Jan 4 05:20:16 iris master[29326]: about to exec /usr/lib/cyrus/bin/imapd Jan 4 05:20:16 iris master[29327]: about to exec /usr/lib/cyrus/bin/pop3d Jan 4 05:20:16 iris master[29325]: about to exec /usr/lib/cyrus/bin/ctl_cyrusdb Jan 4 05:22:49 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f2/0) Jan 4 05:23:28 iris kernel: ldt allocation failed Jan 4 05:23:28 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 05:23:28 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 05:23:28 iris kernel: VM: killing process httpd2-prefork Jan 4 05:23:28 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 05:23:28 iris kernel: klogd 1.4.1, ---------- state change ---------- Jan 4 05:28:06 iris master[1030]: process 17688 exited, signaled to death by 9 Jan 4 05:29:39 iris /USR/SBIN/CRON[29335]: (root) CMD (/etc/webalizer/stats > /dev/null 2>&1) Jan 4 05:28:12 iris imap[29326]: executed Jan 4 05:30:16 iris /USR/SBIN/CRON[29337]: (mailman) CMD (/usr/bin/python -S /usr/lib/mailman/cron/gate_news) Jan 4 05:30:18 iris /USR/SBIN/CRON[29338]: (mailman) CMD (/usr/bin/python -S /usr/lib/mailman/cron/gate_news) Jan 4 05:32:34 iris pop3[29327]: executed Jan 4 05:36:24 iris /USR/SBIN/CRON[29349]: (vscan) CMD ( [ -x /usr/local/sbin/amavis-stats ] && /usr/local/sbin/amavis-stats -l /var/log/amavis.log) Jan 4 05:36:28 iris /USR/SBIN/CRON[29350]: (mailman) CMD (/usr/bin/python -S /usr/lib/mailman/cron/gate_news) Jan 4 05:36:32 iris ctl_cyrusdb[29325]: checkpointing cyrus databases Jan 4 05:48:25 iris master[29369]: about to exec /usr/lib/cyrus/bin/ctl_cyrusdb Jan 4 05:48:26 iris ctl_cyrusdb[29369]: checkpointing cyrus databases Jan 4 05:49:23 iris master[1030]: process 29326 exited, signaled to death by 9 Jan 4 05:51:33 iris kernel: Inspecting /boot/System.map-2.4.21-297-default Jan 4 05:52:12 iris pop3d[29327]: accepted connection Jan 4 05:59:06 iris master[1030]: process 29374 exited, signaled to death by 9 Jan 4 05:59:35 iris master[1030]: process 29327 exited, signaled to death by 9 Jan 4 06:01:17 iris master[1030]: process 29369 exited, signaled to death by 9 Jan 4 06:03:03 iris master[1030]: process 29325 exited, signaled to death by 9 Jan 4 06:16:32 iris -- MARK -- Jan 4 06:25:44 iris kernel: Loaded 21361 symbols from /boot/System.map-2.4.21-297-default. Jan 4 06:25:44 iris kernel: Symbols match kernel version 2.4.21. Jan 4 06:25:44 iris kernel: Loaded 499 symbols from 22 modules. Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process imapd Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process python Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process nmbd Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process flush Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process smtpd Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process imapd Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process cron Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process cron Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process cron Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process qmgr Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process java Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process cron Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris last message repeated 3 times Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process webalizer Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process python Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1f2/0) Jan 4 06:25:45 iris kernel: ldt allocation failed Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process amavis-stats Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process python Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process amavis-stats Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process imapd Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process webalizer Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process python Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process master Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process pop3d Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process amavis-stats Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process ctl_cyrusdb Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process webalizer Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process ctl_cyrusdb Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process proxymap Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process webalizer Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris last message repeated 2 times Jan 4 06:25:45 iris kernel: VM: killing process flush Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process pickup Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process master Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process master Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process master Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris last message repeated 2 times Jan 4 06:25:45 iris kernel: VM: killing process master Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:45 iris kernel: VM: killing process httpd2-prefork Jan 4 06:25:45 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:48 iris kernel: VM: killing process webalizer Jan 4 06:25:48 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:25:48 iris kernel: VM: killing process httpd2-prefork Jan 4 06:29:13 iris syslogd: /var/log/mail.info: Cannot allocate memory Jan 4 06:26:47 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:27:47 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:27:47 iris kernel: VM: killing process webalizer Jan 4 06:28:10 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:29:08 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:29:08 iris kernel: VM: killing process httpd2-prefork Jan 4 06:31:04 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:31:04 iris kernel: VM: killing process httpd2-prefork Jan 4 06:31:43 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:31:43 iris kernel: VM: killing process python Jan 4 06:33:08 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:33:08 iris kernel: VM: killing process httpd2-prefork Jan 4 06:34:04 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:37:27 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:38:37 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:43:48 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:43:48 iris kernel: VM: killing process nod32d Jan 4 06:44:21 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:44:38 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:44:38 iris kernel: VM: killing process webalizer Jan 4 06:48:39 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:51:24 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:51:39 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:53:09 iris master[29391]: about to exec /usr/lib/cyrus/bin/ctl_cyrusdb Jan 4 06:55:00 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 06:55:38 iris last message repeated 2 times Jan 4 06:55:38 iris last message repeated 2 times Jan 4 06:55:38 iris kernel: VM: killing process master Jan 4 06:56:22 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 07:00:06 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 07:01:02 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 07:01:40 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 07:01:40 iris kernel: VM: killing process nod32d Jan 4 07:02:03 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 07:02:03 iris kernel: VM: killing process webalizer Jan 4 07:04:51 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 07:04:51 iris kernel: VM: killing process ctl_cyrusdb Jan 4 07:07:43 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 07:09:39 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 07:09:59 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 4 07:12:33 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) ------------------ -- Anders Norrbring Norrbring Consulting
On Wednesday 04 January 2006 12:39 am, Anders Norrbring wrote:
I have a server running SuSE 9.0 that dies almost every morning around 5.15.. I would need some help in isolating the error so I can fix it.
This is gonna be a long mail with log file extracts, please bare with me..
Any ideas on how to track this down would be greatly appreciated!
If you haven't already, I'd check these areas. 1) cron, see what, if anything, fires off around 5:15am 2) your apache error and access logs, see if there's anything in them around that time. I'd probably also try to capture the ps output at around 5:14am to see if the process mentioned in the log entry provides any clue. Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9 Hopefully someone else more experienced will have some better ideas for you, but the above is where I'd start if it were my issue. Scott -- POPFile, the OpenSource EMail Classifier http://popfile.sourceforge.net/ Linux 2.6.11.4-21.10-default x86_64 SuSE Linux 9.3 (x86-64)
Scott Leighton wrote:
On Wednesday 04 January 2006 12:39 am, Anders Norrbring wrote:
I have a server running SuSE 9.0 that dies almost every morning around 5.15.. I would need some help in isolating the error so I can fix it.
This is gonna be a long mail with log file extracts, please bare with me..
Any ideas on how to track this down would be greatly appreciated!
If you haven't already, I'd check these areas.
1) cron, see what, if anything, fires off around 5:15am
2) your apache error and access logs, see if there's anything in them around that time.
I'd probably also try to capture the ps output at around 5:14am to see if the process mentioned in the log entry provides any clue.
Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9
Hmmm, good ideas, but 9? You might also df, and see if any of your partitions is running out of space. mark -- America was *not* built by "rugged individualists". - whitroth "We must all hang together, or we shall all hang separately" - B. Franklin
On 2006-01-05 04:40 mark wrote:
Scott Leighton wrote:
On Wednesday 04 January 2006 12:39 am, Anders Norrbring wrote:
I have a server running SuSE 9.0 that dies almost every morning around 5.15.. I would need some help in isolating the error so I can fix it.
This is gonna be a long mail with log file extracts, please bare with me..
Any ideas on how to track this down would be greatly appreciated!
If you haven't already, I'd check these areas.
1) cron, see what, if anything, fires off around 5:15am
2) your apache error and access logs, see if there's anything in them around that time.
I'd probably also try to capture the ps output at around 5:14am to see if the process mentioned in the log entry provides any clue.
Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9
Hmmm, good ideas, but 9? You might also df, and see if any of your partitions is running out of space.
mark
Thanks guys! I've made a cron ectry that dumps a ps ax list at 5:10, I'll take it from there next time it happens.. (Didn't this morning..) Also. there's lot of space available, 30+ GB on the / partition, and 20GB on the /var.... Anders.
On 2006-01-05 03:34 Scott Leighton wrote:
On Wednesday 04 January 2006 12:39 am, Anders Norrbring wrote:
I have a server running SuSE 9.0 that dies almost every morning around 5.15.. I would need some help in isolating the error so I can fix it.
This is gonna be a long mail with log file extracts, please bare with me..
Any ideas on how to track this down would be greatly appreciated!
If you haven't already, I'd check these areas.
1) cron, see what, if anything, fires off around 5:15am
2) your apache error and access logs, see if there's anything in them around that time.
I'd probably also try to capture the ps output at around 5:14am to see if the process mentioned in the log entry provides any clue.
Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9
Hopefully someone else more experienced will have some better ideas for you, but the above is where I'd start if it were my issue.
Scott
Now it happened again... This time it looks as the problem begins here: Jan 8 04:34:25 iris pop3d[5739]: login: 812161634-SE-KEMAB.host.songnetworks.se[81.216.16.34] mpb0011 plaintext Jan 8 04:35:35 iris master[1029]: process 5739 exited, status 0 Jan 8 iris kernel: Loaded 21361 symbols from /boot/System.map-2.4.21-297-default. Jan 8 04:39:43 iris kernel: Symbols match kernel version 2.4.21. Jan 8 04:40:29 iris master[5760]: about to exec /usr/lib/cyrus/bin/pop3d Jan 8 04:40:59 iris kernel: Loaded 499 symbols from 22 modules. Jan 8 04:42:19 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 8 04:42:19 iris pop3[5760]: executed Jan 8 04:42:19 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 8 04:42:21 iris master[5767]: about to exec /usr/lib/cyrus/bin/imapd Jan 8 04:43:41 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 8 04:52:19 iris syslogd: /var/log/mail.warn: Cannot allocate memory As you can see, the pop3 executes nicely, and 4 minutes later (04:39:43) suddenly there are messages from the kernel? I just don't get it! -- Anders Norrbring Norrbring Consulting
On 2006-01-08 09:48 Anders Norrbring wrote:
On 2006-01-05 03:34 Scott Leighton wrote:
On Wednesday 04 January 2006 12:39 am, Anders Norrbring wrote:
I have a server running SuSE 9.0 that dies almost every morning around 5.15.. I would need some help in isolating the error so I can fix it.
This is gonna be a long mail with log file extracts, please bare with me..
Any ideas on how to track this down would be greatly appreciated!
If you haven't already, I'd check these areas.
1) cron, see what, if anything, fires off around 5:15am
2) your apache error and access logs, see if there's anything in them around that time.
I'd probably also try to capture the ps output at around 5:14am to see if the process mentioned in the log entry provides any clue.
Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9
Hopefully someone else more experienced will have some better ideas for you, but the above is where I'd start if it were my issue.
Scott
Now it happened again... This time it looks as the problem begins here:
Jan 8 04:34:25 iris pop3d[5739]: login: 812161634-SE-KEMAB.host.songnetworks.se[81.216.16.34] mpb0011 plaintext Jan 8 04:35:35 iris master[1029]: process 5739 exited, status 0 Jan 8 iris kernel: Loaded 21361 symbols from /boot/System.map-2.4.21-297-default. Jan 8 04:39:43 iris kernel: Symbols match kernel version 2.4.21. Jan 8 04:40:29 iris master[5760]: about to exec /usr/lib/cyrus/bin/pop3d Jan 8 04:40:59 iris kernel: Loaded 499 symbols from 22 modules. Jan 8 04:42:19 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 8 04:42:19 iris pop3[5760]: executed Jan 8 04:42:19 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 8 04:42:21 iris master[5767]: about to exec /usr/lib/cyrus/bin/imapd Jan 8 04:43:41 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 8 04:52:19 iris syslogd: /var/log/mail.warn: Cannot allocate memory
In the mail.warn log, I see this: Jan 8 04:39:15 iris postfix/master[1713]: warning: process /usr/lib/postfix/bounce pid 5763 killed by signal 9 Jan 8 04:39:16 iris postfix/master[1713]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling And in the warn log, I grep'ed everything with the word 'warning:' and came up with this: Jan 8 04:34:53 iris postfix/smtpd[5607]: warning: 24-151-7-191.static.nwtn.ct.charter.com[24.151.7.191] sent non-SMTP command: From: "Tiul fjrfmnlzedouh Vsfwekahel" <azpkb@elchavo.com> Jan 8 04:39:02 iris postfix/smtpd[5572]: warning: pcp05206190pcs.sanarb01.mi.comcast.net[68.42.214.215] sent non-SMTP command: From: "Gunz berg"<rogert@yahoo.com> Jan 8 04:39:15 iris postfix/master[1713]: warning: process /usr/lib/postfix/bounce pid 5763 killed by signal 9 Jan 8 04:39:16 iris postfix/master[1713]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling Jan 8 04:39:55 iris postfix/smtpd[5607]: warning: smtpd_peer_init: 24.144.39.75: hostname dhcp39-75.cable.conwaycorp.net verification fail ed: Name or service not known Jan 8 04:42:21 iris postfix/master[1713]: warning: process /usr/lib/postfix/smtpd pid 5764 killed by signal 9 Jan 8 04:45:57 iris postfix/master[1713]: warning: /usr/lib/postfix/smtpd: bad command startup -- throttling Jan 8 04:57:38 iris postfix/master[1713]: warning: process /usr/lib/postfix/smtpd pid 5784 killed by signal 9 Jan 8 04:57:38 iris postfix/master[1713]: warning: /usr/lib/postfix/smtpd: bad command startup -- throttling Jan 8 04:57:38 iris postfix/master[1713]: warning: process /usr/lib/postfix/bounce pid 5766 killed by signal 9 Jan 8 04:57:38 iris postfix/master[1713]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling Jan 8 04:57:38 iris postfix/master[1713]: warning: process /usr/lib/postfix/pickup pid 3703 exit status 1 Jan 8 05:08:19 iris postfix/master[1713]: warning: process /usr/lib/postfix/trivial-rewrite pid 5787 killed by signal 9 Jan 8 05:08:29 iris postfix/master[1713]: warning: /usr/lib/postfix/trivial-rewrite: bad command startup -- throttling Jan 8 05:22:16 iris postfix/master[1713]: warning: process /usr/lib/postfix/bounce pid 5806 killed by signal 9 Jan 8 05:31:00 iris postfix/master[1713]: warning: /usr/lib/postfix/bounce: bad command startup -- throttling Jan 8 05:37:23 iris postfix/lmtp[5671]: warning: timeout on private/defer socket while reading input attribute name Jan 8 05:37:24 iris postfix/master[1713]: warning: process /usr/lib/postfix/smtp pid 5811 killed by signal 9 Jan 8 05:37:25 iris postfix/lmtp[5671]: warning: 7957C49D9: defer service failure Jan 8 05:38:42 iris postfix/smtp[5660]: warning: timeout on private/defer socket while reading input attribute name Jan 8 05:44:37 iris postfix/smtp[5660]: warning: BC5CE49B4: defer service failure Jan 8 05:48:24 iris postfix/lmtp[5702]: warning: timeout on private/defer socket while reading input attribute name Jan 8 05:59:17 iris postfix/lmtp[5702]: warning: 1B23B45D7: defer service failure Jan 8 05:59:38 iris postfix/smtpd[5716]: warning: timeout on public/cleanup socket while reading input attribute name Jan 8 06:02:17 iris postfix/smtpd[5572]: warning: timeout on public/cleanup socket while reading input attribute name Jan 8 06:03:30 iris postfix/smtp[5798]: warning: premature end-of-input on private/defer socket while reading input attribute name Jan 8 06:03:30 iris postfix/lmtp[5671]: warning: premature end-of-input on public/flush socket while reading input attribute name Jan 8 06:03:30 iris postfix/smtp[5798]: warning: 00DC7495F: defer service failure Jan 8 06:03:30 iris postfix/lmtp[5671]: warning: 7957C49D9: flush service failure Jan 8 06:03:31 iris postfix/lmtp[5702]: warning: 1B23B45D7: flush service failure After that, the box was dead hung. -- Anders Norrbring Norrbring Consulting
On Sunday 08 January 2006 12:48 am, Anders Norrbring wrote:
0-order allocation failed
Some googling reveals that message means the kernel couldn't allocate even one page of memory. What's your memory situation on that box, how much swap space? Some of the google results also mention reiserfs issues related to that message, but it is all pretty vague. Scott -- POPFile, the OpenSource EMail Classifier http://popfile.sourceforge.net/ Linux 2.6.11.4-21.10-default x86_64 SuSE Linux 9.3 (x86-64)
On Sunday 08 January 2006 11:59, Scott Leighton wrote:
On Sunday 08 January 2006 12:48 am, Anders Norrbring wrote:
0-order allocation failed
Some googling reveals that message means the kernel couldn't allocate even one page of memory. What's your memory situation on that box, how much swap space?
Some of the google results also mention reiserfs issues related to that message, but it is all pretty vague.
Hi Scott, Hi Anders... OK, if we're going the Google research route, I also did some searching out of curiosity and what I read indicated the same thing... the kernel starts shutting down processes to preserve itself when it is running out of memory. There were no runaway incremental PIDs being generated, so it seems reasonable to suspect the problem is related to a specific application and not the kernel or overall system health. Also, one of the original errors that you posted, Anders, pertained to "fast flush". AIUI, the mail server maintains a cache of addresses with pointers to mails that fail to deliver and are stored so sending can be attempted again repeatedly until either a "give it up" point is reached or the mail successfully sends. It looked like I was digging up situations with that error message where the cache and/or stored mails were growing too large and/or too quickly and/or failing to flush correctly... possibly DOS attack and/or spam related. Could be worthless fodder or clues, I don't know. I'm just hoping to be helpful by tossing it into your salad of things to sample... Good luck! - Carl PS: And Happy New Year!
On 2006-01-08 19:47 Carl Hartung wrote:
On Sunday 08 January 2006 11:59, Scott Leighton wrote:
On Sunday 08 January 2006 12:48 am, Anders Norrbring wrote:
0-order allocation failed
Some googling reveals that message means the kernel couldn't allocate even one page of memory. What's your memory situation on that box, how much swap space?
Some of the google results also mention reiserfs issues related to that message, but it is all pretty vague.
Hi Scott, Hi Anders...
OK, if we're going the Google research route, I also did some searching out of curiosity and what I read indicated the same thing... the kernel starts shutting down processes to preserve itself when it is running out of memory. There were no runaway incremental PIDs being generated, so it seems reasonable to suspect the problem is related to a specific application and not the kernel or overall system health.
Also, one of the original errors that you posted, Anders, pertained to "fast flush". AIUI, the mail server maintains a cache of addresses with pointers to mails that fail to deliver and are stored so sending can be attempted again repeatedly until either a "give it up" point is reached or the mail successfully sends. It looked like I was digging up situations with that error message where the cache and/or stored mails were growing too large and/or too quickly and/or failing to flush correctly... possibly DOS attack and/or spam related.
Could be worthless fodder or clues, I don't know. I'm just hoping to be helpful by tossing it into your salad of things to sample...
Good luck!
- Carl
PS: And Happy New Year!
Thanks Carl, I've been suspecting some e-mail related stuff too.. However, I have no idea on how to solve the situation... One "simple" idea would be to cron script a restart of postfix once a day, but I doubt it would be the right way to go... I guess Postfix has some settings to limit that cache, or flush it every now and then, but I'm not very good with the Postfix options.. Anders.
On 2006-01-08 19:47 Carl Hartung wrote:
Hi Scott, Hi Anders...
OK, if we're going the Google research route, I also did some searching out of curiosity and what I read indicated the same thing... the kernel starts shutting down processes to preserve itself when it is running out of memory. There were no runaway incremental PIDs being generated, so it seems reasonable to suspect the problem is related to a specific application and not the kernel or overall system health.
Also, one of the original errors that you posted, Anders, pertained to "fast flush". AIUI, the mail server maintains a cache of addresses with pointers to mails that fail to deliver and are stored so sending can be attempted again repeatedly until either a "give it up" point is reached or the mail successfully sends. It looked like I was digging up situations with that error message where the cache and/or stored mails were growing too large and/or too quickly and/or failing to flush correctly... possibly DOS attack and/or spam related.
Could be worthless fodder or clues, I don't know. I'm just hoping to be helpful by tossing it into your salad of things to sample...
Good luck!
- Carl
PS: And Happy New Year!
I don't know if this means anything, but looking in todays 'messages', I see this: Jan 9 05:04:19 iris master[3467]: about to exec /usr/lib/cyrus/bin/pop3d Jan 9 05:04:19 iris master[1030]: process 2174 exited, signaled to death by 9 Jan 9 05:04:19 iris master[1030]: process 2170 exited, signaled to death by 9 Jan 9 05:04:43 iris master[1030]: process 2165 exited, signaled to death by 9 Jan 9 05:04:51 iris kernel: __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) Jan 9 05:04:52 iris kernel: VM: killing process imapd Jan 9 05:04:52 iris kernel: klogd 1.4.1, ---------- state change ---------- Jan 9 05:07:31 iris master[1030]: process 2223 exited, signaled to death by 9 Jan 9 05:08:08 iris master[1030]: process 3467 exited, signaled to death by 9 Jan 9 05:09:29 iris imapd[2232]: seen_db: user nb0002 opened /var/lib/imap/user/n/nb0002.seen Jan 9 05:09:49 iris kernel: Inspecting /boot/System.map-2.4.21-297-default Jan 9 05:13:17 iris master[3493]: about to exec /usr/lib/cyrus/bin/imapd Jan 9 05:13:17 iris master[1030]: process 2063 exited, signaled to death by 9 Jan 9 05:13:17 iris imap[3493]: executed Jan 9 05:13:17 iris master[3494]: about to exec /usr/lib/cyrus/bin/imapd Jan 9 05:13:17 iris imapd[3493]: accepted connection Jan 9 05:13:17 iris imap[3494]: executed Jan 9 05:13:17 iris imapd[3494]: accepted connection Jan 9 05:13:17 iris master[3495]: about to exec /usr/lib/cyrus/bin/imapd Jan 9 05:13:17 iris imap[3495]: executed Jan 9 05:13:17 iris master[3496]: about to exec /usr/lib/cyrus/bin/imapd Jan 9 05:13:17 iris imap[3496]: executed Jan 9 05:13:17 iris imapd[3495]: accepted connection Jan 9 05:13:17 iris imapd[3496]: accepted connection Then looking in the processlist that was dumped by 'ps' half an hour earlier, all the processes that got "signaled to death" are imapd processes, that would be my Cyrus-IMAP... I don't know if it's relevant, but it somehow "feels" like it could be a Cyrus problem... 1890 ? Ss 0:04 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -DSSL 2041 ? S 0:00 imapd 2049 ? S 0:00 imapd 2051 ? S 0:00 imapd 2061 ? S 0:00 imapd 2063 ? S 0:00 imapd 2155 ? S 0:00 imapd 2156 ? S 0:00 imapd 2157 ? S 0:00 imapd 2158 ? S 0:00 imapd 2159 ? S 0:00 imapd 2160 ? S 0:00 imapd 2161 ? S 0:00 imapd 2162 ? S 0:00 imapd 2163 ? S 0:00 imapd 2164 ? S 0:00 imapd 2165 ? S 0:00 imapd 2166 ? S 0:00 imapd 2167 ? S 0:00 imapd 2168 ? S 0:00 imapd 2169 ? S 0:00 imapd 2170 ? S 0:00 imapd 2171 ? S 0:00 imapd 2172 ? S 0:00 imapd 2173 ? S 0:00 imapd 2174 ? S 0:00 imapd 2175 ? S 0:00 imapd 2176 ? S 0:00 imapd 2177 ? S 0:00 imapd 2178 ? S 0:00 imapd 2179 ? S 0:00 imapd 2222 ? S 0:00 imapd 2223 ? S 0:00 imapd 2224 ? S 0:00 imapd 2225 ? S 0:00 imapd 2232 ? S 0:00 imapd 2857 ? S 0:01 qmgr -l -t fifo -u 3346 ? S 0:00 imapd 3635 ? S 0:00 imapd 3938 ? S 0:00 /usr/lib/samba/classic/smbd -D -s /etc/samba/smb.conf -- Anders Norrbring Norrbring Consulting
[8<]
2857 ? S 0:01 qmgr -l -t fifo -u 3346 ? S 0:00 imapd 3635 ? S 0:00 imapd 3938 ? S 0:00 /usr/lib/samba/classic/smbd -D -s /etc/samba/smb.conf
But then again, 45 minutes after a fresh reboot, I ran top and saw this PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3165 vscan 15 0 438m 384m 29m S 0.0 38.1 0:15.87 amavisd 2470 vscan 15 0 437m 383m 30m S 0.0 38.0 0:18.42 amavisd 3440 wwwrun 15 0 8684 6464 3988 S 0.0 0.6 0:00.15 httpd2-prefork 596 root 19 0 8936 6268 1716 S 0.0 0.6 0:01.35 java 599 root 15 0 8936 6268 1716 S 0.0 0.6 0:00.01 java 600 root 15 0 8936 6268 1716 S 0.0 0.6 0:00.14 java amavisd consuming 76% of memory? Yikes! -- Anders Norrbring Norrbring Consulting
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Monday 2006-01-09 at 09:01 +0100, Anders Norrbring wrote:
But then again, 45 minutes after a fresh reboot, I ran top and saw this
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3165 vscan 15 0 438m 384m 29m S 0.0 38.1 0:15.87 amavisd 2470 vscan 15 0 437m 383m 30m S 0.0 38.0 0:18.42 amavisd 3440 wwwrun 15 0 8684 6464 3988 S 0.0 0.6 0:00.15 httpd2-prefork 596 root 19 0 8936 6268 1716 S 0.0 0.6 0:01.35 java 599 root 15 0 8936 6268 1716 S 0.0 0.6 0:00.01 java 600 root 15 0 8936 6268 1716 S 0.0 0.6 0:00.14 java
amavisd consuming 76% of memory? Yikes!
I read the full thread before popping in, but I was going to tell you from the start that it is an "out of memory" problem, as explained by Carl Hartung. The important message in the first log you posted are here marked with '*': Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9 Jan 4 05:23:24 iris postfix/master[2376]: warning: unix_trigger_event: read timeout forservice public/flush * Jan 4 05:23:28 iris kernel: ldt allocation failed * Jan 4 05:23:28 iris kernel: VM: killing process httpd2-prefork "VM" is virtual memory. The kernel starts killing everything in sight to save himself - and fails doing so, because the system dies. Linux takes very badly being out of memory. How much do you have, by the way? Then you mention that amavis is eating lots of memory - that's where you should look at: limit the number of children that amavis can span, for starters. My guess is that if you did "ps afx" you would have detected lot of children there. I have seen this situation before, caused by amavis and/or spamassassin. Why at certain hour? probably the hour chosen by some spammers to send you problematic emails. Or a virus out there. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDwkMMtTMYHG2NR9URAlLhAJ9V79o+HXGLq5a2HbxHJHrWT5yrkwCdHrIU V6e73H7HnVW2WYXFhCGWbs8= =IBz4 -----END PGP SIGNATURE-----
On 2006-01-09 12:03 Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Monday 2006-01-09 at 09:01 +0100, Anders Norrbring wrote:
But then again, 45 minutes after a fresh reboot, I ran top and saw this
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3165 vscan 15 0 438m 384m 29m S 0.0 38.1 0:15.87 amavisd 2470 vscan 15 0 437m 383m 30m S 0.0 38.0 0:18.42 amavisd 3440 wwwrun 15 0 8684 6464 3988 S 0.0 0.6 0:00.15 httpd2-prefork 596 root 19 0 8936 6268 1716 S 0.0 0.6 0:01.35 java 599 root 15 0 8936 6268 1716 S 0.0 0.6 0:00.01 java 600 root 15 0 8936 6268 1716 S 0.0 0.6 0:00.14 java
amavisd consuming 76% of memory? Yikes!
I read the full thread before popping in, but I was going to tell you from the start that it is an "out of memory" problem, as explained by Carl Hartung.
The important message in the first log you posted are here marked with '*':
Jan 4 05:17:27 iris master[1030]: process 5996 exited, signaled to death by 9 Jan 4 05:23:24 iris postfix/master[2376]: warning: unix_trigger_event: read timeout forservice public/flush * Jan 4 05:23:28 iris kernel: ldt allocation failed * Jan 4 05:23:28 iris kernel: VM: killing process httpd2-prefork
"VM" is virtual memory. The kernel starts killing everything in sight to save himself - and fails doing so, because the system dies. Linux takes very badly being out of memory. How much do you have, by the way?
Then you mention that amavis is eating lots of memory - that's where you should look at: limit the number of children that amavis can span, for starters. My guess is that if you did "ps afx" you would have detected lot of children there.
I have seen this situation before, caused by amavis and/or spamassassin.
Why at certain hour? probably the hour chosen by some spammers to send you problematic emails. Or a virus out there.
Hi and thanks! It seems like it was spamassassin.. When it ran a lint, every bit of memory was grabbed. I have 2GB of memory in the box, so it seemed a little weird at first. However, I'm running an old SA on that box, apparently not well configured either.. :) I found that the bayes database files was almost 16GB in size! When I deleted those, a lint went just fine! At 4.35 in the morning, cron triggers a rules_du_jour update of the SA rules, and after that, a lint is run... I hope this will help, at least until I have the time and inspiration to upgrade it all to SuSE 10.. -- Anders Norrbring Norrbring Consulting
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Monday 2006-01-09 at 13:23 +0100, Anders Norrbring wrote:
Hi and thanks! It seems like it was spamassassin.. When it ran a lint, every bit of memory was grabbed. I have 2GB of memory in the box, so it seemed a little weird at first.
lint? What is that command? :-?
However, I'm running an old SA on that box, apparently not well configured either.. :) I found that the bayes database files was almost 16GB in size! When I deleted those, a lint went just fine!
Uau! That's big. I dissabled autolearn some time ago, I only allow it to learn manually. And then I purge the database now and then, when it start to not mark spam properly - for which purpose I keep a mailbox full of the last half a year of spam at least.
At 4.35 in the morning, cron triggers a rules_du_jour update of the SA rules, and after that, a lint is run... I hope this will help, at least until I have the time and inspiration to upgrade it all to SuSE 10..
Ah! Could be that, yes. 16Gb! You could increase your swap space :-p Now that I remember, there are some task accounting utilities, that I don't really know how they work; I'm thinking of sa and such, pacckage acct. I understand they might be used to keep track of what the system and users are doing. It might be useful in a situation like this, but I don't know what impact it has on system perfomance, if any. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDwmCstTMYHG2NR9URAnIwAJ9zxh4iJwPNjMx3BO7EEPe50z9EIwCfak0P o/ZgWhfdAUQRBHOn0MdE4dM= =tLeS -----END PGP SIGNATURE-----
participants (5)
-
Anders Norrbring -
Carl Hartung -
Carlos E. R. -
mark -
Scott Leighton