[opensuse] Slow mail processing
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I'm wondering why mail processing is slow in my machine (about 1 mail per second). I know it is because of slow antispam processing, by both amavisd and spamd (spamd is triggered via local .procmailrc rules). The actual antispam is done by spamd, amavis cares of malware. But it should be faster, this is a powerful machine. The processing by amavis takes between 120 ms and 600 ms. Not bad, considering. Spamd is slower, takes around 3 or more seconds per message (3..6). I think it is because of the waiting for online tests, so the solution would be to have more simultaneous processes. I have done this in the past - - but it is not working now. I have done these changes: /etc/sysconfig/spamd: SPAMD_ARGS="-d -c --max-children=7 " /etc/amavisd.conf: $max_servers = 5; # num of pre-forked children (2..30 is common), -m #CER - was 2 /etc/postfix/master.cf: procmail unix - n n - 7 pipe flags=R user=nobody argv=/usr/bin/procmail -t -m /etc/procmailrc ${sender} ${recipient} smtp-amavis unix - - n - 7 lmtp -o lmtp_data_done_timeout=1200 -o lmtp_send_xforward_command=yes I understand that it is postfix which controls how many subprocesses it starts, via those lines in master.cf. But instead of 7 procmail processes I only get two. I can see that by using watch while fetching a bunch of two hundred emails: Every 1,0s: ps afxu|grep 'amavis\|spam'|grep -v 'grep'; echo - ; mailq | grep Req... Mon Apr 23 22:50:50 2012 root 20997 0.0 0.7 144008 64556 ? SNs Apr22 0:07 /usr/sbin/spamd -d -c --max-children=7 -r /var cer 4936 7.2 1.0 165156 87788 ? SN 22:42 0:33 \_ spamd child cer 5996 6.0 0.8 150832 73444 ? SN 22:44 0:23 \_ spamd child root 9591 0.3 0.8 147688 69800 ? SN 22:49 0:00 \_ spamd child cer 10386 0.0 0.0 19288 1092 ? S 22:50 0:00 | \_ /usr/bin/spamc -s 2500000 cer 10363 0.0 0.0 19288 1092 ? S 22:50 0:00 | \_ /usr/bin/spamc -s 2500000 vscan 15648 0.0 0.5 146032 46216 ? Ss Apr22 0:00 amavisd (master) vscan 6422 0.2 0.6 148440 50552 ? S 22:44 0:00 \_ amavisd (ch17-avail) vscan 6718 0.1 0.6 147836 49900 ? S 22:45 0:00 \_ amavisd (ch14-avail) vscan 6792 0.1 0.6 147968 50024 ? S 22:45 0:00 \_ amavisd (ch13-avail) vscan 7501 0.0 0.6 147840 49840 ? S 22:46 0:00 \_ amavisd (ch2-avail) vscan 7575 0.0 0.5 147692 49052 ? S 22:46 0:00 \_ amavisd (ch1-avail) - - - -- 241 Kbytes in 30 Requests. There are at most two spamc processess... something is holding it down. But what? - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) iEYEARECAAYFAk+V8KQACgkQtTMYHG2NR9XITwCeIzDGbC+EU7zwGstG4w4SEQDv ++UAn00+CLKpib8e/3v0U62G1JYXyE8c =IGCM -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
I'm wondering why mail processing is slow in my machine (about 1 mail per second).
If that is real time from beginning of reception to delivery to mailbox, that's not particularly slow.
I know it is because of slow antispam processing, by both amavisd and spamd (spamd is triggered via local .procmailrc rules). The actual antispam is done by spamd, amavis cares of malware.
But it should be faster, this is a powerful machine.
Processing of email is not CPU-intensive, it's much more about DNS and waiting for it.
The processing by amavis takes between 120 ms and 600 ms. Not bad, considering. Spamd is slower, takes around 3 or more seconds per message (3..6).
amavis presumably uses clamav, which is a very fast check. spamassassin has a much more complex ruleset including various DNS lookups (depending on your config).
I think it is because of the waiting for online tests, so the solution would be to have more simultaneous processes. I have done this in the past - - but it is not working now.
I have done these changes:
/etc/sysconfig/spamd:
SPAMD_ARGS="-d -c --max-children=7 "
/etc/amavisd.conf:
$max_servers = 5; # num of pre-forked children (2..30 is common), -m #CER - was 2
/etc/postfix/master.cf:
procmail unix - n n - 7 pipe flags=R user=nobody argv=/usr/bin/procmail -t -m /etc/procmailrc ${sender} ${recipient}
smtp-amavis unix - - n - 7 lmtp -o lmtp_data_done_timeout=1200 -o lmtp_send_xforward_command=yes
I understand that it is postfix which controls how many subprocesses it starts, via those lines in master.cf. But instead of 7 procmail processes I only get two. [snip] There are at most two spamc processess... something is holding it down. But what?
a local delivery maximum setting? -- Per Jessen, Zürich (6.6°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-04-24 10:39, Per Jessen wrote:
Carlos E. R. wrote:
But it should be faster, this is a powerful machine.
Processing of email is not CPU-intensive, it's much more about DNS and waiting for it.
I know, thus the idea of processing several emails at the same time, instead of waiting for the completion of one before starting with the next.
The processing by amavis takes between 120 ms and 600 ms. Not bad, considering. Spamd is slower, takes around 3 or more seconds per message (3..6).
amavis presumably uses clamav, which is a very fast check. spamassassin has a much more complex ruleset including various DNS lookups (depending on your config).
Exactly. Amavis can also check spam, it has the same engine as spamassassin, but I have it disabled, because it does not keep a bayes database per destination user. Spamd/spamc is much more efficient at detecting spam.
But what?
a local delivery maximum setting?
Local delivery should be procmail, no? It is set to "7". I don't have anything in master.cf set to two. 1 yes :-? - -- Cheers / Saludos, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk+Wat4ACgkQIvFNjefEBxqJ0QCaAt9rYdW/3Ty3FwnrmD22GSFY gtAAoLKiGoY8CAt81XHxFu7UiAAdDeYt =XTNC -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
But what?
a local delivery maximum setting?
Local delivery should be procmail, no? It is set to "7". I don't have anything in master.cf set to two. 1 yes :-?
local_destination_concurrency_limit - default is 2. -- Per Jessen, Zürich (8.9°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-04-24 11:18, Per Jessen wrote:
Carlos E. R. wrote:
But what?
a local delivery maximum setting?
Local delivery should be procmail, no? It is set to "7". I don't have anything in master.cf set to two. 1 yes :-?
local_destination_concurrency_limit - default is 2.
Right! I forgot about that one. And I have a comment on that main.cf file on that parameter about this precisely... There is a saying in Spanish, that man is the only animal that stumbles twice on the same stone :-) - -- Cheers / Saludos, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk+WchcACgkQIvFNjefEBxobGQCgxzhLVweFy4D6unMmnkI8Zp+n JWAAn38myTMmG7LyHGjaDy/uSm3w82rn =IhXJ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2012-04-24 at 11:27 +0200, Carlos E. R. wrote:
On 2012-04-24 11:18, Per Jessen wrote:
local_destination_concurrency_limit - default is 2.
Right! I forgot about that one.
And it works, I get about 2 mails per second. I can process emails nearly as fast as I can download them. Every 1,0s: ps afxu|grep 'amavis\|spam\|postfix\|vscan'|grep -v 'grep'; echo - ; mailq | grep Requests. Wed Apr 25 13:25:30 2012 vscan 4141 0.0 1.7 283716 141220 ? Ssl Apr18 1:01 /usr/sbin/clamd vscan 4542 0.0 0.0 47312 1600 ? Ss Apr18 0:10 /usr/bin/freshclam -d vscan 15648 0.0 0.1 146032 13264 ? Ss Apr22 0:01 amavisd (master) vscan 19342 0.0 0.3 147968 27912 ? S Apr24 0:00 \_ amavisd (ch15-avail) vscan 19620 0.0 0.3 147924 27676 ? S Apr24 0:00 \_ amavisd (ch19-avail) vscan 30233 2.5 0.3 147684 25660 ? S 13:25 0:00 \_ amavisd (ch5-avail) vscan 30254 2.7 0.3 147816 27916 ? S 13:25 0:00 \_ amavisd (ch4-avail) vscan 30255 2.6 0.3 147844 28024 ? S 13:25 0:00 \_ amavisd (ch4-avail) root 12928 0.0 0.8 144080 65636 ? Ss Apr23 0:09 /usr/sbin/spamd -d -c --max-children=7 -r /var/run/spamd.pid cer 12948 0.0 0.9 158224 80424 ? S Apr23 0:20 \_ spamd child cer 12949 0.0 1.3 187800 110324 ? R Apr23 0:37 \_ spamd child cer 30286 6.4 0.8 147260 69744 ? S 13:25 0:00 \_ spamd child cer 30287 7.8 0.8 147456 69936 ? S 13:25 0:00 \_ spamd child cer 30288 8.1 0.8 147388 69836 ? S 13:25 0:00 \_ spamd child cer 30307 5.3 0.8 147084 69084 ? S 13:25 0:00 \_ spamd child root 29972 0.1 0.0 20568 1528 ? Ss 13:24 0:00 /usr/lib/postfix/master postfix 29986 0.0 0.0 20532 1340 ? S 13:24 0:00 \_ pickup -l -t fifo -u postfix 29987 0.0 0.0 20904 2124 ? S 13:24 0:00 \_ qmgr -l -t fifo -u postfix 29994 0.0 0.0 20600 1468 ? S 13:24 0:00 \_ showq -t unix -u postfix 30084 0.1 0.0 40880 4000 ? S 13:25 0:00 \_ smtpd -n smtp -t inet -u -o stress= -s 4 postfix 30085 0.0 0.0 20532 1348 ? S 13:25 0:00 \_ proxymap -t unix -u postfix 30086 0.1 0.0 21676 2932 ? S 13:25 0:00 \_ trivial-rewrite -n rewrite -t unix -u postfix 30094 0.2 0.0 27956 2992 ? D 13:25 0:00 \_ cleanup -z -t unix -u postfix 30095 0.0 0.0 37808 2852 ? S 13:25 0:00 \_ lmtp -n smtp-amavis -t unix -u -o lmtp_data_done_timeout=1200 -o lmtp_send_xforwa postfix 30099 0.0 0.0 40756 3900 ? S 13:25 0:00 \_ smtpd -n localhost:10025 -t inet -u -s 2 -o content_filter= postfix 30100 0.0 0.0 20980 2340 ? S 13:25 0:00 \_ local -t unix cer 30320 0.0 0.0 19288 1092 ? S 13:25 0:00 | \_ /usr/bin/spamc -s 2500000 postfix 30124 0.0 0.0 20980 2344 ? S 13:25 0:00 \_ local -t unix cer 30300 0.0 0.0 19288 1088 ? S 13:25 0:00 | \_ /usr/bin/spamc -s 2500000 postfix 30141 0.0 0.0 20980 2136 ? S 13:25 0:00 \_ local -t unix cer 30149 0.0 0.0 19288 1096 ? S 13:25 0:00 | \_ /usr/bin/spamc -s 2500000 postfix 30159 0.2 0.0 27956 2988 ? S 13:25 0:00 \_ cleanup -z -t unix -u postfix 30160 0.0 0.0 20980 2136 ? S 13:25 0:00 \_ local -t unix cer 30170 0.0 0.0 19288 1092 ? S 13:25 0:00 | \_ /usr/bin/spamc -s 2500000 postfix 30161 0.0 0.0 37808 2848 ? S 13:25 0:00 \_ lmtp -n smtp-amavis -t unix -u -o lmtp_data_done_timeout=1200 -o lmtp_send_xforwa postfix 30163 0.0 0.0 20532 1340 ? S 13:25 0:00 \_ scache -l -t unix -u postfix 30172 0.0 0.0 40756 3900 ? S 13:25 0:00 \_ smtpd -n localhost:10025 -t inet -u -s 2 -o content_filter= postfix 30173 0.0 0.0 20980 2140 ? S 13:25 0:00 \_ local -t unix cer 30181 0.0 0.0 19288 1092 ? S 13:25 0:00 | \_ /usr/bin/spamc -s 2500000 postfix 30191 0.0 0.0 40756 3904 ? S 13:25 0:00 \_ smtpd -n localhost:10025 -t inet -u -s 2 -o content_filter= postfix 30205 0.0 0.0 40756 3900 ? S 13:25 0:00 \_ smtpd -n localhost:10025 -t inet -u -s 2 -o content_filter= postfix 30267 0.0 0.0 40756 3900 ? S 13:25 0:00 \_ smtpd -n localhost:10025 -t inet -u -s 2 -o content_filter= postfix 30289 0.0 0.0 20980 2136 ? S 13:25 0:00 \_ local -t unix cer 30305 0.0 0.0 19288 1092 ? S 13:25 0:00 | \_ /usr/bin/spamc -s 2500000 postfix 30308 0.0 0.0 20980 2136 ? S 13:25 0:00 \_ local -t unix cer 30324 0.0 0.0 19288 1088 ? S 13:25 0:00 \_ /usr/bin/spamc -s 2500000 - - - -- 246 Kbytes in 38 Requests. So the throtling process is "local". In the master.cf file the line is: local unix - n n - - local So perhaps instead of adjusting "local_destination_concurrency_limit" globally I can adjust instead the line to: local unix - n n - 7 local - -- Cheers, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) iEYEARECAAYFAk+X4YEACgkQtTMYHG2NR9U1lgCdEhcropwp2klgNwoOc4Wl8HkP 3yIAn3FvmXaI0Ri4N21oQwAI3jyr++lm =zZyP -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
So the throtling process is "local". In the master.cf file the line is:
local unix - n n - - local
So perhaps instead of adjusting "local_destination_concurrency_limit" globally I can adjust instead the line to:
local unix - n n - 7 local
Perhaps, but why not just change "local_destination_concurrency_limit" ? -- Per Jessen, Zürich (15.1°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2012-04-25 at 13:58 +0200, Per Jessen wrote:
local unix - n n - 7 local
Perhaps, but why not just change "local_destination_concurrency_limit" ?
That's what I did, and it works; I just wonder if the other way is better. - -- Cheers, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) iEYEARECAAYFAk+a4HYACgkQtTMYHG2NR9X/ngCeMKVKJo7bboTRdqWAlLz2re/l LxoAnijomTsbMRNjsnI5Es8YRZVEf40X =RMIk -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wednesday, 2012-04-25 at 13:58 +0200, Per Jessen wrote:
local unix - n n - 7 local
Perhaps, but why not just change "local_destination_concurrency_limit" ?
That's what I did, and it works; I just wonder if the other way is better.
They're different - "local_destination_concurrency_limit" is limit of local deliveries to the _same_ local recipient, the process limit is for all local deliveries. -- Per Jessen, Zürich (24.5°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-04-28 11:20, Per Jessen wrote:
Carlos E. R. wrote:
They're different - "local_destination_concurrency_limit" is limit of local deliveries to the _same_ local recipient, the process limit is for all local deliveries.
Ah, I see. - -- Cheers / Saludos, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk+buPYACgkQIvFNjefEBxorGgCfRJbuemplIU8JZOUV58oNC/TN zv0An0BSTls/CmXwqrZguVerwfgxyWEV =ZONQ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (3)
-
Carlos E. R. -
Carlos E. R. -
Per Jessen