-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2014-08-19 at 09:51 -0400, Patrick Shanahan wrote:
* Anton Aylward <> [08-19-14 09:22]:
2. The spam/av processing can be done within procmail so why not eliminate the sendmail/postfix step and have fetchmail hand off to local procmail directly?
What this boils down to is this:
Do you have a good reason to apply the processing that can *only* be carried out by sendmail/procmail in this path?
I poured over this and could not find any.
Please note: I'm not saying "Postfix is not needed" in any universal sense. I said "in this path".
I have postfix and by fetchmail handing to postfix I get *all* mail logged in /var/log/mail, ie: not just fetchmail's and procmail's logs.
It was a *conscious* decision :^).
We forgot another important reason(s): parallelization, speed, and safety. spamd/spamc called from procmail takes 1..3 seconds per email in my machine, but often it goes up to 5 seconds per email (I see this in the log), and even more. This is due not to lack of cpu power, but for waiting for online tests to respond or timeout. See (all are entries trimmed from the 2014-08-15 log): _00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD scantime=1.1,size=5696,user=cer,uid=1000,required_score=5.0, _00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD scantime=1.2,size=6345,user=cer,uid=1000,required_score=5.0, _00,DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,FREEMAIL_FROM,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,URIBL_BLOCKED scantime=1.7,size=7578, _00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,URIBL_BLOCKED scantime=1.9,size=5677,user=cer,uid=1000,re _00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_TVD_MIME_EPI,URIBL_BLOCKED scantime=1.6,size=6297,user=cer _00,DKIM_SIGNED,RCVD_IN_DNSWL_HI,T_DKIM_INVALID,URIBL_BLOCKED scantime=2.7,size=7320,user=cer,uid _00,DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,FREEMAIL_FROM,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALI _00,DKIM_SIGNED,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,URIBL_BLOCKED scantime=1.6,size=7151 _00,RCVD_IN_DNSWL_HI,URIBL_BLOCKED scantime=13.3,size=6114,user=cer,uid=1000,required_score=5.0 _00,RCVD_IN_DNSWL_HI scantime=13.3,size=5946,user=cer,uid=1000,required_score=5.0,rhost=localhost 00,RCVD_IN_DNSWL_HI scantime=4.0,size=7131,user=cer,uid=1000,required_score=5.0,rhost=localhost 00,DKIM_SIGNED,RCVD_IN_DNSWL_HI,T_DKIM_INVALID,URIBL_BLOCKED scantime=4.1,size=8650,user=cer 0,RCVD_IN_DNSWL_HI,URIBL_BLOCKED scantime=5.3,size=6182,user=cer,uid=1000,required_score=5.0 00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HTML_MESSAGE,SPF_SOFTFAIL,URIBL_BLOCKED scantime=6.6,size=45027, You can see one scantime=13.3 8seconds) in there (with size=6114). Two, actually. On occasion, 12 seconds per email becomes typical, till I find out that a particular online spamassassin test has to be disabled because the internet server it queries no longer responds. Well, if you use fetchmail calling procmail directly, only one email at a time can be processed, but if you put postfix in the toolchain, it queues email internally, and does so safely, because fetchmail can not tell the imap/pop server to delete that email till postfix has saved it to a file in the queue, and tells fetchmail "Ok, I got it". This is so in the entire chain of servers and daemons, local and remote (should be). Old Unix hands would tell you to mount the partition used for mail processing with the option "sync", for this safety reason. This way fetchmail can run at top speed, without waiting for spam/av processing, because postfix is safely storing email in queues on disk, prior to processing it. And postfix can feed typically 2 procmail processes per user, but if you have the power, you can let it call dozens simultaneously, and that means several spamc/spamd processes running simultaneously. When each email takes a few seconds to be processed, most of that time waiting idle (no cpu used), well, feeding say, 5 emails simultaneously does speed things up a bit. This is a big issue when you get a queue of a thousand emails. It can takes hours if you do one at a time. Ah, and procmail can handle several simultaneous processes, even when using mbox files, because it by default uses a different lock file per destination folder. Of course, if all email goes to the same folder, it has to wait. And typically, the spamc call is in another rule. - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlP0fMgACgkQtTMYHG2NR9XyyACZAUfz1CJnLJALhrajtbcEkI1O 9kMAn0z1fYqL1JJNMPXn1kHX62gJRC3c =UAnX -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org