SpammAssassin in 9.3 lets spam pass trhough
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I'm getting spam mail that the Bayesian filter marks as 99% spam, but the overall score is below 5. For me, this is a security related problem, as I'm getting intruded by Spam that previously was dumped. For example, the report for one such email says: X-Spam-Status: No, score=2.6 required=5.0 tests=ALL_TRUSTED,BAYES_99, HTML_20_30,HTML_IMAGE_ONLY_24,HTML_MESSAGE autolearn=no version=3.0.2 To me, something that the bayessian filter says is spam within 99%, _is_ spam. However, the scoring used that I see from the tables in /usr/share/spamassassin/50_scores.cf seems to be this: ALL_TRUSTED -2.867 What is that? :-/ BAYES_99 4.070 HTML_20_30 0.567 HTML_IMAGE_ONLY_24 0.787 HTML_MESSAGE 0.001 Total: 2.558 (matches above) What is that "ALL_TRUSTED"? I see in "20_compensate.cf" this: # The message was never sent via an untrustworthy host. header ALL_TRUSTED eval:check_all_trusted() describe ALL_TRUSTED Did not pass through any untrusted hosts tflags ALL_TRUSTED nice It probably means that it was sent through verizon.net. So what? Ok, I'll try to disable it. The default scoring is set in 50_scores.cf as: score ALL_TRUSTED -2.400 -2.820 -2.867 -3.300 The third scoring column is used: Bayessian tests enabled, network tests dissabled. So I go to "/etc/mail/spamassassin/local.cf" and change the score, initially dividing all the values by 4, to: score ALL_TRUSTED -0.600 -0.705 -0.717 -0.825 I suppose that a "score" set in there supplants the default one, no? Well, no! I send that spam email to myself, after restarting the spamd service, and I get: X-Spam-Status: No, score=2.8 required=5.0 tests=ALL_TRUSTED,BAYES_95 autolearn=no version=3.0.2 It is not working. How do I disable that "ALL_TRUSTED" test? I also did the change directly in "/usr/share/spamassassin/50_scores.cf", and the result is the same: X-Spam-Status: No, score=2.8 required=5.0 tests=ALL_TRUSTED,BAYES_95 How on earth do I disable that $&%$/$/* "ALL_TRUSTED" test? How do I make that spam marked as BAYES_99 does get flagged as SPAM: yes? Configuration changes in scoring are not read at all... not even after a reboot. :-/ - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFCfJ2ItTMYHG2NR9URAp7pAKCQfWKZIV8oXw/D+qXrt+6QSvW3GwCdFmU+ Ewo8BKblWvru48yVXhstswQ= =N+7g -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi! First of all spamassassin has to learn what you consider as spam and what you consider as ham (no spam). If you don't teach both you may get false positives. I use cyrus + postfix + cyrus + amavisd-new + serverbased imap-filter-rules for my users - but this setup may be a littlebit too complex ;). In short: One rules sorts spam into spam-folder of the user's imap-folder (it will be deleted manually in case the spam-filter recognises too much as spam). I made a spamassassin imap-folder for each user and a cron-script learnes the bayes for each user every day from this folder. Here is my script for postfix/cyrus/spamassassin: [Begin Script] #!/bin/bash # # Spamassassin Cron-Script # # Target: Learn spam and add to spam-db for all users with their own spam. # # learn spam and clear spamassassin-folder ... # /usr/bin/sa-learn --spam /var/spool/imap/user/*/Spamassassin/*. su - cyrus -c '/usr/lib/cyrus/bin/ipurge -i -b 1 user.*.Spamassassin' # # find users ... # LOCUSERS=`find /home -group users -maxdepth 1` for LOCALUSERS in $LOCUSERS; do LUSERS=$LUSERS" "${LOCALUSERS:6} done # # Copy global config for all users # for namen in $LUSERS; do test -d "/home/$namen/.spamassassin" || { mkdir /home/$namen/.spamassassin; } cp -R /root/.spamassassin/bayes* /home/$namen/.spamassassin chown -R $namen:users /home/$namen/.spamassassin done #echo 'Starte Postfix neu:' #/etc/init.d/postfix restart echo 'User-Spamdb updated!' [End Script] If you don't use imap you can use a pop3-account and move your spam as admin to another users mailbox (maybe user = spam) and execute the following (you have to change the folder at the end to your settings!!!): #learn spam: /usr/bin/sa-learn --spam /var/spool/mail/spam #learn ham: /usr/bin/sa-learn --ham /var/spool/mail/mailbox-with-ham-and-no-spam This mailbox should not include ham otherwise you get false positives! If you want to create a config-file with most options for spamassassin you may use this form: http://www.yrex.com/spam/spamconfig.php Otherwise you should examine the manpages and search a lot with google. Spamfiltering in general based on postifx: Make acl's to filter mails (postfix will reject this mails): - - use amavisd-new to scan for viruses (f-prot is free for use for private users under linux) - - disable notification of virus in amavis and disable any other notifications in spamassassin (postfix should ony rejet this and don't give any answers) - - reject executable extensions (I do this via amavid-new) -> good bye virus - - use a rbl-server for blocking announced spam-providing servers - - non-existing url's (dns-check) - - Implement spamassassin or any other spam-filter. In general: Don't answer to any spam of any kind. With this policy I get about max. 3 spam-mails per day (before a lot more). After learning ham I don't get much lase positives (now exactly 0%). Bet way to learn ham is to learn all your mails from your mailbox containing no spam! Reguards Philippe - -- Diese Nachricht ist digital signiert und enthält weder Siegel noch Unterschrift! Die unaufgeforderte Zusendung einer Werbemail an Privatleute verstößt gegen §1 UWG und 823 I BGB (Beschluß des LG Berlin vom 2.8.1998 Az: 16 O 201/98). Jede kommerzielle Nutzung der übermittelten persönlichen Daten sowie deren Weitergabe an Dritte ist ausdrücklich untersagt! - -- Diese Nachricht ist digital signiert und enthält weder Siegel noch Unterschrift! Die unaufgeforderte Zusendung einer Werbemail an Privatleute verstößt gegen §1 UWG und 823 I BGB (Beschluß des LG Berlin vom 2.8.1998 Az: 16 O 201/98). Jede kommerzielle Nutzung der übermittelten persönlichen Daten sowie deren Weitergabe an Dritte ist ausdrücklich untersagt! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQD1AwUBQnzJ20Ng1DRVIGjBAQIPnwb/bSLKhjDTTzg1tdEjCdZltkO5V2IqVmFF hvYxqnv5qWSpsC1oAko84wJ0bXTqjwPz8WPqwU0Qjh1XKn91OwAK8szsbWlzcOEy Mgv+Em8dSR057ttv5KiGYzllbI5kFCpb1eYoBaTqxPk0zGuxt08a/JVrmRUAPwIe 6R6uqrnnShSp8XhY474sbFa2hzczLEiP77dQlQ+MsGxASb2l2HKNAitJNP78bXvq W2UMxdSOAdiIqKQwHhmNyAcd+6VInAHRb5qYjOQvtvX6jV597s8eo3+5MKI8MzbW 8SHsss9MVvA= =nWE1 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Philippe Vogel schrieb:
Hi!
First of all spamassassin has to learn what you consider as spam and what you consider as ham (no spam). If you don't teach both you may get false positives.
I use cyrus + postfix + cyrus + amavisd-new + serverbased imap-filter-rules for my users - but this setup may be a littlebit too complex ;). In short: One rules sorts spam into spam-folder of the user's imap-folder (it will be deleted manually in case the spam-filter recognises too much as spam). I made a spamassassin imap-folder for each user and a cron-script learnes the bayes for each user every day from this folder.
Here is my script for postfix/cyrus/spamassassin:
[Begin Script]
#!/bin/bash # # Spamassassin Cron-Script # # Target: Learn spam and add to spam-db for all users with their own spam. # # learn spam and clear spamassassin-folder ... # /usr/bin/sa-learn --spam /var/spool/imap/user/*/Spamassassin/*. su - cyrus -c '/usr/lib/cyrus/bin/ipurge -i -b 1 user.*.Spamassassin' # # find users ... # LOCUSERS=`find /home -group users -maxdepth 1` for LOCALUSERS in $LOCUSERS; do LUSERS=$LUSERS" "${LOCALUSERS:6} done # # Copy global config for all users # for namen in $LUSERS; do test -d "/home/$namen/.spamassassin" || { mkdir /home/$namen/.spamassassin; } cp -R /root/.spamassassin/bayes* /home/$namen/.spamassassin chown -R $namen:users /home/$namen/.spamassassin done #echo 'Starte Postfix neu:' #/etc/init.d/postfix restart echo 'User-Spamdb updated!'
[End Script]
If you don't use imap you can use a pop3-account and move your spam as admin to another users mailbox (maybe user = spam) and execute the following (you have to change the folder at the end to your settings!!!):
#learn spam:
/usr/bin/sa-learn --spam /var/spool/mail/spam
#learn ham:
/usr/bin/sa-learn --ham /var/spool/mail/mailbox-with-ham-and-no-spam
This mailbox should not include ham otherwise you get false positives!
If you want to create a config-file with most options for spamassassin you may use this form:
http://www.yrex.com/spam/spamconfig.php
Otherwise you should examine the manpages and search a lot with google.
Spamfiltering in general based on postifx:
Make acl's to filter mails (postfix will reject this mails):
- use amavisd-new to scan for viruses (f-prot is free for use for private users under linux) - disable notification of virus in amavis and disable any other notifications in spamassassin (postfix should ony rejet this and don't give any answers) - reject executable extensions (I do this via amavid-new) -> good bye virus - use a rbl-server for blocking announced spam-providing servers - non-existing url's (dns-check) - Implement spamassassin or any other spam-filter.
In general: Don't answer to any spam of any kind.
With this policy I get about max. 3 spam-mails per day (before a lot more). After learning ham I don't get much lase positives (now exactly 0%).
Bet way to learn ham is to learn all your mails from your mailbox containing no spam!
Reguards
Philippe
I forgot to mention that this db can only work for your server, so don't ask for a spam-db, this is always unique for one server. If your server learned about a month you will notice a lower spam level (if you get a lot 'o' spam). Your server must permanently learnspam and ham! A too low level will filter too much. Reguards Philippe - -- Diese Nachricht ist digital signiert und enthält weder Siegel noch Unterschrift! Die unaufgeforderte Zusendung einer Werbemail an Privatleute verstößt gegen §1 UWG und 823 I BGB (Beschluß des LG Berlin vom 2.8.1998 Az: 16 O 201/98). Jede kommerzielle Nutzung der übermittelten persönlichen Daten sowie deren Weitergabe an Dritte ist ausdrücklich untersagt! - -- Diese Nachricht ist digital signiert und enthält weder Siegel noch Unterschrift! Die unaufgeforderte Zusendung einer Werbemail an Privatleute verstößt gegen §1 UWG und 823 I BGB (Beschluß des LG Berlin vom 2.8.1998 Az: 16 O 201/98). Jede kommerzielle Nutzung der übermittelten persönlichen Daten sowie deren Weitergabe an Dritte ist ausdrücklich untersagt! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQD1AwUBQnzNK0Ng1DRVIGjBAQLeWQb/RUtpBdtoSY3aOsGgX6gEa6ttaYFxx2qS jhYuQjEsG4h9lv3BOcxodHKgs/Z3f+JyfYy7Zxqlf5lHZrxyoHiqEJkbq3mn9byV lP8ymRRWbsWXepG+/2zDCdyYq+5zX4wDusYd+9qN39mxcSlNquXKNnsbR2k/YRrj PEe9UVslIxv1jFBsEE0B4HbdAxIpXeVbvNgV3wXCvYuNmCPXVOS1BlZZA9KUZz1T hdSnJJcM01uvWK0p1Cy6BiTI8H9ODXr6jPI38yVO3TaLonP0Q6b6q87k6IV0qfCH 9oZSEJj0Ico= =DpyF -----END PGP SIGNATURE-----
The Saturday 2005-05-07 at 15:59 +0200, Philippe Vogel wrote:
First of all spamassassin has to learn what you consider as spam and what you consider as ham (no spam). If you don't teach both you may get false positives.
That's not the case, it is well trained: X-Spam-Status: No, score=2.6 required=5.0 tests=ALL_TRUSTED,BAYES_99, Notice the "BAYES_99" token. It has been flagged as 99% probability of being spam, and that is the maximun possible. That is not the problem I reported. What I want to do is remove the "ALL_TRUSTED" test and change the scoring: it can not be changed. It seems to be hardcoded :-/ -- Cheers, Carlos Robinson
On Saturday 07 May 2005 12:50, Carlos E. R. wrote:
The third scoring column is used: Bayessian tests enabled, network tests dissabled. So I go to "/etc/mail/spamassassin/local.cf" and change the score, initially dividing all the values by 4, to: score ALL_TRUSTED -0.600 -0.705 -0.717 -0.825 I suppose that a "score" set in there supplants the default one, no? Well, no! I send that spam email to myself, after restarting the spamd service, and I get: X-Spam-Status: No, score=2.8 required=5.0 tests=ALL_TRUSTED,BAYES_95 autolearn=no version=3.0.2 It is not working. How do I disable that "ALL_TRUSTED" test? I also did the change directly in "/usr/share/spamassassin/50_scores.cf", and the result is the same: X-Spam-Status: No, score=2.8 required=5.0 tests=ALL_TRUSTED,BAYES_95 How on earth do I disable that $&%$/$/* "ALL_TRUSTED" test?
From man Mail::SpamAssassin::Conf Setting a rule's score to 0 will disable that rule from running. I hope that works for you: set "score ALL_TRUSTED 0 0 0 0" into the local.cf, and if it doesn't work try the 50_score.cf, too. If that doesn't work, something is definitely broken, because this works for me (I use amavis, so I need to modify either "amavis" user's local.cf, or the global 50_score.cf).
How do I make that spam marked as BAYES_99 does get flagged as SPAM: yes? Configuration changes in scoring are not read at all... not even after a reboot. :-/
Reboot won't help here. Restart spamd or amavis (whichever you're using) and the changes should be visible. If they are not, you're doing something wrong. -- Jure Koren, n.i.
The Sunday 2005-05-08 at 09:53 +0200, Jure Koren wrote:
From man Mail::SpamAssassin::Conf Setting a rule's score to 0 will disable that rule from running.
I hope that works for you: set "score ALL_TRUSTED 0 0 0 0" into the local.cf,
I have this there: #score ALL_TRUSTED -2.400 -2.820 -2.867 -3.300 score ALL_TRUSTED -0.600 -0.705 -0.717 -0.825 It is not used, I get the old score.
and if it doesn't work try the 50_score.cf, too.
I have had this there for days: #score ALL_TRUSTED -2.400 -2.820 -2.867 -3.300 #score ALL_TRUSTED -0.600 -0.705 -0.717 -0.825 score ALL_TRUSTED 0 It doesn't work either.
If that doesn't work, something is definitely broken,
That's what I've been saying all along :-}
because this works for me (I use amavis, so I need to modify either "amavis" user's local.cf, or the global 50_score.cf).
How do I make that spam marked as BAYES_99 does get flagged as SPAM: yes? Configuration changes in scoring are not read at all... not even after a reboot. :-/
Reboot won't help here. Restart spamd or amavis (whichever you're using) and the changes should be visible. If they are not, you're doing something wrong.
I did, nothing happened. As I power off everyday, I mentioned that reboot didn't help - just in case. [...] At last! I got it working; it took some convincing. It seems that changes in /etc/mail/spamassassin/local.cf takes precedence, so whatever I did to 50_score.cf was ignored. But I still don't see why my initial changes to local.cf did not work. Whatever, it works today :-) Ummm! I still have to figure out why scores of bayes_99 do not trigger an email being spam. Either I modify my rules in procmail, or I modify the scoring of Bayesian tests. You see, even without the ALL_TRUSTED thing, it still is not flagged as spam: X-Spam-Status: No, score=4.9 required=5.0 tests=AWL,BAYES_99,HTML_20_30, HTML_IMAGE_ONLY_24,HTML_MESSAGE autolearn=no version=3.0.2 Perhaps increasing the score of HTML_IMAGE_ONLY_24 :-? -- Cheers, Carlos Robinson
Carlos, El Dom 08 May 2005 07:05, Carlos E. R. escribió:
Ummm! I still have to figure out why scores of bayes_99 do not trigger an email being spam. Either I modify my rules in procmail, or I modify the scoring of Bayesian tests.
You see, even without the ALL_TRUSTED thing, it still is not flagged as spam:
X-Spam-Status: No, score=4.9 required=5.0 tests=AWL,BAYES_99,HTML_20_30, HTML_IMAGE_ONLY_24,HTML_MESSAGE autolearn=no version=3.0.2
A bayes_99 test result contributes in my standard setup of SA 3.0.3 exactly 4.1 points to the final score; see this example report: X-Spam-Report: * 0.5 TO_MALFORMED To: has a malformed address * -2.9 ALL_TRUSTED Did not pass through any untrusted hosts * 2.5 DATE_IN_FUTURE_48_96 Date: is 48 to 96 hours after Received: date * 2.5 DOMAIN_RATIO BODY: Message body mentions many internet domains * 0.5 HTML_COMMENT_SAVED_URL BODY: HTML message is a saved web page * 1.1 HTML_IMAGE_RATIO_02 BODY: HTML has a low ratio of text to image area * 0.0 HTML_90_100 BODY: Message is 90% to 100% HTML * 0.0 HTML_MESSAGE BODY: HTML included in message * 2.7 HTML_IMAGE_ONLY_08 BODY: HTML: images with 400-800 bytes of words * 4.1 BAYES_99 BODY: Bayesian spam probability is 99 to 100% * [score: 1.0000] * 1.2 MIME_HTML_ONLY BODY: Message only has text/html MIME parts * 0.0 MIME_HEADER_CTYPE_ONLY 'Content-Type' found without required MIME headers * 1.8 PRIORITY_NO_NAME Message has priority, but no X-Mailer/User-Agent Your setup requires a final score >= 5.0 to trigger the message being marked as Spam, but the final score on your message is only 4.9. The Bayes testing score alone will never trigger a message to be marked as Spam, unless you increment its contributing score or lower the threshold for a message to be marked as Spam. In your current setup, this message, even with a positive BAYES_99 test result, isn't 'spammy' enough to reach 5.0 points. Saludos, -- Andreas Philipp Noema Ltda. Bogotá, D.C. - Colombia http://www.noemasol.com
The Sunday 2005-05-08 at 10:37 -0500, Andreas Philipp wrote:
X-Spam-Status: No, score=4.9 required=5.0 tests=AWL,BAYES_99,HTML_20_30, HTML_IMAGE_ONLY_24,HTML_MESSAGE autolearn=no version=3.0.2
A bayes_99 test result contributes in my standard setup of SA 3.0.3 exactly 4.1 points to the final score; see this example report:
True.
X-Spam-Report: * 0.5 TO_MALFORMED To: has a malformed address * -2.9 ALL_TRUSTED Did not pass through any untrusted hosts * 2.5 DATE_IN_FUTURE_48_96 Date: is 48 to 96 hours after Received: date * 2.5 DOMAIN_RATIO BODY: Message body mentions many internet domains * 0.5 HTML_COMMENT_SAVED_URL BODY: HTML message is a saved web page * 1.1 HTML_IMAGE_RATIO_02 BODY: HTML has a low ratio of text to image area * 0.0 HTML_90_100 BODY: Message is 90% to 100% HTML * 0.0 HTML_MESSAGE BODY: HTML included in message * 2.7 HTML_IMAGE_ONLY_08 BODY: HTML: images with 400-800 bytes of words * 4.1 BAYES_99 BODY: Bayesian spam probability is 99 to 100% * [score: 1.0000] * 1.2 MIME_HTML_ONLY BODY: Message only has text/html MIME parts * 0.0 MIME_HEADER_CTYPE_ONLY 'Content-Type' found without required MIME headers * 1.8 PRIORITY_NO_NAME Message has priority, but no X-Mailer/User-Agent
Your setup requires a final score >= 5.0 to trigger the message being marked as Spam, but the final score on your message is only 4.9. The Bayes testing score alone will never trigger a message to be marked as Spam, unless you increment its contributing score or lower the threshold for a message to be marked as Spam. In your current setup, this message, even with a positive BAYES_99 test result, isn't 'spammy' enough to reach 5.0 points.
I know... it is unfortunate. Either SA needs updating for new kind of spams with new rules, or I change the scoring. I'm inclined to do that, but I'm unsure of what score to give it. Probably just increase all of the bayes scores proportionately till BAYES_99 is 4.9 or 5. -- Cheers, Carlos Robinson
On Sunday 08 May 2005 10:37 am, Carlos E. R. wrote:
The Sunday 2005-05-08 at 10:37 -0500, Andreas Philipp wrote:
X-Spam-Status: No, score=4.9 required=5.0 tests=AWL,BAYES_99,HTML_20_30, HTML_IMAGE_ONLY_24,HTML_MESSAGE autolearn=no version=3.0.2
A bayes_99 test result contributes in my standard setup of SA 3.0.3 exactly 4.1 points to the final score; see this example report:
True.
X-Spam-Report: * 0.5 TO_MALFORMED To: has a malformed address * -2.9 ALL_TRUSTED Did not pass through any untrusted hosts * 2.5 DATE_IN_FUTURE_48_96 Date: is 48 to 96 hours after Received: date * 2.5 DOMAIN_RATIO BODY: Message body mentions many internet domains * 0.5 HTML_COMMENT_SAVED_URL BODY: HTML message is a saved web page * 1.1 HTML_IMAGE_RATIO_02 BODY: HTML has a low ratio of text to image area * 0.0 HTML_90_100 BODY: Message is 90% to 100% HTML * 0.0 HTML_MESSAGE BODY: HTML included in message * 2.7 HTML_IMAGE_ONLY_08 BODY: HTML: images with 400-800 bytes of words * 4.1 BAYES_99 BODY: Bayesian spam probability is 99 to 100% * [score: 1.0000] * 1.2 MIME_HTML_ONLY BODY: Message only has text/html MIME parts * 0.0 MIME_HEADER_CTYPE_ONLY 'Content-Type' found without required MIME headers * 1.8 PRIORITY_NO_NAME Message has priority, but no X-Mailer/User-Agent
Your setup requires a final score >= 5.0 to trigger the message being marked as Spam, but the final score on your message is only 4.9. The Bayes testing score alone will never trigger a message to be marked as Spam, unless you increment its contributing score or lower the threshold for a message to be marked as Spam. In your current setup, this message, even with a positive BAYES_99 test result, isn't 'spammy' enough to reach 5.0 points.
I know... it is unfortunate. Either SA needs updating for new kind of spams with new rules, or I change the scoring. I'm inclined to do that, but I'm unsure of what score to give it. Probably just increase all of the bayes scores proportionately till BAYES_99 is 4.9 or 5.
Seems like a problem of addition to me. It should have scored around 10 by just adding up all the scores, even with the negative one. -- _____________________________________ John Andersen
Am Samstag, 7. Mai 2005 12:50 schrieb Carlos E. R.:
(...). How on earth do I disable that $&%$/$/* "ALL_TRUSTED" test? (...).
http://wiki.apache.org/spamassassin/TrustPath HTH, Jan -- He that would first govern others, first should be a master of himself.
participants (6)
-
Andreas Philipp
-
Carlos E. R.
-
Jan Ritzerfeld
-
John Andersen
-
Jure Koren
-
Philippe Vogel