The answer is to have two or more accounts, one that you will only give to friends, family, etc. And some othes, you communicate on regular basis ... when those accounts become to flooded with moron-mail ... dump it. Keep track of the "companies" that give your email to spammers, and make sure you *know* how to deal with them properly. On Thursday 25 December 2003 16:46, Jon Clausen wrote:

On Wed, Dec 24, 2003 at 01:31:23PM -0500, Bruce Marshall wrote:

...

On Wed December 24 2003 12:57 pm, Jon Clausen wrote:

...

Hi list

At this point I'm more looking for opinions than actual help...

I have the server set up with system-wide spamassassination. Lately SA misses spam more frequently, and I need to train it.

Not sure training is really the answer. The spam'rs are getting more clever and they aren't putting much in the emails that you can train on.

True... :(

...

For example, one good trick of theirs is to put only an image in their emails... It has all the text and pics they want but it's only one image.

SA will catch this as HTML_IMAGE_ONLY but it doesn't give a very high score for it. (maybe 1.5)

You may want to go look at some of the spam you're getting, determine what they are using (like the above) and adjust the scores accordingly. Same goes for the scores where bayes says it is 99% probable spam. Jack up the score on that.

Most of what I get is more or less entirely html, so that's a place to start...

...

I'd be glad to send you my local.cf with a lot of mods to it.

Please do. I always like seeing real-life examples.

Thx, Jon -- Whatever rocks your boat!

On Fri, 26 Dec 2003 12:26:32 +0100 Jon Clausen <dsl23212@vip.cybercity.dk> wrote:

- I want the users to access mail through imap.

- Setting up 'shared' {spam|ham|forget}-folders should be No Big Deal, right?

- In order for the sa-learn script to not have to keep processing the same messages, content will have to be removed from said folder after each run. So is it neccessary to run something like 'reconstruct' to keep Cyrus happy?

Hmmm... Maybe the above belongs in a separate thread;

'(Cyrus) Imap "mechanics"'

I currently launch sa-learn manually. I'd have to look in spamassassin -r again cos in 2.60 I had problems in reporting spam to razor due to more strict control on tainted code. Since I'm not a perl hacker and I'm not planning to become one (I'm enough busy learning my stuff <g>) I didn't check if after updating perl and Spamassassin to 2.61 things got fixed. And since once I'll decide to look at it I'd like to solve the problem completely I'm still waiting for the right inspiration to automate this kind of stuff as well. One further problem could be which user/email address should report spam to Razor. Currently I'm running a hand-made package of UWIMAP but I had some problems with the one Windows client (Pegasus) and multiple connections. I'm not an expert of IMAP RFC so I don't know if it is Pegasus misbehaving or UW... and I seldom use Windows at all, so solving this problem has been postponed too. UW saves emails in mbox format, so it is easy to implement a script to run in cron. Anyway nobody I know use UW... and I may guess its limits outside "lan personal use". So I thought that if I'll move to another IMAP server I'll have to find a different solution than just sa-learn --mbox --spam < file One way to go even if it may be a bit overkill to read local files through IMAP protocol could be: http://imapfilter.hellug.gr/ This will make your script independent on the file format and the file location.

On Fri, Dec 26, 2003 at 02:41:22PM +0100, Carlos E. R. wrote:

The Friday 2003-12-26 at 12:26 +0100, Jon Clausen wrote:

<snip>

...

This, however, begs the next (couple of) question(s);

I have another thought:

It may happen that what one user considers spam, for another one it might be useful mail. There would be problems if spam criteria is not unified amongst your users. That must be the reasoning behind the idea of setting spam filtering for each user, instead of globally - or at least, the customization or bayesian filters and such things.

True. Except that the one server I'm talking about is my personal one, where it's all me anyway, and the other one is the one that will eventually handle mail for the company. And since anything the company server is going to handle is company mail, the criteria for spammyness is a (well, will be anyway) company policy. But it's a valid point of course :)

On the other hand, some of those files are big, with one set for each user:

<snip> Who said size doesn't matter? ;D Cheers, Jon -- Whatever rocks your boat!

On Mon, Dec 29, 2003 at 03:38:22AM +0100, Carlos E. R. wrote:

The Saturday 2003-12-27 at 21:19 +0100, Jon Clausen wrote:

...

And since anything the company server is going to handle is company mail, the criteria for spammyness is a (well, will be anyway) company policy.

Mmmm... not if you allow users to directly update the Bayesian filter database. If you want to keep the company policy, then spam should be forwarded to a person responsible of checking them and feeding the result to sa-learn. Otherwise, you might end by someone sending an email to be considered spam, and another user sending the same email to the list of non spam - the filters would get mad at you :-)

Now *that's* a point. So the question then becomes how to avoid that situation... My first impression would be to set up some kind of 'mutex-logic' to make sure no identical mails exist in the two folders... I don't know. In the end it might actually be simpler (and less cpu-intensive) to maintain user-specific filters... The thing is it's a small company, and I doubt this task of 'spam-screening' would be welcome with *any* of the guys...

...

<snip>

Who said size doesn't matter? ;D

X-)

I've never said file size is not important, contrary to what many developers and other people think :-)

They think you said that? Shame on them !-) /Jon -- Whatever rocks your boat!

On Mon, Dec 29, 2003 at 04:44:24PM +0100, Carlos E. R. wrote: <snip>

Now it must be time to go have a look at the spamassassin web site, and learn what they think/say about it - they must have some general setup documentation.

<slaps forehead> Indeed there's a link from there to; http://www.geekly.com/entries/archives/00000155.htm which has a step by step on this. There are a couple of loose ends, but at least it gives a general idea.

Last time I looked (over a year ago) I think they recommended a per user setup, but there must be a better way for a server with many users...

Probably, but I won't be having 'many' users. At least not on any of the servers that I'm planning.

you can not give a bash account to everybody. On the other hand, it might be a good idea, so that they try Linux ;-)

Nyeah... Methinks these guys are better off without me cursing at them for destroying everything they can get their hands on... ;) Or TPIAW; Pointy-clicky is the way to go... The boss asked me if I could fix his '98. "Sure" I said, and next time he looked I had to start explaining KDE for him... This was when 8.2 was released, and he's getting used to it. Like the fact that the stuff just works... ;) But there's a long way from that to actually letting him do anything in a shell...

...

The thing is it's a small company, and I doubt this task of 'spam-screening' would be welcome with *any* of the guys...

Right.

Which is the whole idea of setting up something that doesn't involve anything more complicated than moving 'misses' to another folder. /Jon -- Whatever rocks your boat!

On Fri, Jan 02, 2004 at 12:40:15AM +0100, Carlos E. R. wrote:

I have recently updated my SA from sources (as posted on the list) and found some very interesting info for you in 'Mail-SpamAssassin-2.61/sql/README':

Thanx for the pointer. I'll have a look at it.

| Using SpamAssassin With An SQL Database | ---------------------------------------

<snip>

I don't know if the bayes database can go there, but my feeling is "no" - I hope to be wrong. It is promissing...

Indeed. This looks like it's worth investigating. Maybe I should think about updating my SA too... I'm currently looking into Mail::IMAPClient, partly because it seems to be the way to go for this particular situation, partly because if I learn how to use that, then I can do something *else* that I've wanted for a while; A max on the number of msgs in any specific folder (like this one, SLE). What I want is to basicaly turn the folder into a FIFO, and have a script that rotates the oldest msgs out into a dated "archive". I would, in fact, have preferred to work directly on the files, but I was told Cyrus wouldn't like this... I guess it's not such a bad idea to learn the Perl/IMAP interface, it's just the same old; "Every time I wanna do something, I gotta do/fix/learn something else first" -thing... Oh well... ;P

...

...

you can not give a bash account to everybody. On the other hand, it might be a good idea, so that they try Linux ;-)

Nyeah... Methinks these guys are better off without me cursing at them for destroying everything they can get their hands on... ;)

X-)

Ahh... you *know*? ;)

...

Or TPIAW; Pointy-clicky is the way to go... The boss asked me if I could fix his '98. "Sure" I said, and next time he looked I had to start explaining KDE for him... This was when 8.2 was released, and he's getting used to it. Like the fact that the stuff just works... ;)

But there's a long way from that to actually letting him do anything in a shell...

:-)

Why not? If he knows MsDos, it is not so different; and you can not do much harm on a user account (unless bad intentioned, of course).

But he doesn't. And besides, you don't know this guy... He has a natural ability to ...well... be 'unlucky'... <snip>

There are some specialized companies out there (not ISPs) that filter you email before you retrieve it, for a fee.

Yeah, well... one other part of this whole idea, is to eventually cut/get rid of external expenses... Cheers, Jon -- Whatever rocks your boat!

On Dec 26, 2003, at 5:26, Jon Clausen wrote:

Now, since setting up {spam|ham|forget}@domain seems to require too much of the users (in that they would be required to edit the headers, before forwarding to those accounts) I think the {spam|ham|forget}-mailbox+cronjob-sa-learn is probably the better solution. Because this would only require the users to move the message(s) to said mailbox(es).

This, however, begs the next (couple of) question(s);

- I want the users to access mail through imap.

- Setting up 'shared' {spam|ham|forget}-folders should be No Big Deal, right?

This is how I set up my mail server. The only problem when you have multiple users is that one person's spam is another person's ham. In my case I am the only user so this works fine.

- In order for the sa-learn script to not have to keep processing the same messages, content will have to be removed from said folder after each run. So is it neccessary to run something like 'reconstruct' to keep Cyrus happy?

What I did was to create a new Cyrus account which has only permissions to access the shared "ham" and "spam" folders. A cron job runs a script which calls a python program that logs into the server using that account and reads the emails, processes them and then removes them. Here is the script. It can probably be improved, but it works for me. Be aware though that I am using spamassasin and amavisd-new on a Mandrake 9.1 system so it may work a bit differently under SuSE (such as file locations). First is the small shell script that calls the python script: ------------- BEGIN process_spam_and_ham ------------ #!/bin/bash /usr/local/bin/process_spam_and_ham.py spam /usr/local/bin/process_spam_and_ham.py ham ------------- END process_spam_and_ham ------------ And this is the python script: ------------- BEGIN process_spam_and_ham.py ------------ #!/usr/bin/python import imaplib, string, popen2, os, sys if len(sys.argv) <> 2 or (sys.argv[1] != 'spam' and sys.argv[1] != 'ham'): print >>os.sys.stderr, 'Usage: ' + sys.argv[0] + ' {spam|ham}' sys.exit(1) type = sys.argv[1] print 'Type: ' + type # Connect to the mail server m = imaplib.IMAP4("mail-server") m.login("username", "password") # Select the inbox m.select(type) #Get a list of messages in the mailbox status, messages = m.search(None, "ALL") # Write each message to a file using the message number for num in messages[0].split(): print 'writing message ' + num # Fetch the email status, data = m.fetch(num, "(RFC822)") # and write it into a file f=open('/var/spool/amavis/bayesian/' + num, 'w') f.write(data[0][1]) f.close() # delete the email m.store(num, 'FLAGS', r'(\Deleted)') m.close() m.logout() # Execute the Bayesian tutor os.system('/usr/bin/sa-learn --' + type + ' -p /var/lib/amavis/.spamassassin/user_prefs --dir /var/spool/amavis/bayesian/') # For each spam message invoke razor-report and then delete the message (spam and ham) for num in messages[0].split(): if os.path.exists('/var/spool/amavis/bayesian/' + num): if type == 'spam': print 'Razoring ' + num os.system('/usr/bin/razor-report -f /var/spool/amavis/bayesian/' + n um) os.unlink('/var/spool/amavis/bayesian/' + num) else: print 'Something happened to /var/spool/amavis/bayesian/' + num + '. It is missing' ------------- END process_spam_and_ham.py ------------ I hope this helps someone. Avi