-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2014-05-05 at 12:29 -0700, Linda Walsh wrote:

Carlos E. R. wrote:

...

1 gigabyte resident size?

ps afxu:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 27884 0.0 0.8 153440 66892 ? Ss May04 0:06 /usr/sbin/spamd -d -c --max-children=8 -r /var/run/spamd.pid root 4034 1.1 14.6 1295260 1201160 ? S May04 8:07 \_ spamd > child root 4035 0.5 13.5 1206876 1112952 ? S May04 4:09 \_ spamd child

Current date is Mon May 5 11:42:23 CEST 2014, so it has been running just about 11 hours.

---

That's after nearly a week of uptime & 2h20m of cpu time, but note, my command line has:

root 2363 0.0 0.0 167108 66548 ? Ss Apr28 2:20 /usr/bin/spamd --syslog=local7 --allow-tell --create-prefs --daemonize --max-children=24 --min-spare=6 --max-conn-per-child=1 --user-config --listen-ip=127.0.0.1 --allowed-ips=192.168.3.0/24,127.0.0.1/32,192.168.4.0/24 --ipv4 -g spamd -r /var/run/spamd.pid

** Significantly: "--max-conn-per-child=1"

It is not in the spamd manual. There is this paragraph in the man: See the README file in the "spamd" directory of the SpamAssassin distribution for more details. I see "/usr/share/doc/packages/spamassassin/README", but it also says nothing of that variable.

...

I may undertand the size increase while processing a problematic email but the size should reduce itself after finishing with it. It did not.

---

Where is 'this' (size reduction) documented? That may be the prob....

Dunno, it stands to reason. Ok, see here: /usr/share/doc/packages/spamassassin/README: When spamd receives a connection, it spawns a child to handle the request. The child will expect to read an email message from the network socket, which should then be closed for writing on the other end (so spamd receives an EOF). spamd will then use SA to rewrite the message, and dump the processed message back to the socket before closing the connection. The child process then dies. But that paragraph has this prefix, which makes it obsolete: /* * FIXME: The following paragraph(s) have to be updated as childs are now * pre-spawned */ So the entire documentation is wrong. - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlNpXAcACgkQtTMYHG2NR9WJLACfaYAX+4Id1/avR7964jDXX5IW 1HsAn3RwUYtU0n99vfltHOgYXC+wmj4Q =lAyy -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 05/07/2014 12:37 AM, Per Jessen wrote:

Carlos E. R. wrote:

...

On Monday, 2014-05-05 at 12:29 -0700, Linda Walsh wrote:

...

** Significantly: "--max-conn-per-child=1"

It is not in the spamd manual.

It is in mine, but my spamassassin is backlevel and built from source:

NAME spamd - daemonized version of spamassassin

SYNOPSIS spamd [options]

Options:

[snip] -m num, --max-children=num Allow maximum num children --min-children=num Allow minimum num children --min-spare=num Lower limit for number of spare children --max-spare=num Upper limit for number of spare children --max-conn-per-child=num Maximum connections accepted by child before it is respawned

So what's the significance of this? Like many others here I run a single user system, its just me who who uses it. I bring in my mail using fetchmail which feeds into procmail. The procmail makes use of spamc, the client that calls on spamd. It processes one mail message at a time. I can see that by watching the logs with 'tail -f'. I emphasise: ONE MESSAGE AT A TIME. There is no parallelism. Spamc keeps getting called, one message at a time. There is noone else running any mail service, nothing else calling on spamd. Only me. And the procmail gets the result from spamc, spamc has exited, procmail delivers the message and goes on to the next one. So why have many spamd children? I can see the logic in having one forked off ready, but having 4? Having 8? Yes that would make sense in a richly multi-user context where there are going to be many users/threads calling on it in parallel. But that doesn't apply for my context. And I think many other users are signle users of their machines. -- "Of all tyrannies a tyranny sincerely exercised for the good of its victims may be the most oppressive. It may be better to live under robber barons than under omnipotent moral busybodies, the robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for own good will torment us without end, for they do so with the approval of their own conscience." -- C.S. Lewis, _God in the Dock_ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 05/07/2014 08:38 AM, Per Jessen wrote:

Anton, I'm sure you're making a point here, but I can't see what it is?

Same one I always make :-) Context is Everything -- Things are more like they are now than they ever were before. -- Dwight D. Eisenhower -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2014-05-07 14:01, Anton Aylward wrote:

So what's the significance of this?

Like many others here I run a single user system, its just me who who uses it.

Like me.

I bring in my mail using fetchmail which feeds into procmail. The procmail makes use of spamc, the client that calls on spamd.

My fetchmail sends to postfix, who eventually calls procmail. And it can send several posts at the same time. If the email is processed by procmail faster than fetchmail gets them, it goes one by one. If not, it starts sending more simultaneously, up to a limit defined in "/etc/postfix/master.cf": procmail unix - n n - 7 pipe flags=R user=nobody argv=/usr/bin/procmail -t -m /etc/procmailrc ${sender} ${recipient} 7 is the limit in my case, and they do get used. And I need more than one, because each email takes time to process; currently: spamd: clean message (-2.7/5.0) for cer:1000 in 2.7 seconds, 6835 bytes. spamd: clean message (-2.6/5.0) for cer:1000 in 3.7 seconds, 8887 bytes. spamd: clean message (0.4/5.0) for cer:1000 in 3.5 seconds, 8195 bytes. spamd: clean message (-2.6/5.0) for cer:1000 in 3.7 seconds, 7195 bytes. spamd: clean message (-1.6/5.0) for cer:1000 in 3.4 seconds, 11125 bytes. Some take longer: spamd: clean message (-101.9/5.0) for cer:1000 in 9.7 seconds, 3561 bytes. Some way longer: spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes. (see the size) or spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by At times, I got consistent processing times of 6..12 seconds each. A hundred emails would be processed in 1200 seconds, which I find intolerable - with 5% CPU load at most. The hurdle were the network test timeouts, I think. So yes, spamd with fetchmail paralelizes - if you put postfix in the equation. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On 2014-05-07 15:35, Per Jessen wrote:

Carlos E. R. wrote:

...

Some way longer:

spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.

(see the size)

or

spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by

That's a very large spam-mail - quite unusual. 213 seconds to process it could be caused by lots of DNS timeouts.

But it is not spam :-) It was sent by me (and to me), from my android phone. It is a dump of just one WhatsApp chat. Yes, it could be DNS timeouts. I have the vague idea that cpu load was high, but I'm unsure. I would have to force filter it again with spamd - ah, I know how. Send to another local account.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13292 cer2 20 0 390280 296056 2112 R 99,87 3,613 0:20.82 spamd child

So, it is not DNS timeouts, it is really processing load.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13292 cer2 20 0 390280 296056 2112 R 99,85 3,613 2:39.40 spamd child

It finished:

<2.6> 2014-05-07 17:19:23 Telcontar spamd 13292 - - spamd: clean message (4.3/5.0) for cer2:1012 in 172.9 seconds, 15982025 bytes. <2.6> 2014-05-07 17:19:23 Telcontar spamd 13292 - - spamd: result: . 4 - DRUGS_MUSCLE,FREEMAIL_FROM,FUZZY_XPILL,LOTS_OF_MONEY,NORMAL_HTTP_TO_IP,T_FRT_COCK,T_FRT_FUCK1,WEIRD_QUOTING scantime=172.9,size=15982025,user=cer2,uid=1012,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=46429,mid=,autolearn=no <2.6> 2014-05-07 17:19:23 Telcontar spamd 13289 - - prefork: child states: BI <2.6> 2014-05-07 17:19:23 Telcontar spamd 13289 - - spamd: handled cleanup of child pid [13292] due to SIGCHLD: exit 0 <2.6> 2014-05-07 17:19:23 Telcontar spamd 13289 - - spamd: server successfully spawned child process, pid 10121 <2.6> 2014-05-07 17:19:23 Telcontar spamd 13289 - - prefork: child states: II <2.6> 2014-05-07 17:19:23 Telcontar postfix 10015 - - 27ED861664: to=, relay=local, delay=174, delays=0.43/0/0/173, dsn=2.0.0, status=sent (delivered to command: /usr/bin/procmail) <2.6> 2014-05-07 17:19:23 Telcontar postfix 3805 - - 27ED861664: removed

It went a bit faster than the other time, /just/ 173 seconds. And this time it was thought clean. Go figure. At least the child was instantly respawned, as I had changed the config, just one child per email. And memory size did not go that up - or it was not that precise email, there were several similar posts sent in a batch. Maybe I have to send several, locally, and remove --max-conn-per-child=1 temporarily, to reproduce the problem. Doable. Coffee first. :-) -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On 2014-05-07 17:51, Anton Aylward wrote:

On 05/07/2014 11:36 AM, Carlos E. R. wrote:

...

But it is not spam :-)

It was sent by me (and to me), from my android phone. It is a dump of just one WhatsApp chat.

I make aggressive use of whitelisting and blacklisting in my procmail BEFORE submitting to spamassassin, even though spamassassin config also has its own whitelist and blacklist.

If I submit from my android sent by me to me it will never go though spamassassin.

Yes this does take a bit more checking to make sure it is not being spoofed, but procmail calling formmail and grep is cheaper than spamassassin. It would be more so for you if you are calling upon a outside RBL service and the network timeout.

It is something to consider, thanks. But takes time and some work to do... -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

Anton Aylward wrote:

Actually the whitelisting, which is a lot easier, probably does more to take the load off spamassassin than the blacklisting. I've whitelisted my common correspondents and the lists I subscribe to:

:::::::::::::: whitelist.rc ::::::::::::::

# Test if the email's sender is whitelisted; if so, send it straight # to $DEFAULT. Note that this comes before any other filters. :0: * ? formail -x"From" -x"From:" -x"Sender:" \ -x"Reply-To:" -x"Return-Path:" -x"To:" \ | egrep -is -f ${HOME}/.whitelist

I'm sure that works very well for you, but because email-addresses are so easily forged, using the SA whitelisting directives: whitelist_from_rcvd, whitelist_from_spf, or whitelist_from_dkim would be "safer". Just FYI. -- Per Jessen, Zürich (11.0°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On 2014-05-07 15:35, Per Jessen wrote:

...

Carlos E. R. wrote:

...

Some way longer:

spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.

(see the size)

or

spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by

That's a very large spam-mail - quite unusual. 213 seconds to process it could be caused by lots of DNS timeouts.

But it is not spam :-)

According to spamassasin it was :-)

Yes, it could be DNS timeouts. I have the vague idea that cpu load was high, but I'm unsure. I would have to force filter it again with spamd - ah, I know how. Send to another local account.

...

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13292 cer2 20 0 390280 296056 2112 R 99,87 3,613 0:20.82 spamd child

So, it is not DNS timeouts, it is really processing load.

Then it's simply due to the size of it. SA is generally speaking a large collection of regex'es, they're totally CPU-bound.

It went a bit faster than the other time, /just/ 173 seconds.

And this time it was thought clean. Go figure.

If you want to know, look at the rules that were hit. -- Per Jessen, Zürich (14.5°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2014-05-07 17:36, Carlos E. R. wrote:

On 2014-05-07 15:35, Per Jessen wrote:

Maybe I have to send several, locally, and remove --max-conn-per-child=1 temporarily, to reproduce the problem. Doable.

Coffee first. :-)

I did send a bunch, and the sizes went up. At the end, several children were killed, but two remained, one .4 GB, another .6 GB (RSS is ps output). The solution would be to not have any child pre-spawned, or allow only one connection per child. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On 2014-05-08 07:32, Per Jessen wrote:

Carlos E. R. wrote:

Not be too picky, but that's only a work-around, not a solution. Anyway, sounds like you've got a reproducable problem, good going. I'm sure the SA guys will want to know.

It will have to wait. I have to hit the road. But I doubt they don't know that children do not release memory. I think then that the parent should watch that, and respawn children automatically when they grow a lot. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On 05/07/2014 09:21 AM, Carlos E. R. wrote:

Some way longer:

spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.

(see the size)

or

spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by

At times, I got consistent processing times of 6..12 seconds each. A hundred emails would be processed in 1200 seconds, which I find intolerable - with 5% CPU load at most. The hurdle were the network test timeouts, I think.

WOW! I process about 200 messages a day and grepping though the logs I see one at 7 seconds, two at 5 second and the rest at 1-2 seconds. My load average over that period is less than 0.5 (It occasionally peaked when I was running the nvidia driver from nvidia! Burt that's a different matter) This is with a single core CPU: Model name: AMD Athlon(tm) 64 Processor 3000+ Stepping: 2 CPU MHz: 1889.930 BogoMIPS: 3779.86 Perhaps you are right about the network - Razr? - timeouts. I have that disabled.

So yes, spamd with fetchmail paralelizes - if you put postfix in the equation.

Unless you have a very quick description of the mail.cf as well as the master.cf lets discuss that off-line. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Per Jessen wrote:

Anton Aylward wrote:

...

On 05/07/2014 09:21 AM, Carlos E. R. wrote:

...

Some way longer:

spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.

(see the size)

or

spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by

At times, I got consistent processing times of 6..12 seconds each. A hundred emails would be processed in 1200 seconds, which I find intolerable - with 5% CPU load at most. The hurdle were the network test timeouts, I think.

WOW!

I process about 200 messages a day and grepping though the logs I see one at 7 seconds, two at 5 second and the rest at 1-2 seconds.

My load average over that period is less than 0.5

(It occasionally peaked when I was running the nvidia driver from nvidia! Burt that's a different matter)

This is with a single core CPU: Model name: AMD Athlon(tm) 64 Processor 3000+ Stepping: 2 CPU MHz: 1889.930 BogoMIPS: 3779.86

Yeah, the CPU is largely irrelevant

Except for people scanning excessively large emails of several megabytes :-) -- Per Jessen, Zürich (14.0°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2014-05-07 23:54, Anton Aylward wrote:

On 05/07/2014 02:48 PM, Per Jessen wrote:

...

...

...

Yeah, the CPU is largely irrelevant Except for people scanning excessively large emails of several megabytes :-)

I don't know about you, but in my .procmailrc I have a rule that limits how large an email file should be submitted to spamassassin:

As I do, but I get large spam emails. I just saw one of 20 MB, but many are over 400KB (a hundred at least are in my training folder). With a low limit, they pass through. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

Anton Aylward wrote:

On 05/07/2014 02:48 PM, Per Jessen wrote:

...

...

...

Yeah, the CPU is largely irrelevant Except for people scanning excessively large emails of several megabytes :-)

I don't know about you, but in my .procmailrc I have a rule that limits how large an email file should be submitted to spamassassin:

Alternatively, use the spamc '-s' option to the max size to be processed. (the end effect is the same of course). -- Per Jessen, Zürich (11.2°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Anton Aylward wrote:

So why have many spamd children?

I can see the logic in having one forked off ready, but having 4? Having 8? Yes that would make sense in a richly multi-user context where there are going to be many users/threads calling on it in parallel. But that doesn't apply for my context.

And I think many other users are signle users of their machines.

The only time my spamd needs that many is after my 'fetchmail' process has been down a while (system down, or rebooted, and didn't restart fetchmail right away, or detected some other problem and killed the fetcher...etc.). Spamd takes about 7-10 seconds/email. But most of that time is *waiting* (network lookups). So, right after a restart of I might have a couple of hundred messages come in (have processed as many as 500-800 at once). They still all get processed in about a minute, maybe two at most. Sure, it could process those, say 700 emails sequentially, and take up to 2 hours to process the queue. BUT if it can process a few hundred in parallel, why not let it go? Does that answer your Q? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On 2014-05-07 06:37, Per Jessen wrote:

...

Carlos E. R. wrote:

...

On Monday, 2014-05-05 at 12:29 -0700, Linda Walsh wrote:

...

** Significantly: "--max-conn-per-child=1"

It is not in the spamd manual.

It is in mine, but my spamassassin is backlevel and built from source:

Then the openSUSE package has the wrong manuals. :-?

Yes, that's what it sounds like. -- Per Jessen, Zürich (14.9°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 05/05/2014 05:08 PM, Cristian Rodríguez wrote:

spamd is a perl program right ? it might be expected that it does not free memory. I do not know if perl frees memory only at exit, has a garbage collector or a combination of both.

With perl there is the interpreter written in C and the script which is a text file, and then there is the data-that-is-data. I realise also that there are some plugins that are compiled C code. So what is being reported here? As far as the interpreter itself goes, I expect it will be treated like any other compiled program with a core 'working set'. That code will be shared with any other instance of perl and with the user-context-client that is spawned to process a client's mail message. That client copy will process the mail message as data-that-is-data, and if I read correctly with exit when done. Strictly speaking there only needs to be one copy of spamd the parent process. :-) HA HA. See how Apache works and wonder how many 'parent' copies you want to be able to deal with various incoming mail streams. I'm the only user of my system, and mail comes in via fetchmail+procmail and procmail invoking spamc, only one source at a time, so having only one parent process and only one child at a time (-max-children) is all I need. YMMV. -- To do anything in this world worth doing, we must not stand back shivering and thinking of the cold and danger, but jump in, and scramble through as well as we can. Sydney Smith -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2014-05-06 at 00:26 -0400, Anton Aylward wrote:

Strictly speaking there only needs to be one copy of spamd the parent process. :-) HA HA. See how Apache works and wonder how many 'parent' copies you want to be able to deal with various incoming mail streams. I'm the only user of my system, and mail comes in via fetchmail+procmail and procmail invoking spamc, only one source at a time, so having only one parent process and only one child at a time (-max-children) is all I need. YMMV.

There is one parent, and several children when needed, up to a maximum of 8. I allow several children because some of the network based tests seem to need up to 6 seconds, as spammed waits for an answer and doing nothing all that time. So, when processing a bunch of email from this email list, say 50, it took a long time to do. Having several children waiting speeds things a lot. - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlNpXTUACgkQtTMYHG2NR9V/XACfQNRa60unhtzuzt5x192UoGXY QiYAoIw8mC5JcTcBqfeS3Qr71WWz3pDd =z+tz -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2014-05-06 at 09:15 +0200, Per Jessen wrote:

Carlos E. R. wrote:

...

top:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4034 root 20 0 1295260 1,146g 2780 S 0,000 14,66 8:07.59 spamd child 4035 root 20 0 1206876 1,061g 2908 S 0,000 13,58 4:09.77 spamd child

1 gigabyte resident size?

That seems pretty unusual. I run a test-system cluster with 4 very old boxes, each with only 384Mb RAM. They each run typically 2-3 spamd children, but don't think I've ever seen that kind of memory usage. Mind you, I only scan mails that are less than 2Mb.

I increased that limit, because I often got spam mails way bigger than that, that went through to my inbox.

...

Ah, and that email is not spam. It is a dump of a WhatsApp chat with multimedia attachments.

How big is the email itself?

"Just" 19 MB. Not that /huge/.

spamd does need to keep it in-core, so until a child is re-spawned, the memory usage might remain at the amount required for the largest email.

The huge memory usage was by the children, not the parent. That email contains about a hundred attachments. They are marked like: 25 47 KB Video/3GPP (Name: "PTT-20130807-WA0018.3gp") 26 ~10 KB Text/* (Name: "PTT-20130807-WA0019.3ga") 27 89 KB Image/JPEG (Name: "IMG-20130807-WA0020.jpg") 28 74 KB Image/JPEG (Name: "IMG-20130808-WA0000.jpg") 29 ~17 KB Text/* (Name: "PTT-20130808-WA0001.3ga") 30 ~20 KB Text/* (Name: "PTT-20130808-WA0002.3ga") 31 ~10 KB Text/* (Name: "PTT-20130808-WA0003.3ga") Notice that the .3ga are marked text, when they are multimedia (voice messages, I think). Thunderbird tries to display them, and obviously fails. WhatsApp has made a mess of it. I have a /etc/sysconfig/spamd with these settings (old): SPAMD_AWL=no SPAMD_NICE=yes SPAMD_ARGS="-d -c --max-children=8 " SPAM_SA_UPDATE="no" SPAM_SA_COMPILE="no" SPAM_AMAVISD_RESTART="no" SPAM_SPAMD_RESTART="yes" But there is also a "spampd" file with this single line: SPAMPD_OPTIONS="--port=10025 --relayhost=127.0.0.1:10026 --user=spamfilter --tagall --children=5 --maxsize=250" which are some of the options that Linda has, it looks like. But: cer@Telcontar:/etc/sysconfig> rpm -qf /etc/sysconfig/spampd file /etc/sysconfig/spampd is not owned by any package cer@Telcontar:/etc/sysconfig> It is dated "Feb 25 21:45". I don't know what "SPAMPD" is. - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlNpYysACgkQtTMYHG2NR9WH3gCeMuIF3vxDDW+JNaNjGNDXt40l XdAAn10/kB3tiaUVREVsryV85abx4xns =BzmK -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2014-05-07 06:33, Per Jessen wrote:

Carlos E. R. wrote:

...

...

How big is the email itself?

"Just" 19 MB. Not that /huge/.

Any chance previous emails might have been much bigger? The number of emails processed by spamd before it is respawned is set with "--max-conn-per-child". Maybe you can find a much bigger mail in the log?

No, I just looked. Typically, when I fetch email the number of childs increase, and at the end of activity it drops to just two. It just chanced that the two were those emails, because they took several minutes to process.

...

That email contains about a hundred attachments.

Okay, that is pretty unusual, I would say - maybe you could reproduce the problem with another such email? If spamassassin has some sort of ballooning memory issue when processing mails with many attachments, I'm sure the developers would like to know.

I can send it to myself, internally. It would get processed. [...] Nope, it did not, just tried. I do not know why :-? Ah, yes, I do, the local account is not filtered. However, I can not provide this sample post to anyone, it is private.

...

I have a /etc/sysconfig/spamd with these settings (old):

SPAMD_AWL=no SPAMD_NICE=yes SPAMD_ARGS="-d -c --max-children=8 " SPAM_SA_UPDATE="no" SPAM_SA_COMPILE="no" SPAM_AMAVISD_RESTART="no" SPAM_SPAMD_RESTART="yes"

But there is also a "spampd" file with this single line:

SPAMPD_OPTIONS="--port=10025 --relayhost=127.0.0.1:10026 --user=spamfilter --tagall --children=5 --maxsize=250"

Those are clearly options for the spamd daemon. If you google "SPAMPD_OPTIONS" you'll see a few hits here and there.

But they are not used, as seen in the "ps" output: root 13289 0.0 0.8 151520 66388 ? Ss 00:30 0:05 /usr/sbin/spamd -d -c --max-children=8 --max-conn-per-child=1 -r /var/run/spamd.pid I added the max-conn-per-child yesterday. But notice the max-children=8, not 5. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On 2014-05-07 15:17, Per Jessen wrote:

Carlos E. R. wrote:

To quickly test if the many attachments is the issue, you could just send yourself another mail with a lot of attachments.

I have first to modify first procmail so that local email is processed by spamd. It may be the many attachments, but it can also be the type of those. Notice that whatsapp classifies as plain text the voice messages, so spamd might try to analyse that "text". But the issue, to me, is not really the huge memory, but that the child was not restarted after processing, if memory can not be freed. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On 2014-05-07 16:55, Per Jessen wrote:

Carlos E. R. wrote:

...

But the issue, to me, is not really the huge memory, but that the child was not restarted after processing, if memory can not be freed.

Ah okay - I didn't realise that was the issue. That sounds like something to report to the spamassassin people.

Yes, memory is there to be used as needed, as long as it is released when not. The max-conn-per-child switch handles that part, I think. Maybe that causes some performance decrease, spawning more often - but Linux is good at that. I can do some limited testing now, I have to prepare for a little field trip. Reporting takes longer. And if the openSUSE package is different than the upstream one (the obsolete man page indicates that), I can't report to them, but to openSUSE instead. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On Wed, 7 May 2014 15:02:59 Carlos E. R. wrote:

On 2014-05-07 06:33, Per Jessen wrote:

...

Carlos E. R. wrote:

...

...

...

How big is the email itself?

"Just" 19 MB. Not that /huge/.

[...]

...

...

SPAMD_AWL=no SPAMD_NICE=yes SPAMD_ARGS="-d -c --max-children=8 " SPAM_SA_UPDATE="no" SPAM_SA_COMPILE="no" SPAM_AMAVISD_RESTART="no" SPAM_SPAMD_RESTART="yes"

Carlos, I've not read this entire thread, but I thought that it might be useful if I mentioned that I have spamd running on a Raspberry Pi filtering ALL my incoming mail from 5 different accounts, and the load average rarely gets above 0.05. RAM usage never gets above 25% (of 512MB). The "magic bullet" on this was a) use spamc/spamd instead of calling spamassassin for every mail (which I note that you are doing) and b) use COMPILED rulesets (I see above that SPAM_SA_COMPILE is set to "no"). When I moved it to compiled rulesets the load average dropped from 28 (no, that is not a typo) to 0.05. I think the maximum of children it is allowed to spawn is 5. Try running sa-compile on your ruleset and set SPAM_SA_COMPILE to "yes" and see if that makes any difference for your setup. After about a week of experimenting when I first set it up on the Rpi I was just about ready to give it up as impossible; the compiled rulesets were the key. BTW, that Rpi runs fetchmail, sendmail (yes, sendmail 8.1.5; not postfix - I could never get along with it), procmail, spamassassin, clamav and dovecot and handles all my email (including remote access from outside). Ticks along beautifully, makes no noise and uses less than 5W of power. :) Regards, Rodney. -- ============================================================== Rodney Baker VK5ZTV rodney.baker@iinet.net.au ============================================================== -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org